In experiments, researchers and practitioners commonly allocate subjects randomly to the different treatment conditions equally before the experiment starts. While this approach is intuitive, it means that new information gathered during the experiment is not utilized until after the experiment has ended. Based on methodological approaches from other scientific disciplines such as computer science and medicine, we suggest a machine learning algorithm for subject allocation in randomized experiments.
Specifically, we discuss a Bayesian multi-armed bandit algorithm for randomized controlled trials and use Monte Carlo simulations to compare its efficiency with randomized controlled trials that have a fixed and balanced subject allocation. Our findings indicate that a randomized allocation based on Bayesian multi-armed bandits is more efficient and ethical in most settings. We develop recommendations for researchers and discuss the limitations of our approach.
Project participants: Prof. Torsten Biemann and Chris Kaibel
Results were accepted for publication by the journal „Organisational Research Methods“:
A related tool for the deployment of Multi-armed bandits can be found here:
Relying on algorithm-based decisions seems to be a promising step towards more data-driven decision-making and an alternative to subjective human decisions in the recruiting and employee selection context. However, the question that remains to be answered is whether algorithm-based hiring decisions help organizations attracting and recruiting a diverse workforce with unique talents.
Building on established characteristics of procedural justice, we conducted two experimental studies to examine the effects of algorithm vs. human decision-makers on applicants’ perceptions of the selection process and organizational attractiveness. We found that applicants do not automatically perceive algorithms as purely negative or positive for the selection process. Instead, the perceptions seem to differ between outcome variables and dependent on applicants’ experiences and believes. While algorithms might help to attract applicants that have made discrimination experiences, algorithms seem to discourage applicants with unique characteristics. Implications and future research directions are discussed.
Outcomes were published in „Best Paper Proceedings“ of the Academy of Management 2019.