Analyzing the Effects of a Human-in-the-Loop Candidate Recommendation Algorithm at Jobindex, by Mesut Kaya & Toine Bogers, published in ACM Transactions on Recommender Systems

Recruiting is the process of assessing relevant candidates for an open position based on their education, work experience, and knowledge, skills, and abilities. As part of a collaborative project between academia and industry, we developed an automated candidate recommendation system to support recruiters in this time-consuming task of matching CVs to job postings. We chronicle the development and deployment of a candidate recommender system at Jobindex, expanding our focus from a pure machine learning problem to a holistic overview of the development and deployment of such a system in a real-world setting. After extensive offline and online experimentation, we integrated our algorithm into the recruiters’ everyday workflow. Our second contribution is a detailed analysis of how these recruiters have adopted this candidate recommender system, and which factors influence their engagement with the slate of suggested candidates. In this article, we present the results of 17 months of data (corresponding to 41,390 jobs) and show how engaging with these recommendations has impacted the recruiters’ work, and which factors influence their task success. While adoption of the new system was initially hindered by deeply-ingrained habits and a lack of trust in AI, over time the combination of human and automated recruitment shows considerable promise across a variety of job and recruiter characteristics.