AI hiring tools show racial bias against Black and Asian applicants, Stanford study reveals

AI hiring tools are biasing job decisions

The researchers don't yet know why these tools are biased. They argue that this work supports the need for more transparency into how AI hiring tools work, and they note that companies are still responsible for checking for biases in their hiring tools and processes. About 90 percent of employers use AI to some extent in hiring, yet research on how this is impacting job seekers is virtually nonexistent. In one of the first studies to analyze AI hiring tools, Stanford researchers discovered that, for many job applications, the algorithms were making racially biased decisions. "A lot of prior studies had shown racial bias in hiring, when people are making the decisions," said co-author Dan Jurafsky, the Jackson Eli Reynolds Professor in Humanities in the School of Humanities and Sciences and a professor of computer science in the School of Engineering. "It was surprising that AI systems that use game-based assessment to rank people were still biased against Black and Asian applicants." The team also found evidence that some candidates were repeatedly turned away from multiple jobs - a sign that companies' reliance on algorithms all produced by the same vendor could shut out some candidates. The researchers presented their results at the ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT) in Montréal on June 27. The rise of AI tools in hiring While job listing websites and the expansion of remote roles have made it easier to apply for more jobs, it's also hard for candidates to stand out among growing heaps of applications. In 2024, for instance, Google received more than 3 million job applications for about 20,000 roles. Many employers have contracted with third-party AI vendors to help screen candidates. In addition to managing the flood of applications, AI-based tools often promise to reduce the human biases that can hurt some job seekers. But this shift also means that screening decisions at numerous companies have been turned over to a relatively small number of AIs. The authors of this study wondered what effect this "algorithmic monoculture" could be having on the application process. "Many different employers use hiring AI tools, sometimes the exact same tools or tools built by the same vendor, and we were interested in what the consequences of that are," said lead author Rishi Bommasani, senior research scholar at Stanford's Institute for Human-Centered Artificial Intelligence. To find out, the research team tapped a dataset from the company Pymetrics. The dataset consisted of more than 4 million applications submitted between 2018 and 2022 to nearly 2,000 positions. After initially applying for a job, the applicants were redirected to Pymetrics' game-based assessments, which aim to measure soft skills such as risk tolerance, focus, and generosity. Based on their scores, algorithms then sort candidates into "recommend" and "do not recommend" categories. Using applications for which demographic information was included, the researchers searched for evidence of racial bias. They used a threshold set by the U.S. government called the "four-fifths rule." If one group is recommended for a position at less than 80 percent of the rate of the most-recommended group, it's a red flag for potential discrimination. When the researchers first investigated the data, they asked whether the applications, as a whole, were within this standard. They found that they were, overall. "There might be some bias, but not rising to the levels of legal concern," said Bommasani. But a new picture emerged when they calculated the rate at which groups were recommended for each individual job opening. They found that 15 percent of Asian applicants and 26 percent of Black applicants applied to jobs where the AI tool appeared to be biased against their racial group. The screening algorithms for those jobs were recommending Asian and Black candidates at a rate less than 80 percent of the leading group, often white candidates. The researchers calculated that if racial groups had been selected at the same rate, 40,000 more applications from Asian and Black candidates would have been recommended. "We definitely didn't expect this," said Bommasani, especially since prior analyses of the aggregate applications didn't show very much bias. "Some companies think that AI will help them be more fair in their decision-making," he added. "That's not necessarily what our results suggest." The researchers also considered, for applicants submitting to multiple positions, how often they would be rejected by all - an outcome they called "systemic rejection." They found that 4 percent of applicants who applied to 10 positions using the games-based assessment were given a "do not recommend" by the AIs for all positions. This rate was higher than what would be expected if companies were making independent decisions about whether to move an application forward. "The AI algorithms we studied were much more likely to act identically, leading a person to be universally rejected, than if the companies were acting independently," said Jurafsky. "That suggests that this kind of monoculture, in which every algorithm is identical, can cause problems." Making hiring tools fair and transparent It's no secret that human hiring managers can introduce bias into job decisions, which studies have shown for decades. The new study shows that AIs, too, can make biased decisions even when they are judging seemingly neutral criteria such as the gameplay scores. "We don't yet understand which kinds of algorithms exhibit these differential impacts for different applicant groups and we don't know what is causing these disparities," said Jurafsky. "The most important thing we need is continued study. We can't fix a disparity if we don't know what's causing it." The results reveal how only looking at the average rates at which applicants are moving forward across all jobs can hide disparities. "One lesson from doing this work is that it is important to always disaggregate, for there could be a lot of complexity that's covered up by averages," said co-author Percy Liang, a professor of computer science. The findings also underscore the need for independent research of such third-party tools. But hiring data like the team used tends to be kept private by companies, preventing such scrutiny. New policies requiring AI companies to share their data could help hiring processes be more transparent. "Absent policy, it's incredibly unlikely we'll see more research into the effects of AI and hiring," said Bommasani. "There's just not really any way to get data." The results also show that employers, who ultimately bear the responsibility of preventing discrimination, should question the vendors they hire for AI-based screening to see if they have verified that their algorithms are not discriminating, said Bommasani. "There is a clear incentive for firms to internalize this and make more sophisticated procurement decisions."

AI hiring software screens millions of applicants, but new evidence shows racial bias can hide job by job

About 90% of employers use AI to some extent in hiring, yet research on how this is impacting job seekers is virtually nonexistent. In one of the first studies to analyze AI hiring tools, Stanford researchers have discovered that for many job applications, the algorithms were making racially biased decisions. "A lot of prior studies had shown racial bias in hiring, when people are making the decisions," said co-author Dan Jurafsky, the Jackson Eli Reynolds Professor in Humanities in the School of Humanities and Sciences and a professor of computer science in the School of Engineering. "It was surprising that AI systems that use game-based assessment to rank people were still biased against Black and Asian applicants." The team also found evidence that some candidates were repeatedly turned away from multiple jobs -- a sign that companies' reliance on algorithms all produced by the same vendor could shut out some candidates. The researchers presented their results at the ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT) in Montréal on June 27. The findings are published on the arXiv preprint server. The rise of AI tools in hiring While job listing websites and the expansion of remote roles have made it easier to apply for more jobs, it's also hard for candidates to stand out among growing heaps of applications. In 2024, for instance, Google received more than three million job applications for about 20,000 roles. Many employers have contracted with third-party AI vendors to help screen candidates. In addition to managing the flood of applications, AI-based tools often promise to reduce the human biases that can hurt some job seekers. But this shift also means that screening decisions at numerous companies have been turned over to a relatively small number of AIs. The authors of this study wondered what effect this "algorithmic monoculture" could be having on the application process. "Many different employers use hiring AI tools, sometimes the exact same tools or tools built by the same vendor, and we were interested in what the consequences of that are," said lead author Rishi Bommasani, senior research scholar at Stanford's Institute for Human-Centered Artificial Intelligence. To find out, the research team tapped a dataset from the company Pymetrics. The dataset consisted of more than four million applications submitted between 2018 and 2022 to nearly 2,000 positions. After initially applying for a job, the applicants were redirected to Pymetrics's game-based assessments, which aim to measure soft skills such as risk tolerance, focus, and generosity. Based on their scores, algorithms then sort candidates into "recommend" and "do not recommend" categories. Using applications for which demographic information was included, the researchers searched for evidence of racial bias. They used a threshold set by the U.S. government called the "four-fifths rule." If one group is recommended for a position at less than 80% of the rate of the most-recommended group, it's a red flag for potential discrimination. When the researchers first investigated the data, they asked whether the applications, as a whole, were within this standard. They found that they were, overall. "There might be some bias, but not rising to the levels of legal concern," said Bommasani. But a new picture emerged when they calculated the rate at which groups were recommended for each individual job opening. They found that 15% of Asian applicants and 26% of Black applicants applied to jobs where the AI tool appeared to be biased against their racial group. The screening algorithms for those jobs recommended Asian and Black candidates at a rate less than 80% of the leading group, often white candidates. The researchers calculated that if racial groups had been selected at the same rate, 40,000 more applications from Asian and Black candidates would have been recommended. "We definitely didn't expect this," said Bommasani, especially since prior analyses of the aggregate applications didn't show very much bias. "Some companies think that AI will help them be more fair in their decision-making," he added. "That's not necessarily what our results suggest." The researchers also considered, for applicants submitting to multiple positions, how often they would be rejected by all -- an outcome they called "systemic rejection." They found that 4% of applicants who applied to 10 positions using the games-based assessment were given a "do not recommend" by the AIs for all positions. This rate was higher than what would be expected if companies were making independent decisions about whether to move an application forward. "The AI algorithms we studied were much more likely to act identically, leading a person to be universally rejected, than if the companies were acting independently," said Jurafsky. "That suggests that this kind of monoculture, in which every algorithm is identical, can cause problems." Making hiring tools fair and transparent It's no secret that human hiring managers can introduce bias into job decisions, which studies have shown for decades. The new study shows that AIs, too, can make biased decisions even when they are judging seemingly neutral criteria such as the gameplay scores. "We don't yet understand which kinds of algorithms exhibit these differential impacts for different applicant groups and we don't know what is causing these disparities," said Jurafsky. "The most important thing we need is continued study. We can't fix a disparity if we don't know what's causing it." The results reveal how only looking at the average rates at which applicants are moving forward across all jobs can hide disparities. "One lesson from doing this work is that it is important to always disaggregate, for there could be a lot of complexity that's covered up by averages," said co-author Percy Liang, a professor of computer science. The findings also underscore the need for independent research of such third-party tools. But hiring data like the team used tends to be kept private by companies, preventing such scrutiny. New policies requiring AI companies to share their data could help hiring processes be more transparent. "Absent policy, it's incredibly unlikely we'll see more research into the effects of AI and hiring," said Bommasani. "There's just not really any way to get data." The results also show that employers, who ultimately bear the responsibility of preventing discrimination, should question the vendors they hire for AI-based screening to see if they have verified that their algorithms are not discriminating, said Bommasani. "There is a clear incentive for firms to internalize this and make more sophisticated procurement decisions."