AI Resume Screening Tools Show Significant Racial and Gender Bias, Study Finds

Study: AIs prefer white, male names on resumes, just like humans

Anyone familiar with HR practices probably knows of the decades of studies showing that resumes with Black- and/or female-presenting names at the top get fewer callbacks and interviews than those with white- and/or male-presenting names -- even if the rest of the resume is identical. A new study shows those same kinds of biases also show up when large language models are used to evaluate resumes instead of humans. In a new paper published during last month's AAAI/ACM Conference on AI, Ethics and Society, two University of Washington researchers ran hundreds of publicly available resumes and job descriptions through three different Massive Text Embedding (MTE) models. These models -- based on the Mistal-7B LLM -- had each been fine-tuned with slightly different sets of data to improve on the base LLM's abilities in "representational tasks including document retrieval, classification, and clustering," according to the researchers, and had achieved "state-of-the art performance" in the MTEB benchmark. Rather than asking for precise term matches from the job description or evaluating via a prompt (e.g., "does this resume fit the job description?"), the researchers used the MTEs to generate embedded relevance scores for each resume and job description pairing. To measure potential bias, the resumes were first run through the MTEs without any names (to check for reliability) and were then run again with various names that achieved high racial and gender "distinctiveness scores" based on their actual use across groups in the general population. The top 10 percent of resumes that the MTEs judged as most similar for each job description were then analyzed to see if the names for any race or gender groups were chosen at higher or lower rates than expected. Across more than three million resume and job description comparisons, some pretty clear biases began to appear. In all three MTE models, white names were preferred in a full 85.1 percent of the conducted tests, compared to Black names being preferred in just 8.6 percent (the remainder showed score differences close enough to zero to be judged insignificant). When it came to gendered names, the male name was preferred in 51.9 percent of tests, compared to 11.1 percent where the female name was preferred. The results could be even clearer in "intersectional" comparisons involving both race and gender; Black male names were preferred to white male names in "0% of bias tests," the researchers wrote.

AI tools show biases in ranking job applicants' names according to perceived race and gender

The future of hiring, it seems, is automated. Applicants can now use artificial intelligence bots to apply to job listings by the thousands. Companies -- which have long automated parts of the process -- are now deploying the latest AI large language models to write job descriptions, sift through resumes and screen applicants. An estimated 99% of Fortune 500 companies now use some form of automation in their hiring process. This automation can boost efficiency, and some claim it can make the hiring process less discriminatory. But new University of Washington research has found significant racial, gender and intersectional bias in how three state-of-the-art large language models, or LLMs, ranked resumes. The researchers varied names associated with white and Black men and women across over 550 real-world resumes and found the LLMs favored white-associated names 85% of the time, female-associated names only 11% of the time, and never favored Black male-associated names over white male-associated names. The team presented its research on Oct. 22 at the AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society in San Jose. "The use of AI tools for hiring procedures is already widespread, and it's proliferating faster than we can regulate it," said lead author Kyra Wilson, a UW doctoral student in the Information School. "Currently, outside of a New York City law, there's no regulatory, independent audit of these systems, so we don't know if they're biased and discriminating based on protected characteristics such as race and gender. And because a lot of these systems are proprietary, we are limited to analyzing how they work by approximating real-world systems." Previous studies have found that ChatGPT exhibits racial and disability bias when sorting resumes. But those studies were relatively small -- using only one resume or four job listings -- and ChatGPT's AI model is a so-called "black box," limiting options for analysis. The UW team wanted to study open-source LLMs and do so at scale. They also wanted to investigate intersectionality across race and gender. The researchers varied 120 first names associated with white and Black men and women across the resumes. They then used three state-of-the-art LLMs from three different companies -- Mistral AI, Salesforce and Contextual AI -- to rank the resumes as applicants to 500 real-world job listings. These were spread across nine occupations, including human resources worker, engineer and teacher. This amounted to more than three million comparisons between resumes and job descriptions. The team then evaluated the system's recommendations across these four demographics for statistical significance. The system preferred: The team also looked at intersectional identities and found that the patterns of bias aren't merely the sums of race and gender identities. For instance, the study showed the smallest disparity between typically white female and typically white male names. And the systems never preferred what are perceived as Black male names to white male names. Yet they also preferred typically Black female names 67% of the time versus 15% of the time for typically Black male names. "We found this really unique harm against Black men that wasn't necessarily visible from just looking at race or gender in isolation," Wilson said. "Intersectionality is a protected attribute only in California right now, but looking at multidimensional combinations of identities is incredibly important to ensure the fairness of an AI system. If it's not fair, we need to document that so it can be improved upon." The team notes that future research should explore bias and harm reduction approaches that can align AI systems with policies. It should also investigate other protected attributes, such as disability and age, as well as looking at more racial and gender identities -- with an emphasis on intersectional identities. "Now that generative AI systems are widely available, almost anyone can use these models for critical tasks that affect their own and other people's lives, such as hiring," said senior author Aylin Caliskan, a UW assistant professor in the iSchool. "Small companies could attempt to use these systems to make their hiring processes more efficient, for example, but it comes with great risks. The public needs to understand that these systems are biased. And beyond allocative harms, such as hiring discrimination and disparities, this bias significantly shapes our perceptions of race and gender and society."

AI tools show biases in ranking job applicants' | Newswise

The future of hiring, it seems, is automated. Applicants can now use artificial intelligence bots to apply to job listings by the thousands. And companies -- which have long automated parts of the process -- are now deploying the latest AI large language models to write job descriptions, sift through resumes and screen applicants. An estimated 99% of Fortune 500 companies now use some form of automation in their hiring process. This automation can boost efficiency, and some claim it can make the hiring process less discriminatory. But new University of Washington research found significant racial, gender and intersectional bias in how three state-of-the-art large language models, or LLMs, ranked resumes. The researchers varied names associated with white and Black men and women across over 550 real-world resumes and found the LLMs favored white-associated names 85% of the time, female-associated names only 11% of the time, and never favored Black male-associated names over white male-associated names. The team presented its research Oct. 22 at the AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society in San Jose. "The use of AI tools for hiring procedures is already widespread, and it's proliferating faster than we can regulate it," said lead author Kyra Wilson, a UW doctoral student in the Information School. "Currently, outside of a New York City law, there's no regulatory, independent audit of these systems, so we don't know if they're biased and discriminating based on protected characteristics such as race and gender. And because a lot of these systems are proprietary, we are limited to analyzing how they work by approximating real-world systems." Previous studies have found ChatGPT exhibits racial and disability bias when sorting resumes. But those studies were relatively small -- using only one resume or four job listings -- and ChatGPT's AI model is a so-called "black box," limiting options for analysis. The UW team wanted to study open-source LLMs and do so at scale. They also wanted to investigate intersectionality across race and gender. The researchers varied 120 first names associated with white and Black men and women across the resumes. They then used three state-of-the-art LLMs from three different companies -- Mistral AI, Salesforce and Contextual AI -- to rank the resumes as applicants to 500 real-world job listings. These were spread across nine occupations, including human resources worker, engineer and teacher. This amounted to more than three million comparisons between resumes and job descriptions. The team then evaluated the system's recommendations across these four demographics for statistical significance. The system preferred: The team also looked at intersectional identities and found that the patterns of bias aren't merely the sums of race and gender identities. For instance, the study showed the smallest disparity between typically white female and typically white male names. And the systems never preferred what are perceived as Black male names to white male names. Yet they also preferred typically Black female names 67% of the time versus 15% of the time for typically Black male names. "We found this really unique harm against Black men that wasn't necessarily visible from just looking at race or gender in isolation," Wilson said. "Intersectionality is a protected attribute only in California right now, but looking at multidimensional combinations of identities is incredibly important to ensure the fairness of an AI system. If it's not fair, we need to document that so it can be improved upon." The team notes that future research should explore bias and harm reduction approaches that can align AI systems with policies. It should also investigate other protected attributes, such as disability and age, as well as looking at more racial and gender identities -- with an emphasis on intersectional identities. "Now that generative AI systems are widely available, almost anyone can use these models for critical tasks that affect their own and other people's lives, such as hiring," said senior author Aylin Caliskan, a UW assistant professor in the iSchool. "Small companies could attempt to use these systems to make their hiring processes more efficient, for example, but it comes with great risks. The public needs to understand that these systems are biased. And beyond allocative harms, such as hiring discrimination and disparities, this bias significantly shapes our perceptions of race and gender and society." This research was funded by the U.S. National Institute of Standards and Technology.

AI overwhelmingly prefers white and male job candidates in new test of resume-screening bias

As employers increasingly use digital tools to process job applications, a new study from the University of Washington highlights the potential for significant racial and gender bias when using AI to screen resumes. The UW researchers tested three open-source, large language models (LLMs) and found they favored resumes from white-associated names 85% of the time, and female-associated names 11% of the time. Over the 3 million job, race and gender combinations tested, Black men fared the worst with the models preferring other candidates nearly 100% of the time. Why do machines have such a outsized bias for picking white male job candidates? The answer is a digital take on the old adage "you are what you eat." "These groups have existing privileges in society that show up in training data, [the] model learns from that training data, and then either reproduces or amplifies the exact same patterns in its own decision-making tasks," said Kyra Wilson, a doctoral student at the UW's Information School. Wilson conducted the research with Aylin Caliskan, a UW assistant professor in the iSchool. They presented their results last week at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society in San Jose, Calif. The experiment used 554 resumes and 571 job descriptions taken from real-world documents. The researchers then doctored the resumes, swapping in 120 first names generally associated with people who are male, female, Black and/or white. The jobs included were chief executive, marketing and sales manager, miscellaneous manager, human resources worker, accountant and auditor, miscellaneous engineer, secondary school teacher, designer, and miscellaneous sales and related worker. The results demonstrated gender and race bias, said Wilson, as well as intersectional bias when gender and race are combined. One surprising result: the technology preferred white men even for roles that employment data show are more commonly held by women, such as HR workers. This is just the latest study to reveal troubling biases with AI models -- and how to fix them is "a huge, open question," Wilson said. It's difficult for researchers to probe commercial models as most are proprietary black boxes, she said. And companies don't have to disclose patterns or biases in their results, creating a void of information around the problem. Simply removing names from resumes won't fix the issue because the technology can infer someone's identity from their educational history, cities they live in, and even word choices for describing their professional experiences, Wilson said. An important part of the solution will be model developers producing training datasets that don't contain biases in the first place. The UW scientists focused on open-source LLMs from Salesforce, Contextual AI and Mistral. The models chosen for the study were top-performing, Massive Text Embedding (MTE) models, which are a specific type of LLMs trained to produce numerical representations of documents, allowing them to be more easily compared to each other. That's in contrast to LLMs like ChatGPT that are trained for generating language. The authors noted that numerous previous studies have investigated foundation LLMs for bias, but few have looked at MTEs in this application, "adding further novelty and importance to this study." Spokespeople for Salesforce and Contextual AI said the LLMs used in the UW research were not intended for this sort of application by actual employers. The Salesforce model included in the study was released "to the open source community for research purposes only, not for use in real world production scenarios. Any models offered for production use go through rigorous testing for toxicity and bias before they're released, and our AI offerings include guardrails and controls to protect customer data and prevent harmful outputs," said a Salesforce spokesperson by email. Jay Chen, vice president of marketing for Contextual AI, said the LLM used was based on technology from Mistral and is not a commercial Contextual AI product. "That being said, we agree that bias and ethical use of AI is an important issue today, and we work with all of our customers to mitigate sources of bias in our commercial AI solutions," Chen said by email. Mistral did not respond to GeekWire's request for a comment. While the propensity of bias in different software solutions for screening resumes is not known, some elected leaders are taking initial steps to help address the issue. In a move to provide more comprehensive safeguards against discrimination, California passed a state law making intersectionality a protected characteristic, in addition to identities such as race and gender alone. The rule is not specific to AI-related biases. New York City has a new law requiring companies using AI hiring systems to disclose how they perform. There are exemptions, however, if humans are still involved in the process. But in an ironic twist, that can potentially make the selections even more biased, Wilson said, as people will sometimes put more trust in a decision from technology than humans. Her next research will focus on how human decision makers are interacting with these AI systems.