AI is increasingly being used in recruitment processes, although critics worry that existing biases may be baked into its algorithms. Now researchers claim that even without pre-existing biases, AI models can develop entirely new social biases.
In a recently published article to learnA group of researchers from Princeton University and the University of Chicago had a group of LLMs complete a recruitment game previously run with human participants. In the recruitment task, participants were asked to assign candidates to specific roles and then received feedback on whether their decisions resulted in successful recruitment. Candidates were equally likely to succeed in any job, but they all belonged to one of the four established ethnic groups: Tufa, Aima, Reku or Weki. When the human participants completed this task, the feedback they received caused them to develop a certain bias against each fictional ethnic group. For example, if they hire a Tufa as a doctor and get negative feedback, they are unlikely to hire another Tufa again. Participants retained these prejudices against the fictional ethnic group even after the game ended. When the researchers had LLMs perform this task instead of humans, they found that bias rates were higher.
“LLMs can spontaneously develop new social biases about artificial demographic groups, even in the absence of natural differences,” the researchers wrote in the study. “These results show that LLMs are not just passive mirrors of people’s social biases, but can actively create new ones from experience, raising urgent questions about how these systems shape societies over time.”
At the heart of this problem is a decision-making principle called exploration-exploitation trade-offs. The term describes the mindset we as humans go through every day when making decisions: should you choose something you’ve never tried before, so research and learn more, but if it turns out to be the wrong decision, it will cost you, or should you choose what you’ve chosen before and like? When the odds seem high, people often choose to go with what they know and trust (aka exploit) rather than bet on something new (aka explore). AI systems are less motivated to explore and tend to exhibit reward-enhancing behavior, researchers say, creating a perfect storm for stereotyping.
The researchers tested 15 models from providers such as OpenAI, Anthropic, DeepSeek, Meta, Google and Alibaba. Of all the models, OpenAI’s o3 the reasoning model most severely stratified fraudulent applicants. The researchers found that newer, larger models with more reasoning power within a model family produced more biased results.
“One simple reason is that better models make more accurate inferences about past results: Instead
With random selection, stronger LLMs may favor candidates from a group if previous assignments of similar jobs have been successful,” the researchers wrote.
More than 90% of companies use artificial intelligence in their talent acquisition process last request From ManPower Group. As AI recruiting software increasingly automates their hiring processes, job seekers are complaining about the unintended consequences of some of these opportunities, which they claim have actually hit them hard. Workday, a major software provider for human capital management, faces off class action claims that the AI-powered recruiting tools it provides to its clients are discriminatory. AI’s tendency to focus on past results has also led to claims of discrimination elsewhere in the workplace. Metaa group of employees sued the tech giant, alleging that it based its layoff decisions on an AI system that biased workers with disabilities or those who should take protected medical or family leave.
The consequences of this go far beyond just the workplace. AI systems have previously been accused of producing biased results in several use cases that affect the lives of real people. healthcare for tenant screening programs used in housing decisions.
The researchers note that LLM’s ability to quickly find patterns and its tendency to generalize underlie its ability to learn new tasks without relying on massive databases, but that’s what makes its use in real-world settings dangerous.
“The challenge is to develop interventions that selectively inhibit harmful pattern matching while preserving the constructive forms of abstraction that make LLMs powerful,” the researchers write. “Finding this balance may not be simple, but it will pave the way for fair and socially beneficial AI systems.”
