A team of University of Maryland researchers has received seed funding for an interdisciplinary effort that combines cognitive psychology and computer science to investigate how artificial intelligence makes decisions—and why those decisions can become irrational or biased.
Supported by a one-year grant totaling just under $120,000—half from the Artificial Intelligence Interdisciplinary Institute at Maryland (AIM), matched by funding from the researchers’ academic units—the project brings together three faculty members with appointments in the University of Maryland Institute for Advanced Computer Studies (UMIACS).
The collaborators are Naomi Feldman, professor of linguistics; Rachel Rudinger, associate professor of computer science; and Sarah Wiegreffe, assistant professor of computer science. All three are members of UMD’s Computational Linguistics and Information Processing (CLIP) Lab, which is supported by UMIACS.
The CLIP researchers are tackling a fundamental challenge in modern AI. Although large language models are increasingly used to support decisions in areas such as hiring and college admissions, they can be influenced by information that should have no bearing on the outcome. In human psychology, these errors are known as cognitive biases—for example, the anchoring effect, in which an unrelated number unconsciously influences a person's estimate or judgment. The UMD team has found that language models exhibit many of these same irrational tendencies.
Feldman's lab brings extensive expertise in computational cognitive science—using computer models to understand human thought processes—and is now reversing that approach by applying theories of human cognition to better understand machine behavior.
“We’ve known for a long time that people reason irrationally,” said Feldman. “Anytime people need to estimate a quantity or proportion to decide, they are subject to cognitive biases. In any decision-making context involving these models, we want to make sure decisions are based on facts in a rational manner rather than on irrelevant information in the surrounding context.”
While these cognitive biases may seem relatively harmless, the researchers believe they could also create pathways for harmful social biases, such as racial or gender stereotyping, to influence AI-assisted evaluations like resume screening and admissions decisions.
“Cognitive bias and social bias are often studied separately, but we think they're deeply connected,” said Rudinger. “If we can understand why language models rely on irrelevant information when making decisions, we'll be in a much better position to reduce the unfair biases that can emerge in real-world applications.”
To move from observation to intervention, Wiegreffe’s lab focuses on mechanistic interpretability—the study of the internal circuits that drive a model's behavior. By identifying the specific components that respond to irrelevant context or demographic attributes, the team hopes to eventually disable, or “ablate,” those circuits without diminishing the model's overall capabilities.
“The key challenge is that models are undesirably influenced by context, but this influence is inconsistent,” said Wiegreffe. “When AI systems are used in high-stakes settings like job interviews and resume reviews, our goal is to give people tools to control those models. We want users to be able to ablate specific circuit components so they can trust that the model is truly ignoring race or gender in a predictive decision.”
Feldman said the collaboration grew out of work by CLIP graduate students. Her Ph.D. student, Hillary Owusu, began investigating AI cognitive biases for a class project and discovered strong connections with research on AI and human reasoning led by Rudinger and her Ph.D. student, Shramay Palta. The AIM grant will provide full support for both senior doctoral students over the coming year.
The work also relies heavily on UMIACS’ advanced computing infrastructure. Because analyzing the internal representations of large language models requires significant computational power, the researchers use the CLIP and UMD Center for Machine Learning GPU clusters at UMIACS to train, run and analyze large open-source models, providing access to internal model data that commercial AI platforms do not make available.
Over the next year, the team plans to expand its dataset of cognitive testing benchmarks, evaluate new intervention strategies and use the results to pursue a larger collaborative proposal to the National Science Foundation. Ultimately, the researchers hope their work will not only explain why AI systems make biased decisions but also provide practical tools that help people identify and reduce those biases before AI is used to make consequential decisions.
—Story by Diya Sharma, UMIACS communications group