Deleting an abusive comment or filtering out spam is one part of online moderation. Guiding a struggling community member or keeping an expert discussion group friendly, fair and nuanced demands much more.
A University of Maryland study finds that large language models (LLMs) could help volunteer moderators screen distressing content and retrieve information to guide decisions. But responsibilities such as building trust, interpreting subtle social cues and maintaining a community’s identity remain harder to automate.
The paper, published earlier this year in ACM Transactions on Computer-Human Interaction, draws on interviews with moderators of r/AskHistorians and several law-related subreddits, including r/LegalAdvice, r/BestOfLegalAdvice, r/legaladvicecanada, and r/legaladviceofftopic. By comparing their needs with AI capabilities documented in prior research, the team developed guidance for tools that support moderators while leaving final authority with people. The study offers qualitative insights and design recommendations rather than results from testing a deployed AI system.
The research was led by Lovely-Frances Domingo, a sixth-year Ph.D. student in the College of Information, alongside her advisers Katie Shilton, a professor in the College of Information, and Hal Daumé III, a professor of computer science. Both advisers also hold appointments in the University of Maryland Institute for Advanced Computer Studies (UMIACS).
The researchers found that moderators organize their work around three core values: care, wisdom and civics. Care involves shielding users from harm and assessing sensitive content. Wisdom draws on subject-matter expertise to judge borderline posts, evaluate contested claims and interpret intent. Civics centers on building trust, shaping culture and maintaining a community’s identity.
Domingo was particularly struck by the volume of invisible teamwork within the moderation ranks. Automated tools that focus on individual posts, she noted, overlook much of the collaboration that keeps communities running.
“Community moderators perform an immense amount of unseen labor,” Domingo said. “Because so much of that work happens behind the scenes, people assume it doesn't exist—but these volunteers are constantly working to keep interactions safe and build healthy communities.”
The team identified several ways language models could support that work. Screening potentially traumatic content could help reduce moderators’ exposure to distressing material. Retrieving relevant context—including past moderation decisions—could help them evaluate difficult cases.
“An LLM could analyze past decisions on a specific topic and pull them all together for a moderator,” Domingo said. “By bringing those precedents to the surface, it helps moderators quickly see how similar cases were handled in the past.”
The multidisciplinary research team also included Michelle Mazurek, a professor of computer science with an appointment in UMIACS and director of the Maryland Cybersecurity Center; Yang (Trista) Cao, an applied scientist at Amazon who received her Ph.D. in computer science from UMD in 2024; and Sarah Gilbert, research director of the Citizens and Technology Lab at Cornell University. Mazurek, Shilton and Daumé are also members of the Institute for Trustworthy AI in Law & Society, which Daumé directs.
Domingo credits her advisers with grounding the study in human values. Shilton’s research on ethics in design helped frame everyday moderation practices, while Daumé’s work on trustworthy AI helped the team assess how models could serve moderators’ needs.
Other responsibilities require a deeper understanding of people and their circumstances. Domingo described an instance on r/BestOfLegalAdvice in which a member asked seemingly routine legal questions, but moderators recognized signs of a possible suicide crisis and connected the user with crisis support.
Moderators also use warmth, tone and humor to enforce standards while keeping members engaged. These judgments depend on relationships and an understanding of community norms that extend beyond the wording of a single post. The moderators interviewed expressed skepticism that automated AI tools could handle these complex interactions.
The study recommends that AI tools communicate uncertainty and leave decisions to human moderators. Recognizing the limits of an answer—and knowing when to seek more information—is an ability Domingo sees as a weakness of current language models.
“They don’t know that they don’t know,” Domingo said. “They have no sense of whether they are right or wrong. Humans are far better at recognizing when something feels off, asking the right questions, and knowing where to look when they lack an answer.”
—Story by Melissa Brachfeld, UMIACS communications group