Annotator Disagreement | Tharindu Cyril Weerasooriya

Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive

Sat, 02 Dec 2023 00:00:00 +0000

We ran a massive experiment: 9 different AI content moderation systems analyzed 92 million YouTube comments about US politics. The results were shocking—different AI systems flagged wildly different content as offensive, with almost no consistency. When we asked humans to label the same content, political identity was a huge factor: Democrats and Republicans disagreed strongly on what’s offensive, especially on hot topics like abortion and gun rights. This proves that “offensiveness” isn’t a fact that AI can learn—it’s a subjective judgment shaped by values. Current content moderation practices that treat one group’s perspective as “truth” are fundamentally unfair.

Example from the Annotation

Disagreement Matters: Preserving Label Diversity by Jointly Modeling Item and Annotator Label Distributions with DisCo

Sat, 01 Jul 2023 00:00:00 +0000

Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning

Sat, 01 Jul 2023 00:00:00 +0000

Most AI training forces annotators to agree on a single “correct” label, erasing legitimate differences of opinion. But when humans disagree (like on whether a tweet is offensive), that disagreement is often meaningful—not noise. CrowdOpinion solves this by clustering similar content and predicting the full distribution of opinions people might have, rather than picking one “winner.” We tested it on Twitter, Reddit, Gab, and Facebook data, where disagreement is common. This approach ensures minority viewpoints aren’t silenced, which is crucial for building fair AI that respects diverse perspectives—especially important for content moderation and other subjective decisions.

Annotator Response Distributions as a Sampling Frame

Sat, 01 Jan 2022 00:00:00 +0000