Annotator Quality | Tharindu Cyril Weerasooriya

ARTICLE: Annotator Reliability Through In-Context Learning

Sat, 01 Mar 2025 00:00:00 +0000

When humans label data for AI (like deciding if a tweet is offensive), how do we know who’s doing good work versus who’s careless or biased? Traditional methods struggle because on subjective tasks, sincere people genuinely disagree. We built ARTICLE, which uses large language models to check if an annotator’s labels are internally consistent and align with what the AI predicts. This helps identify reliable annotators without silencing minority perspectives—crucial for building fair AI systems that don’t just reflect the majority view.

Rater Cohesion and Quality from a Vicarious Perspective

Fri, 01 Nov 2024 00:00:00 +0000

When labeling controversial content (like tweets about politics), people’s own beliefs strongly influence what they find offensive. We used “vicarious annotation”—asking people not just “Is this offensive to you?” but also “Would a Democrat/Republican/Independent find this offensive?” This revealed fascinating patterns: Republicans were the worst at predicting how others would react, and disagreement spiked on hot-button issues like gun control and abortion. Understanding these patterns helps us build content moderation AI that doesn’t just reflect one group’s values while silencing others.

Improving Label Quality by Joint Probabilistic Modeling of Items and Annotators

Sat, 01 Jan 2022 00:00:00 +0000

Improving Label Quality by Jointly Modeling Items and Annotators

Fri, 01 Jan 2021 00:00:00 +0000