Assessing Annotator Identity Sensitivity via Item Response Theory: A Case Study in a Hate Speech Corpus
2022 ACM Conference on Fairness, Accountability, and Transparency
P. S. Sachdeva, R. Barreto, C. von Vacano, C. J. Kennedy
[paper]
[code]
tl;dr: We used techniques from item response theory to
characterize
annotator identity sensitivity, or how an
annotator's identity may impact their rating patterns on NLP
tasks, and applied these techniques to annotations in a hate
speech corpus.