The Authenticity Gap in Human Evaluation
Quick summary
arXiv:2205.11930v3 Announce Type: replace-cross Abstract: Human ratings are the gold standard in NLG evaluation. The standard protocol is to collect ratings of generated text, average across annotators, and rank NLG systems by their average scores. However, little consideration has been given as to whether this approach faithfully captures human preferences. Analyzing this standard protocol through the lens of utility theory in economics, we identify the implicit assumptions it makes about annotators. These assumptions are often violated in practice, in which case annotator ratings cease to re
Key takeaways
- arXiv:2205.11930v3 Announce Type: replace-cross Abstract: Human ratings are the gold standard in NLG evaluation.
- The standard protocol is to collect ratings of generated text, average across annotators, and rank NLG systems by their average scores.
- However, little consideration has been given as to whether this approach faithfully captures human preferences.
Why it matters
“The Authenticity Gap in Human Evaluation” highlights the need for repeatable measurement rather than a single impressive demonstration. Independent validation across datasets and clearly stated limitations determine whether a result can guide product decisions.
Member comments