Choosing Future AGI: Does unanimous agreement make a label correct?
A fictional pottery exhibit shows why reviewer agreement and comparison against an authoritative definition are different measures.
Future AGI lists annotator agreement analytics in its September 2025 release notes. Its documentation describes the statistics and access conditions. Agreement can help show whether a definition is easy to apply across reviewers. It measures whether people made the same judgment, not whether they matched the authoritative definition.
Everyone can agree and still use the wrong version
A fictional pottery exhibit labels items “glazed” or “unglazed.” The latest list says A glazed, B unglazed, C unglazed and D glazed. Both reviewers use the same old definition and mislabel the smooth item C as glazed. They get the other three right. They agree on all four, but each matches only three on the current list. These original paper cards do not measure an algorithm, a reviewer’s skill or any real person’s performance.
Keep two columns: one for agreement across reviewers on identical inputs and one for each match with the versioned reference. Cover the reference version in the exercise. Then correctness is pending, not something you can infer from 4/4 agreement. After an official update, check whether the shared misunderstanding clusters around a category instead of just looking at completion speed.
When an independent spot check is enough
Consider multi-reviewer statistics for a large annotation workload with frequent handoffs and a traceable reference set. For a few cards, a small proofing sheet can show the answers and disagreements side by side. Allow for defining and revising labels and resolving disagreements. Nothing here connects to live data, samples real people or runs an evaluation or agent.
Content note:This article is based on publicly available information and is intended as a reference for evaluating tools. Features, pricing, and related policies may change at any time; please refer to the latest official information.
Original source:docs.futureagi.com