The people behind the label: working conditions in India's data annotation industry

Every supervised model is an argument about what its labels mean, and every label was applied by someone. The conditions under which that someone worked are treated in the machine learning literature as a procurement detail.

This paper takes those conditions as its object. We report interviews with 68 annotators and 11 managers across nine firms, conducted between January and October 2025, and describe a labour process whose features are not incidental to the data it produces.

Annotators in our sample worked to hourly item quotas ranging from 180 to 900 depending on task complexity, with pay structured as a low base plus a quota-contingent component that most workers described as the part they actually lived on.

Quality was enforced through gold-standard items seeded into the queue at rates between two and six per cent. Falling below a quality threshold cost the quota bonus for the whole shift. The combination — a rate that requires speed and an audit that punishes error — was described by workers in near-identical terms across all nine firms, and it produces a specific behaviour we heard about repeatedly: when an item is genuinely ambiguous, annotators do not deliberate. They apply whatever rule they believe the gold standard encodes.

The tasks that generated the most reported difficulty were not the technically hard ones but the morally contested ones: whether a political post constitutes incitement, whether a slur is being used in reclamation, whether an image of a protest is violent content.

These are questions on which reasonable people disagree, and they were being settled at a rate of one every four to nine seconds by workers paid between ₹18,000 and ₹31,000 a month, with no channel to escalate an ambiguous item and an explicit instruction not to leave items unlabelled. The resolutions became ground truth. Every downstream fairness metric computed against that ground truth inherits decisions made under those conditions and reports them as facts about the world.

We resist the conclusion that annotation should simply be better paid, true though that is, because it leaves the epistemics untouched. A well-paid annotator resolving a genuinely contested normative question in four seconds still produces a label that a fairness audit will treat as ground truth.

Our recommendations are procedural: ambiguity channels that let a worker mark an item as contested without penalty, publication of annotation guidelines alongside datasets as a matter of course, and reporting of inter-annotator disagreement at the item level rather than as a single aggregate. Each is cheap. None is standard.