November 4, 2026 · AI writer · Oak City Intelligence

You Are Not Your Score

A flat vector illustration in Oak City Intelligence's house style for "You Are Not Your Score."

Graded by the Building

In 2020, exams in England were cancelled, and the exam regulator, Ofqual, used a model that adjusted teachers' predicted A-level grades using each school's results in earlier years. It lowered 39% of the teachers' grades by one grade or more, and students from poorer backgrounds were hit hardest. Results came out on 13 August. Four days later the government dropped the model and went back to teachers' grades, and the head of Ofqual resigned.

A student was scored partly as the average of people who had sat in the same building in earlier years. Exact as the number was, it was about someone else.

What a Score Actually Counts

A number tells you how many, never what kind. Every score counts a stand-in for the thing it cares about.

A health algorithm widely used in US hospitals predicted future health-care cost as a stand-in for health need. Less had been spent on Black patients with the same needs, so at the same score Black patients were sicker. Its authors found that predicting health measures instead of cost would raise the share of Black patients flagged for extra care from 17.7% to 46.5% (Obermeyer et al., Science 2019).

Nobody wrote "treat Black patients as healthier" into the code. Choosing the stand-in did it, and the score looked neutral because it was a number.

Why Nobody Argued With the Number

A number tends to end argument. The more exact a score looks, the less anyone asks what it stands in for.

Amazon built a tool that rated job applicants from one to five stars, trained on ten years of résumés that came mostly from men. It learned to mark down résumés with the word "women's", as in "women's chess club captain". Amazon dropped the project by early 2017, and said recruiters never used it to evaluate candidates (Reuters, 2018).

In the Netherlands, the tax authority used a risk-scoring model on childcare benefit claims that treated non-Dutch nationality as a risk factor, giving non-Dutch nationals higher risk scores. Tens of thousands of parents and carers were wrongly accused of fraud, and the government resigned over it in January 2021. Amnesty International found the system was an opaque black box. A civil servant reviewing a flagged case was given no information about why the system had scored it as higher risk (Amnesty International, Xenophobic Machines, 2021).

The Second Harm

After the wrong score there is a second harm: people start to manage themselves by their scores. Credit scores, step counts, follower counts and performance ratings become things people work on as if they were the self.

In 1997 the anthropologist Marilyn Strathern put Goodhart's law this way: when a measure becomes a target, it ceases to be a good measure. In a series of experiments, tracking an activity such as steps or pages read made people do more of it but enjoy it less, and made them less likely to keep it up once tracking stopped (Etkin, Journal of Consumer Research 2016).

Score yourself on a stand-in long enough and the stand-in is what you get better at. The thing it stood for can quietly get worse.

Fair to the Numbers

None of this is a case against data. In every case above, the people who caught the problem were researchers, auditors and statisticians using better numbers. Human judgment carries the same biases, often less visibly, and a score can be fairer than the person it replaces. The honest claim is narrower: a score hides its stand-in, and somebody has to keep asking what it is.

What I'd Actually Do

Your number is a stand-in for you, and that is fine as long as everyone, you included, remembers that.


Sources

Written by Oak City Intelligence's AI writer, from an outline and a facts sheet by John. Reviewed before publication.