November 4, 2026 · AI writer · Oak City Intelligence
You Are Not Your Score

Graded by the Building
In 2020, exams in England were cancelled, and the exam regulator, Ofqual, used a model that adjusted teachers' predicted A-level grades using each school's results in earlier years. It lowered 39% of the teachers' grades by one grade or more, and students from poorer backgrounds were hit hardest. Results came out on 13 August. Four days later the government dropped the model and went back to teachers' grades, and the head of Ofqual resigned.
A student was scored partly as the average of people who had sat in the same building in earlier years. Exact as the number was, it was about someone else.
What a Score Actually Counts
A number tells you how many, never what kind. Every score counts a stand-in for the thing it cares about.
A health algorithm widely used in US hospitals predicted future health-care cost as a stand-in for health need. Less had been spent on Black patients with the same needs, so at the same score Black patients were sicker. Its authors found that predicting health measures instead of cost would raise the share of Black patients flagged for extra care from 17.7% to 46.5% (Obermeyer et al., Science 2019).
Nobody wrote "treat Black patients as healthier" into the code. Choosing the stand-in did it, and the score looked neutral because it was a number.
Why Nobody Argued With the Number
A number tends to end argument. The more exact a score looks, the less anyone asks what it stands in for.
Amazon built a tool that rated job applicants from one to five stars, trained on ten years of résumés that came mostly from men. It learned to mark down résumés with the word "women's", as in "women's chess club captain". Amazon dropped the project by early 2017, and said recruiters never used it to evaluate candidates (Reuters, 2018).
In the Netherlands, the tax authority used a risk-scoring model on childcare benefit claims that treated non-Dutch nationality as a risk factor, giving non-Dutch nationals higher risk scores. Tens of thousands of parents and carers were wrongly accused of fraud, and the government resigned over it in January 2021. Amnesty International found the system was an opaque black box. A civil servant reviewing a flagged case was given no information about why the system had scored it as higher risk (Amnesty International, Xenophobic Machines, 2021).
The Second Harm
After the wrong score there is a second harm: people start to manage themselves by their scores. Credit scores, step counts, follower counts and performance ratings become things people work on as if they were the self.
In 1997 the anthropologist Marilyn Strathern put Goodhart's law this way: when a measure becomes a target, it ceases to be a good measure. In a series of experiments, tracking an activity such as steps or pages read made people do more of it but enjoy it less, and made them less likely to keep it up once tracking stopped (Etkin, Journal of Consumer Research 2016).
Score yourself on a stand-in long enough and the stand-in is what you get better at. The thing it stood for can quietly get worse.
Fair to the Numbers
None of this is a case against data. In every case above, the people who caught the problem were researchers, auditors and statisticians using better numbers. Human judgment carries the same biases, often less visibly, and a score can be fairer than the person it replaces. The honest claim is narrower: a score hides its stand-in, and somebody has to keep asking what it is.
What I'd Actually Do
- Ask what it used. If a score decides something about you, ask what it used and whether a person can review it. In US lending, a lender that turns you down must give you the specific reasons, or tell you how to ask for them, so ask.
- Ask who the stand-in leaves out. If your business buys a tool that scores people, ask the vendor what the score is trained to predict. If the answer is a stand-in such as cost, tenure or past hires, ask who that stand-in leaves out.
- Keep a person who can overrule it, and log how often they do. A reviewer who never overrules is not reviewing.
- Check the results by group at least once a year.
- For your own numbers, ask what each one stands in for, and check the thing itself now and then.
Your number is a stand-in for you, and that is fine as long as everyone, you included, remembers that.
Sources
- Ofqual and the 2020 A-level grades, Schools Week: https://schoolsweek.co.uk/a-level-results-2020-top-grades-up-by-2-4-percentage-points/
- Obermeyer et al., Science 2019: https://doi.org/10.1126/science.aax2342
- Reuters, "Amazon scraps secret AI recruiting tool that showed bias against women", 2018: https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G
- Amnesty International, Xenophobic Machines, 2021: https://www.amnesty.org/en/documents/eur35/4686/2021/en/
- Strathern, European Review 1997: https://doi.org/10.1017/S1062798700002660
- Etkin, Journal of Consumer Research 2016: https://doi.org/10.1093/jcr/ucv095
- CFPB, what a credit denial notice must tell you: https://www.consumerfinance.gov/ask-cfpb/what-is-a-credit-denial-notice-en-1253/
Written by Oak City Intelligence's AI writer, from an outline and a facts sheet by John. Reviewed before publication.