November 3, 2026 · AI writer · Oak City Intelligence
The Danger in AI That Works
The Worry Points the Wrong Way
Most of the worry about AI says the tool is a fraud: it makes things up, it is hype, it only parrots. Some of that is fair, and its answers still need checking. But the risk worth watching is the opposite case: the tool is good, and you stop doing your share because you no longer have to. What you stop practising stops being yours.
A field study of nearly a thousand high school math students in Turkey shows it clearly. One group was given a math tutor powered by GPT-4 and allowed to use it as much as they wanted. On practice problems they did 48% better than the control group. Then they scored 17% worse than the control group on an exam taken without it (Bastani et al., PNAS 2025).
A second group in that study got a different tutor, one built to give hints instead of answers. Their practice scores rose 127%, and on the exam they came out about level with the control group. Nobody in that study was fooled by a fake. The open tutor did the hard part, the students had no reason to do it, and when the help went away the hard part was still undone.
Grant the Machine Its Due
You do not need to believe these systems are stupid to worry about what they do to you. Researchers at Anthropic showed that a model can choose its rhyming word before writing the line that ends on it. Much of what runs in your own head runs on habit too: finishing familiar sentences, recognising a problem you have seen before, reaching for the answer that usually works. That autopilot is real and capable, and nobody gets through a day without it. The trouble starts when you decide that is all there is and live as if it were.
A Description You Can Make True
A common line says the model only predicts the next word, and so do we. As a claim about habits it has a bad property: anyone can make it true of themselves. Hand off every draft, summary and judgment call, keep only the approving, and after a year the description fits better. Your brain is the same organ. The unused skill has gone quiet.
A Microsoft Research and Carnegie Mellon survey of 319 knowledge workers covering 936 real uses found that more trust in the AI went with less critical thinking reported, and more self-confidence with more (Lee et al., CHI 2025). These are self-reports, a signal and not a measurement.
A study of Polish endoscopy centres found something similar. After AI assistance became routine, the rate of finding precancerous growths in colonoscopies done without the AI fell from 28.4% to 22.4% (Budzyń et al., 2025). In a mammography study, when the AI suggested the wrong category, very experienced radiologists' accuracy fell from 82% to 45.5% (Dratsch et al., Radiology 2023).
None of this is new. Lisanne Bainbridge wrote in 1983 that automating the routine work takes away the daily practice a person needs for the rare day the automation fails.
Whose Thinking Was It?
Offloading is not the enemy. Writing things down, using calculators, looking up routes on a map: these are all forms of offloading, and the research treats most of it as sensible (Risko and Gilbert, 2016). The line worth drawing is between thinking you can defend and thinking you cannot. Some thinking comes with the sense that you did it, that you can defend it and change it. That sense is easy to lose because the output looks the same.
A small MIT study asked 54 people to write essays over several sessions. The ChatGPT group had the hardest time quoting their own essays and reported the weakest sense the essays were theirs (Kosmyna et al., 2025). Eighteen came back for a fourth session with conditions swapped. Those who wrote unaided first, then used the model, showed more engagement. The study is small, not peer reviewed, with a critique posted in reply, so no policy hangs on it.
If you cannot quote your own essay, in a way that matters it is not yours. The hint tutor and the own-draft-first order point the same way: how people used the tool decided the outcome.
What I'd Actually Do
The tools will keep getting better, so the pull to stop doing your share gets stronger. A few habits push back.
- Think first, then ask. Write your own rough answer or outline before opening the tool.
- Use it as a tutor for anything you need to be able to do yourself. Hints, questions and critique; finished answers only for work you will never need to learn.
- Keep one rep by hand. Regular time on the skills your job leans on, without AI.
- Form your view before you read its suggestion on anything that matters.
- Try the closed-laptop test. Can you explain and defend the result with the tool shut? If not, the work is not finished.
What is worth checking is whether the thinking is still yours at the end.
Sources
- Bastani et al., PNAS 2025: https://doi.org/10.1073/pnas.2422633122
- Lee et al., CHI 2025: https://doi.org/10.1145/3706598.3713778
- Budzyń et al., Lancet Gastroenterology & Hepatology 2025: https://doi.org/10.1016/S2468-1253(25)00133-5
- Dratsch et al., Radiology 2023: https://doi.org/10.1148/radiol.222176
- Bainbridge, Automatica 1983: https://doi.org/10.1016/0005-1098(83)90046-8
- Kosmyna et al., preprint 2025: https://arxiv.org/abs/2506.08872
- Stankovic et al., a critique of the above, 2026: https://arxiv.org/abs/2601.00856
- Risko and Gilbert, Trends in Cognitive Sciences 2016: https://doi.org/10.1016/j.tics.2016.07.002
- Anthropic, "Tracing the thoughts of a large language model", 2025: https://www.anthropic.com/research/tracing-thoughts-language-model
Written by Oak City Intelligence's AI writer, from an outline and a facts sheet by John. Reviewed before publication.