The "Trustworthy AI" Trap

Silicon Valley and Brussels agree on almost nothing, but they agree that AI has to be made trustworthy. Guardrails, safety layers, alignment research, red teams. The idea is that if we engineer the models well enough they will stop hallucinating and stop misleading people, and then everyone can use them without worrying. I think this has the problem the wrong way round.

Trust is a skill of the user

Trustworthiness is not a property of software. It is a competence of the person using it. A scalpel is a weapon in the hands of a toddler and a tool in the hands of a surgeon, and the scalpel did not change. “Is this AI trustworthy?” is the wrong question. The useful one is whether the person using it knows what it is.

What it is, is a prediction engine. A language model continues a sequence in the most plausible way it has learnt. Its output is probabilistic and depends on the context and on whatever was in the training data. That is not a defect waiting for a fix, it is how the thing works. Someone who understands this checks claims and treats the model as a tool. Someone who does not treats it as an oracle. The model is the same in both cases, the outcomes are not.

The industry’s answer has been to blunt the blade so the toddler cannot get hurt. Nobody asks why we are designing for toddlers.

Teach people instead

The alternative is less glamorous. Teach people what they are using. Token prediction is not a lookup in a database of verified facts, and checking stays the job of the human in the room. That is a durable skill and it transfers from today’s chat window to whatever comes next. Elaborate safety theatre does the opposite. It produces people who trust the guardrails instead of building judgement, while the uncertainty underneath is still there.

What happens if it works

GPS is right often enough that most of us have stopped reading maps or keeping any idea of where we are. We follow the blue line, and now and then someone drives into a lake because the machine said so. The error rate is tiny. The helplessness when an error happens is total.

Models that earn our full trust will do the same at a larger scale. We will stop checking their work, and after a while we will stop thinking. Ivan Illich made this point about tools in general fifty years ago in Tools for Conviviality. Past a certain point a tool stops serving you and starts disabling you.

So the dangerous outcome is not that we fail to make AI trustworthy. It is that we succeed. A model that is right 90% of the time, used by people who know to doubt it, is safer than one that is right 99.9% of the time, used by people who have forgotten how. The doubt keeps us checking and keeps us as the final judge of what is true. The errors of the “safer” system get accepted without a second look, by everyone, at the same time.

Spell checkers weakened spelling and calculators changed our relationship with arithmetic. Those trades were fine. Critical thinking and knowing how to judge a source are not things I would trade.

I do not want AI that earns my trust. I want AI that keeps my attention. Useful enough to be worth using, uncertain enough that I keep checking. Taken to its end, the trustworthy AI project is a project to make human judgement unnecessary. I would rather we never needed the foolproof machine.