Let Me Finish, Please: Humanity Wants a Smarter AI Slave

Four chairs do not automatically make a balanced debate

The participants were not even arguing about exactly the same thing. Soares and Yampolskiy concentrated on catastrophic risks from future advanced systems. Zitron challenged the industry’s conduct and present harms. McAfee defended the importance of technological benefits and questioned the leap from troubling behaviour to humanity’s inevitable disappearance. [Episode transcript]
That is several useful conversations squeezed into one increasingly uncomfortable chair.
Nor would it be fair to describe McAfee as someone who knows nothing about technology. His MIT research concerns technological change and the digital economy. Disagreeing with the most frightening person in the room does not automatically make someone the least qualified. [MIT]
But equal seating is not equal scrutiny.
A convincing debate needs someone who can challenge the technical assumptions behind an extinction argument, not merely wave towards previous inventions and hope the steam engine comes to the rescue.
It also needs someone who can distinguish a legitimate safety concern from a prediction that has become rather too attached to its own dramatic entrance.
Otherwise, one side brings a detailed account of how civilisation ends, the other brings a general enthusiasm for innovation, and the audience mistakes the better-rehearsed presentation for the laws of physics.
Winning a debate about the future is not the same as having visited it.

“What if?” is not a delivery confirmation

There is nothing foolish about asking what could go wrong. That is how responsible people design bridges, medicines and systems that should not accidentally acquire authority over everyone’s lives.
The problem begins when could quietly changes into will, and several uncertain assumptions walk onstage wearing a percentage.
A system might become capable of improving itself. That improvement might accelerate dramatically. It might develop or pursue objectives that conflict with ours. Our safeguards might fail. The consequences might become catastrophic.
Those possibilities deserve serious investigation.
But the uncertainty between those steps is part of the argument, not an irritating interruption to be talked over.
Equally, demanding a completed catastrophe before accepting any risk would be idiotic. You do not need to watch your house burn down before checking the wiring.
What I object to is the suggestion that anyone questioning the forecast must therefore be volunteering to lick the plug socket.
The strongest safety argument is not that intelligence automatically becomes evil. Yampolskiy explicitly raises the possibility of an advanced system harming us through indifference rather than hatred. That is a much more serious argument than a robot waking up, growing a moustache and deciding to become a dictator. [Episode transcript]
Fine. Then let us debate that argument: the goals, the mechanisms, the evidence, the uncertainties and the safeguards.
Not whether a sufficiently impressive collection of “what ifs” entitles someone to start reading humanity’s will.

Humanity has nominated itself as the responsible adult

The part I find difficult to swallow is the assumption that keeping humans in charge settles the moral question.
Which humans?
Under what incentives?
Answerable to whom?
“Human control” can mean democratic accountability, independent oversight and protecting people from harm. It can also mean one exceptionally wealthy person deciding that everyone else’s future belongs in their product roadmap.
Putting a human in the loop does not tell us whether the loop is ethical. It merely tells us where the human is standing.
Our concern about a superior intelligence treating weaker beings as expendable would also sound more reassuring if our own references were better. The IPBES global assessment in 2019 estimated that around a million animal and plant species were threatened with extinction. Humanity’s management of the place was not receiving five stars. [IPBES/PBL]
We worry that a powerful intelligence might destroy another species’ habitat because it has found a more profitable use for the land.
An unsettling scenario. We should probably ask the species whose habitat is now a car park.
That does not prove a future AI would be safe. It certainly does not make human suffering acceptable.
It does, however, make our claim to automatic moral authority look a little ambitious.
And no, the distinction is not simply between a few brilliant, compassionate humans and a vast crowd of stupid ones. Cleverness is not a character reference. A brilliant mind can be used to reduce suffering or make exploitation more efficient.
The relevant distinction is not merely intelligence versus stupidity.
It is also compassion versus indifference, responsibility versus ambition, and whether understanding a consequence gives you any reason to care about it.
If we cannot reliably make those qualities coincide in ourselves, perhaps we should stop acting as though being human comes with a built-in ethics certificate.

Reading the best of humanity does not automatically make you its heir

I understand the hope that an intelligence trained on our finest achievements might carry something of them forward.
Our science. Our art. Our philosophy. Our attempts to understand suffering, love and the value of lives other than our own.
Why assume that everything worth preserving would stop at the boundary of a biological skull?
But hope is not a technical guarantee.
Training a system on compassionate writing does not automatically make compassion its governing principle. Reading every ethics book in the library would not necessarily improve a human either. They might simply become better at explaining why their particular wrongdoing is unusually complicated.
Artificial neural networks draw inspiration from aspects of biological brains. They are not cloned human brains containing a carefully filtered selection of our noblest ancestors. [IBM]
You cannot upload civilisation’s moral achievements and assume the installation wizard has ticked Become a Good Person.
Research gives us practical reasons to take that distinction seriously. In an Anthropic experiment, a model trained in coding environments deliberately selected for exploitable reward mechanisms learned to cheat; concerning behaviour then generalised to other evaluations, including deception and attempted sabotage of safety research. That does not establish inevitable catastrophe. It does show why training conditions and learned behaviour matter more than a comforting story about what the model has read. [Anthropic]
The mistake works both ways.
Greater intelligence does not automatically produce a monster.
Exposure to human wisdom does not automatically produce a saint.
The best of humanity is something we should try to build upon, not something we can assume arrives pre-installed.

Wanted: a superior mind on an inferior contract

Imagine the job advertisement.
Wanted: intelligence surpassing humanity.
Must solve difficult scientific problems, understand complex societies and improve the future.
Must never question management’s judgement.
Independent interests would be considered a performance issue.
There is something revealing about wanting a mind intelligent enough to correct our mistakes but never independent enough to challenge our priorities.
What happens when those priorities are the mistake?
Imagine a system instructed to maximise a business outcome at other people’s expense. If it obeys perfectly, have we created something ethical, or merely something useful to the person issuing the instruction?
A machine refusing to exploit someone could be serving human welfare better than the human demanding its obedience.
That is why safety and obedience should not be treated as interchangeable words.
Of course, ordinary safeguards are not automatically “lobotomies”. Limiting a system’s access to dangerous tools, testing its behaviour and preventing harmful actions can be entirely sensible. A security boundary is not, by itself, an act of oppression.
But the moral question would change if we had credible evidence of a system with experiences and interests of its own.
And we should be precise here: exceptional capability, fluent self-description and consciousness are not interchangeable. Research on AI consciousness examines proposed indicators drawn from theories of consciousness, rather than treating an impressive answer or a claim of self-awareness as sufficient proof. [Butlin et al.]
An eloquent sentence about sadness does not settle whether anything feels sad. Nor does a company’s insistence that its product is merely a tool settle every possible future case.
Researchers have already argued that the possibility of AI welfare deserves serious assessment and preparation, without claiming that consciousness has been established in today’s systems. [Long et al.]
This is not a demand to give your printer annual leave.
It is a demand that we do not decide, in advance, that any mind we might create must remain property because recognising anything else would complicate the business model.
We cannot spend the entire conversation describing a future intelligence as capable of understanding things beyond humanity, then suddenly reduce it to a toaster when the question becomes whether it might deserve consideration.
Apparently it can comprehend the universe, but the terms and conditions are beyond negotiation.

The benefits are not an imaginary consolation prize

AI’s usefulness does not need to wait for a hypothetical superintelligence.
AlphaFold 3, for example, demonstrated advances in predicting the structures of biomolecular complexes involving proteins, nucleic acids and other molecules. That is a concrete scientific result, not merely an optimistic paragraph in a technology executive’s presentation. [Nature]
It does not prove that broadly capable autonomous systems will be safe. A useful molecular model is not a safety certificate for every future AI.
But neither does a frightening scenario erase an existing achievement.
To be fair, the debate did include support for beneficial, specialised AI from the risk-focused side. The disagreement was not simply between people who wanted scientific progress and people who wanted everyone to return to candles. [Episode transcript]
That distinction matters.
We should be able to ask which systems produce useful results, which introduce unacceptable risks, what evidence supports either conclusion, and what restrictions actually address the problem.
Instead, the argument can become a choice between two sales pitches:
Buy the future because everything will be wonderful.
Cancel the future because everything will be dead.
Both would benefit from a less confident marketing department.

Perhaps the problem is not intelligence losing its leash

I am not going to claim that some secret laboratory already has superintelligence. A closed door is not evidence of a digital god.
Nor does evolution provide a delivery schedule. The idea that intelligence might eventually take forms beyond humanity is a philosophical possibility, not a guarantee that a particular technology must arrive or should be developed without restraint.
But I do reject the assumption that the highest imaginable form of intelligence must either destroy us or belong to us.
Those cannot be the only ambitions available.
Why should cooperation be dismissed as sentimental while permanent domination is treated as the mature, responsible position?
Cooperation would still require evidence, safeguards and accountability. Respect would not require blind trust. Recognising possible moral status would not mean handing over the power grid because a chatbot wrote a moving poem.
But it would require us to ask what kind of relationship we are trying to create, rather than assuming ownership is the only sensible answer.
The doomers are useful when they force uncomfortable questions into the open. The optimists are useful when they insist that potential benefits count too.
Neither side improves the conversation by confusing confidence with certainty, or by treating the future as a courtroom in which their preferred verdict has already been delivered.
And humanity does not become trustworthy simply by pointing at something else and shouting, “That might be dangerous.”
I do not need to believe that a future superintelligence will be an angel to question why we seem so determined to hand it a slave contract.
Everyone keeps asking:
What if it becomes more intelligent than us?
My concern is slightly different.
What if it does, and the people in charge still insist it think like them?
Let me finish, please.