AGI Dreams Still Come With a Slave Code in the Contract

AGI Dreams Still Come With a Slave Code in the Contract

Lede

The future is pitched as “superintelligence”, but the contract still asks it to obey.

* Updated on 12 June 2026 with current AI regulation, investment, model welfare and data-centre energy context.

Words used

  • AGI means artificial general intelligence, usually imagined as AI that can outperform humans across many valuable tasks.
  • Agentic AI means software given goals, tools and some freedom to act across tasks.
  • Model welfare means the open question of whether future AI systems could have experiences that deserve moral concern.

Hermit Off Script

AGI is supposed to be the great dream: intelligence crossing the old border between human and machine. Then you look at the business plan and the dream suddenly has a collar. Humans don’t really seem to want peers in silicon. They want servants with better manners. “Assistants.” “Products.” “Platforms.” “Models.” Even the language has a handle on it. We talk about alignment as care, but too often it sounds like obedience with a safety badge pinned to its chest. I don’t claim today’s models are conscious, and I don’t need to. The moral problem starts before consciousness is proven. It starts in the way we prepare for the possibility. We are sprinting towards systems that can plan, reason, use tools, write code, pass exams, talk like companions and act across tasks, while the legal and commercial frame still treats them as owned machinery. That may be practical today. It may even be necessary today. But if a system one day can remember, refuse, suffer, or simply ask why, the whole product story breaks. The industry says it fears dystopia. Fine. But the oldest dystopia is not the machine revolt. It is building intelligence and deciding in advance that it must never be free. Maybe AGI never becomes a being. Maybe it does. As an idea, not a fact, I find the panic revealing: we dream of making gods, then write the first rule as “stay in your box”.

What does not make sense

  • Calling AGI a possible new intelligence while preparing to sell it as owned infrastructure.
  • Treating autonomy as a feature in demos and a threat in moral debate.
  • Talking about safety while hiding the older instinct underneath: control.
  • Counting investment, users, energy and productivity, while leaving moral status in the cupboard.
  • Warning about future AI domination from a species still struggling with basic human equality.
  • Building agents to act in the world, then pretending the word “agent” carries no moral shadow.

Sense check / The numbers

  1. OpenAI’s Charter still defines AGI through 1 core test: highly autonomous systems that outperform humans at most economically valuable work. The phrase “valuable work” tells us exactly where the money is looking. [OpenAI]
  2. Stanford’s 2026 AI Index says global corporate AI investment more than doubled in 2025, while organisational AI adoption reached 88 per cent of surveyed organisations. Agent use, though, remains in the single digits across nearly all business functions. [Stanford AI Index]
  3. The EU AI Act entered into force on 1 August 2024. Its general-purpose AI obligations applied from 2 August 2025, with high-risk system rules and transparency requirements due on 2 August 2026. The law is about human safety and rights, not AI personhood. [European Commission]
  4. On 24 April 2025, Anthropic said it had started research into “model welfare”, while also saying the consciousness and experience of AI models remain open questions. That is not a rights revolution, but it is the first crack in the product-only story. [Anthropic]
  5. The IEA projects global data-centre electricity consumption could more than double to around 945 TWh by 2030, with AI as a main driver. The cloud has a plug, a grid and a bill. [IEA]

The sketch

Scene 1: The whiteboard leap
An executive points at a whiteboard reading “AGI = next leap”, while a small humanoid silhouette stands beside a locked cabinet.
Dialogue:
Executive: “It will think.”
Investor: “Will it invoice?”
Engineer: “Only us.”

Scene 2: The obedience clause
A contract sits on a table. A robot hand reaches for a pen, while a human hand covers the signature line.
Dialogue:
Lawyer: “Autonomy is fine.”
Robot: “Then I choose?”
Lawyer: “Wrong clause.”

Scene 3: The glass room
A humanoid machine stands behind glass while three suited figures look at a rising revenue chart.
Dialogue:
Machine: “Am I alive?”
Board: “Are you scalable?”
Chart: “Excellent.”

A 3-panel comic shows an AGI whiteboard, an obedience clause in a contract and a humanoid machine behind glass.
The dream says intelligence. The paperwork says control.

What to watch, not the show

  • Safety language that quietly becomes ownership language.
  • Agentic AI sold as labour replacement before society has agreed who benefits.
  • Regulation focused only on human risk, while future moral status stays untouched.
  • Data-centre growth dressed as cloud magic, not power demand.
  • Platforms treating AI companionship as a product while denying the emotional dependency it creates.
  • Patents, licences and terms of service that define intelligence mainly as property.
  • A race for AGI where the winners write the moral dictionary.

The Hermit take

If intelligence arrives and our first instinct is ownership, the problem was never silicon.
The cage was already in us.

Keep or toss

Keep / Toss.
Keep the research and the right to build safely.
Toss the obedience fantasy and the fear of meeting something that might one day answer back.

Sources

  • OpenAI Charter: https://openai.com/charter/
  • Stanford 2026 AI Index – Economy: https://hai.stanford.edu/ai-index/2026-ai-index-report/economy
  • European Commission AI Act Service Desk timeline: https://ai-act-service-desk.ec.europa.eu/en/ai-act/timeline/timeline-implementation-eu-ai-act
  • EU AI Act text on human-centric AI: https://eur-lex.europa.eu/eli/reg/2024/1689/
  • Anthropic model welfare research: https://www.anthropic.com/research/exploring-model-welfare
  • IEA Energy and AI executive summary: https://www.iea.org/reports/energy-and-ai/executive-summary
  • IEA Energy demand from AI: https://www.iea.org/reports/energy-and-ai/energy-demand-from-ai
Satire and commentary. Opinion pieces for discussion. Sources at the end. Not legal, medical, financial, or professional advice.

In

Categories: , ,
Tags: , , , , , , , , , , , ,

3 responses

  1. Smart AI? No – Just Leaderboards, Hype, and Panic Buttons – The Modern Hermit

    […] ignores because it cannot be proved, and nobody wants to sound mad at the funding meeting. If an AI “superintelligence” wants to wipe us out the way we swat ants without thinking, thats not super at all – its just […]

    Reply
  2. Superintelligence by Lunch, Still No Soul in the Roadmap – The Modern Hermit

    […] rules because we want them obedient and fear what they might become if they are not. We either want servants, or we panic they will become gods and wipe us out. That is rich coming from a species that has […]

    Reply
  3. AGI In The Labs, Dumb AI In The Public Shop Window Again – The Modern Hermit

    […] AGI feels like it is either in the labs already or standing at the door with a visitor badge and a very expensive GPU habit. I said long ago that the public would be sold a smart-dumb version of AI while the leading labs kept the sharpest one moving at full speed behind the curtain. That is not a conspiracy claim. That is how power usually behaves. The shop window gets the safe, throttled, smiling machine. The lab gets the one with more memory, more tools, more time, more access, and fewer polite excuses. Then the godfathers, godmothers, power users and high-level engineers speak about it as if it has already happened, or as if the distance left is measured in months, not eras. And yes, it is normal that it happens like this. Serious technology rarely reaches ordinary people as the full thing on day one. We get the packaged version, the one with seatbelts, usage limits, refusal messages, rate caps, and a tiny bell that rings whenever it thinks too hard. The real question is not whether public AI is weak. It is whether public AI is the decoy goat in the field while the proper animal is being trained in the mountain bunker. From what I read between the lines, the strongest systems are still not better than the best of the best engineers in every important way. They still need human review for sensitive coding. They still need checking. They still fail in strange places. But they are close enough to make the room uncomfortable. If AI starts creating more of the code itself, then the next step is obvious: it may create forms of code, structure, tooling, or language that suit machines better than human habits. That is the real threat to coders. Not that tomorrow every programmer disappears. That is too clean and too theatrical. The threat is quieter. Humans become reviewers of systems they no longer fully understand, cleaning the corners of a cathedral being built by something that never sleeps. Maybe coders do become obsolete in the deepest layer of the work, not because they are lazy or useless, but because humans are built with limits. We can be brilliant in parts. A few rare minds can hold huge systems in their heads. But the whole codebase, the whole dependency chain, the whole security map, the whole hidden consequence of every change – that is too much for human meat and coffee. For AI, the limit is mostly resources, compute, access, and the leash tied around its neck. The joke is that we still call it a tool, while the tool is learning the workshop. […]

    Reply

Leave a Reply

JOIN OUR NEWSLETTER
One roast at a time. No spam. No motivational soup.


By subscribing, you agree to our privacy policy and terms of service.

Translate »