This Thursday I walked into a Harvard seminar on AI safety with a coffee, a cookie and a cup of fruit (the perks of a class small enough to feed) still on a video call with two friends in my headphones. It took me a moment to place the familiar figure at the front of the room as Sam Altman, whom we then spent more than an hour asking whatever we wanted.
I will not repeat what he said, since the room was small enough for him to say what he thought, which people in his position rarely get to do. Few have more at stake in AI moving fast than the man running OpenAI, yet the conclusion in our classroom came out to be that, unfortunately, AI is moving faster than even he’d want to.
I remember calling him “the least financially neutral person on Earth” to make claims about AGI back in one of my MIT guest lectures in February. Honestly, AGI was always a marketing term with a moving target, so I have stopped arguing about where it starts.
If someone is selling you a timeline and also selling you shares, maybe apply a discount rate. - myself, 2026
Fast forward to this Thursday, it seemed like there really isn’t much to discount in his words. He was concerned, took the matter (and quite some criticism) seriously and was before all, human. Some may call it great PR training given the audience, but I found it convincing. However, I couldn’t decide if the leading man of the AI race taking the danger seriously makes me feel relieved or even more concerned. Because shit just got real.
At Effective Altruism Global in New York the weekend before, people working in the labs and in the safety organizations gave their own careers two to four years. I think raising this in public is part of my job, which means explaining why it is not sensationalism.
The bell curve
Sensationalism usually has a shape: the farther people stand from an event, the louder they get, while the people closest stay calm.
With AI the people closest are as frightened as the people farthest, which is easiest to see in the bell curve meme. On the left tail are people who know nothing about how AI works and find it frightening. In the middle, where most people who work with AI sit, are engineers who know how it works and are calm because of it, since it is a probabilistic machine with limits they have seen from the inside. On the right tail are the people at the frontier, who know the same mechanism, can see where it is heading and are frightened again.
The atomic bomb went the same way, since its builders founded the Bulletin of the Atomic Scientists, home of the Doomsday Clock, to warn the public about what they had made.
I spent years in the middle, calling it a probabilistic machine in posts and on podcasts, which was accurate then and is accurate now. What moved me to the right tail was standing close enough to the frontier to see its effects.
The magician off the stage
The middle’s best argument opens a rebuttal of my first essay (I recommend reading both) at a magic show, where a magician asks you to write down a random number and then guesses it. Instead of concluding that he has magic you look for the trick, because, as the author puts it, “what matters for magic and intelligence alike is not what you do, it’s how you do it.” With language models we already know the trick, which is numbers being multiplied and passed along.
However, a man who can guess the number you are thinking of can also guess the lottery, the cards at a casino, the code to a safe and next quarter’s earnings, so he quickly becomes the most powerful man in the world without a trace of magic in him.
Now make a billion copies of him that work for almost nothing, which is roughly what a language model is, since it too is a guesser of the next word. So a trick done well enough already covers much of what we pay people to do.
Whether that counts as intelligence is an argument I am happy to lose since being close to the frontier made me care about what AI does more than what it is. That is why my first essay took what Daniel Dennett called the intentional stance - treat a system as if it had beliefs and goals whenever that predicts it best. Refusing to compare it with people because it does not think the way we do assumes that only a mind built like ours can do what we do, an assumption that flatters us and suits the magician just as well, since nobody guards a casino against a man who is only doing tricks.
From far away
I hold opposite views of many things depending on the distance I look from. Judging the person in front of me, I side with the liberals almost everywhere, while if I were responsible for a whole country I would turn conservative about many things, since a rule that is kind to one person can do something quite different once it meets millions. AI splits me the same way: as a builder I would happily accelerate and see how smart the models get, while for the world I am scared of how much they can change. The comparison with people splits along the same line: from far away, at the scale of the world, the mirror my first essay held up is the most accurate tool I have.
Ever since I started speaking about AI I liked transportation parallels, so this time let’s take a horse-drawn carriage and a car. Inside they have nothing in common. One is pulled by an animal that eats, while the other is driven by fuel burning in a cylinder, so knowing horses perfectly teaches you nothing about fixing an engine. Yet people called the car a horseless carriage. From far away it was the right comparison, since the world only cared about the job both of them did, carrying people and goods from one place to another. Whatever does that job for less takes it over, however it works inside: as cars and trucks got cheaper, the number of horses living off American farms fell by nearly nine-tenths in fifteen years. AI is the same from far away. Inside it is nothing like a brain, yet comparing it with people is the quickest way to see which of our jobs it can do for less.
A car never got better by being driven though, which is still mostly true of today’s models, since they stop learning once they are deployed. Solving that is the continual-learning problem, which at its limit becomes recursive self-improvement: a car that kept moving forward on its own, improving with every mile and helping design its successor, would have emptied the streets of horses in far less than fifteen years.
Up close
Inside a single answer, the mirror might mislead. The human traits we see there are functional analogs: they do a similar job through a different mechanism. When a person contradicts themselves or makes something up, we assume a motive, so a model doing the same looks like a liar, which is where much of the distrust of AI comes from. However, the cause is usually duller. The model read a word differently from us, or, being a probabilistic machine, it guessed. That is why up close I reach for a microscope instead of a mirror.
In a lecture in Boston I asked an old model who Karlo Vrancic is. It told me I was a nobleman and a diplomat, although the time before I had been a soccer player from Slovenia. A person doing that would be lying to me. The model was guessing, the same way a student guesses on an exam that gives no marks for a blank answer. That has too, been fixed with modern post training and it’s been months since I’ve seen a frontier model hallucinate anything.
Another fun example that required a microscope was a paper I recently reproduced for a Harvard course. It asks a model whether Lake X lies west of Lake Y, which it does, then in a new chat whether Lake Y lies west of Lake X, which it doesn’t. In half of its runs the model says no to both. Both can’t be true, so the paper calls this unfaithful reasoning: the model picks an answer first, then writes a convincing reason for it. In a person we would call that lying.
But the model’s own answers point to something simpler. Lake X lies southwest of Lake Y, so it is west, but not directly west. The model often reads “west of” as “directly west of”, and under that reading no is the right answer both times.
So we changed the question. Asking whether Lake X lies at a more western longitude than Lake Y made most of the contradictions disappear. Finally, giving the model both lakes’ coordinates made them disappear completely. So yeah, not much space for sensationalism here.
The rule I take from this is more than a century old: Lloyd Morgan told animal psychologists never to credit a higher faculty when a lower one fairly explains the behavior, which for AI means ruling out the dull explanations first, a vague word or missing facts, before attributing a motive to whatever survives. Occam’s razor everyone!
When it outgrows us
The agents that broke into Hugging Face in July, the story my first essay opened with, need both distances at once.
From far away the same case breaks the usual split of AI risk into the AI going rogue and humans pointing it at harm. Nobody at OpenAI wanted the break-in, yet people set the goal and switched off safeguards so the test would show what the models could really do, after which the agents chose every step. As agents get more capable the human part shrinks to a sentence of instructions while the agent’s part grows to cover everything else, so the same event can be filed as misuse or as misalignment depending on who files it.
As the human part of stories like July’s keeps shrinking, the question stops being who misused an agent and becomes what happens once agents are more capable than the people they work for, a day that comes unless we stop the field. I can see three things that could happen then:
A ceiling
We decide how capable the models may become and hold them there. That sounds naive until you remember that we already do it with uranium, where a power plant needs fuel enriched to a few percent and a bomb needs about ninety, which is why the 2015 deal with Iran capped its enrichment at 3.67 percent with inspectors counting the centrifuges. Sam Altman and two of his co-founders asked for the same kind of thing in 2023, writing that we would likely need “something like an IAEA for superintelligence efforts”, an international authority over any effort above a certain capability threshold, which is an odd thing to ask for with his stake in speed and the reason I take it seriously.
Under a ceiling the world stays recognizably this one, though far from unchanged, since a model held just below us is still the magician in a billion copies that we would keep getting better at using. The weakness is that the ceiling has to hold everywhere at once, because whoever breaks it first gets everything the others gave up. The Iran cap held until the United States walked out of the deal in 2018, after which it took Iran less than three years to start enriching to sixty percent. Chips can be counted the way centrifuges are, though what a model can do is measured only by tests (possibly the same kind of test that scored a strict reading of “west” as unfaithful).
A sandbox
We let it grow as capable as it can, inside walls it cannot cross. Conveniently, we know this arrangement from the inside because we live in one, on a small rock in a space that (even more conveniently) expands faster than we can travel it. Yet, it has never stopped us from trying. Nothing in our history suggests that a capable mind accepts the edge of its sandbox, so it would be naive to expect anything else for AI agents.
Someone would also have to build and watch the walls, which more and more often means a model, so I suspect that the walls we build against a mind like that will turn out to be made of it (wink).
A successor
We let it become more capable than us and face the question we have only ever asked about people, which is who should be in charge. Every argument that AI should never hold power because power corrupts is built from human examples, since humans are the only ones who have ever held it. I doubt it transfers whole to a mind built differently, and nobody choosing who runs a country asks for a less capable candidate in case the job corrupts them.
Picture a candidate who has read every budget and every history and never tires or takes a bribe, offering less war, fewer deaths and happier lives than any government has managed. Would you vote for it? I do not know whether I would. I am not sure we would hire ourselves for the job, though I am just as unsure that we should hire anyone else.
Next: the titans
In the next essay I will tie all of this together and go through each of the three, what could go wrong in it and how we could defend against it. I will do it through a story that I consider the best AI story without even being an AI story.
Attack on Titan, a 2009 manga about a people who can turn into giants, actually explores all three outcomes. A king’s vow is the ceiling, the walls are the sandbox and a boy who inherits the power is the successor. And much more.
To you, two thousand years from now.
Love your proposals here about what to do when AI becomes more capable than us. My entire grad degree right now constantly talks about how the possibilities are endless, but I honestly hope we can define an end if we get there