Intelligence Beyond the Human Measure

This is a sequel to my previous post: ‘We’re Still Pumping Water from Mines: path to superintelligence.’

In the margins of his notebooks, around 1490, Leonardo da Vinci drew a machine for flying. It had a frame for a man to lie in, a system of pulleys and levers, and two great wings that the pilot would beat against the air by working his arms and legs. It was a beautiful piece of engineering and it could never have flown. Leonardo, like nearly everyone who tried to leave the ground for the next four hundred years, had made a single quiet assumption that doomed the entire project before the first lever was cut: that to fly is to do what a bird does.

This is the assumption of the ornithopter, the flapping-wing machine, and it is one of the most persistent dead ends in the history of technology. Generation after generation of would-be aviators strapped on wings and threw themselves off towers, because the bird was the only example of flight anyone had ever seen, and so flight was the bird. The thing to be achieved and the one creature that achieved it had fused into a single idea. You could not think about the first without picturing the second.

Flight became possible only when that fusion came apart. In the early nineteenth century George Cayley did something stranger and more difficult than building a better wing: he asked what flight actually is, underneath the bird. He decomposed it into forces (lift, drag, thrust, weight) and noticed that the bird solves all four at once, with the same flapping motion, but that there is no law of nature requiring them to be solved together. You could separate lift from propulsion. You could get your lift from a fixed, curved surface that does not move at all, and your thrust from somewhere else entirely. A century later the Wright brothers added the last missing piece, control, and yes, they studied birds to find it: the way a wing twists to bank into a turn. But they did not flap. The airplane that finally flew was not a mechanical bird. It was a different kind of thing that happened to share one property, staying aloft, with birds.

And here is the part that matters for us. Once we had airplanes, nobody asked whether they had achieved bird-level flight. The question evaporated. It would be a strange thing to ask. A 747 outclimbs, outruns, and outlifts every bird that has ever existed by orders of magnitude, and yet it cannot land on a branch, cannot dart through a forest canopy, cannot fold itself up and disappear into a hedge, cannot heal a torn wing, cannot make another 747. We do not say the jet is “above” the sparrow, or “below” it. We say it is a different solution to a problem we had been mis-stating for four hundred years. The bird was never the target. It was just the only clue we had.

I wrote in the previous post that the large language model is the steam engine of artificial intelligence: a real breakthrough, magnificent within its niche, and almost certainly a stepping stone rather than the destination. That essay was about architecture: we built systems that know about the world but cannot act in it, that compress human knowledge brilliantly but cannot learn from experience, and I argued the road forward runs through experience, not just scale. I still believe that. But I have come to see this is not the only confusion. Using fixed wings was an engineering decision, not a discovery about flight: we built them because we could not build working flapping wings, not because we had finally understood the bird. The attention mechanism is the same kind of move. It works, but nobody claims it is how a human reads a sentence. The difference is that with the airplane we never mistook the workaround for the real thing, while with the language model we wrap the world’s knowledge in prediction and call the result intelligence. And this engineering sits on top of a deeper one, and the deeper one is about our ruler. We are still building ornithopters. We are still measuring intelligence in units of us.

Intelligence as Spikes

What really is intelligence? There is no settled answer, only a succession of definitions that each era rewrites. Ours is special because for the first time in history we have more than one kind of mind to look at. Every account we inherited was drawn from a single example, the human; the kind of machine intelligence currently deployed displays some features of intelligence in abundance while lacking others entirely, which forces the old definitions open again. But beneath the revisions one claim holds, and it is the one that matters here: intelligence is not a scalar.

It is not an index number (the kind you find on the Artificial Analysis website). It is not a single quantity you have more or less of, a height you can be measured against a doorframe. It is a profile: a shape, a vector with many components, spiky and uneven, strong along some dimensions and weak along others, and never, in any real system, equally powerful across all of them at once.

Psychology has been circling this idea for decades without quite landing on it. In 1983 Howard Gardner published Frames of Mind, arguing that there is no single intelligence to be scored but a family of them: linguistic, logical-mathematical, musical, spatial, bodily-kinesthetic, interpersonal, intrapersonal. The theory was a deliberate rebuttal to the IQ tradition and its dream of one number that sums up a mind. Psychometricians have pushed back hard ever since, and they are not wrong to. The abilities Gardner separated tend to correlate, the brain does not file them in tidy modules, and the strict version of his claim has little hard evidence behind it. But take Gardner not as settled neuroscience but as the right instinct aimed slightly wrong. He saw that intelligence is a profile and not a scalar, and on that I believe he was right. His error was to draw the axes from inside a single species, as if the only intelligences worth naming were the ones a human school might hope to cultivate.

Look at how intelligence is actually built into living tissue and the scalar picture falls apart completely. There is no general-purpose thinking organ sitting above the fray. Cognition is specialized and localized, wired directly into the sensory and motor systems it grew out of, and shot through with emotion and drive. The parts of you that reason are not floating free of the parts that fear, want, and flinch; they are extensions of them. Intelligence in a living creature is not a pure faculty bolted onto an animal. It is the animal’s machinery for finding food, avoiding harm, and finding a mate, gradually elaborated. It exists to serve those older imperatives, and it carries their fingerprints everywhere.

Max Bennett makes this case with great force in A Brief History of Intelligence, which tells the story of mind as a sequence of five evolutionary breakthroughs, each one a solution to a concrete survival problem. First came steering, the simple approach-or-avoid valence of the earliest wriggling animals. Then reinforcement learning, the dopamine-driven trial and error of the first vertebrates. Then simulation, the mammalian ability to run a model of the world and rehearse an action before taking it. Then mentalizing, the primate trick of modeling other minds. And last, language. At every stage, intelligence appears because it pays: it earns its keep in food found, predators dodged, offspring protected. No animal ever evolved a capacity to think for the pleasure of thinking, or to know for the sake of knowing. Cognition is metabolically expensive, and biology does not pay for what does not pay it back. Whatever else intelligence is, it began as survival gear, and its shape was bent at every turn by the particular problems a particular body faced.

A human being has a particular spike profile, and it is a strange and specific one. We are extraordinary at a tight cluster of things: learning a model of the physical and social world from astonishingly little data, transferring it to situations we have never seen, reading other minds, compressing a lifetime of experience into a story we can hand to a child in an afternoon. We are mediocre or worse at others: we cannot hold sixty things in working memory, cannot multiply large numbers in our heads, and forget most of what we encounter. Other limits are not failures of intelligence at all but facts of the substrate: we cannot copy ourselves, cannot run faster by adding hardware, and we die. This profile is not “general intelligence” in some pure, dimensionless sense. It is the shape that intelligence took when it had to fit inside a bipedal mammal that needed to find food, avoid predators, raise slow-maturing young, and coordinate with a few hundred others on the African savanna. Our intelligence is tightly coupled to a biological substrate and to a set of senses and motor faculties built for that niche. It is, in the most literal sense, a special case. Survival gear for one species.

When you see it this way, the squirrel from my first essay stops being a humbling counterexample and becomes a vivid illustration. The squirrel is held up to shame the LLM: look, this creature has the experiential intelligence our systems lack. That was right as far as it went. But the deeper point is that the squirrel does not have less intelligence than the LLM, sitting lower on the same ladder. It has a different spike. It is superb at a real-time, embodied, single-shot learning problem that no language model can touch, and hopeless at the cross-domain compression that the language model performs effortlessly. Two different shapes. Neither is a rung beneath the other. And the artificial systems we are now building have their own emerging shape, already superhuman along the axes of recall, breadth, speed, parallelism, and tireless consistency, already subhuman along the axes of grounded continual adaptation that the squirrel owns. A jagged, alien, non-human profile.

Intelligence as Height

If intelligence is a profile, the words we use for its future ought to name shapes. Instead they name an altitude. By 2026, most people have heard the word AGI: artificial general intelligence. The label is recent; the thing it points at is old. Turing was already asking in 1950 whether a machine could do what we mean by thinking, and for the next half-century the field called that goal Strong AI. The phrase comes from the philosopher John Searle, who in 1980 drew a line between weak AI, a machine that merely simulates a mind, and strong AI, a machine that genuinely has one, with real understanding and mental states of its own. I prefer the older phrase, and not out of nostalgia. The trouble with “AGI” is the middle letter. General as opposed to specific? And if general means average, do we mean the whole spectrum between a half-wit and Einstein, or between a flatworm and a person, or something wider still that takes in minds no animal has ever had? The word pretends to name a clear target while quietly burying the fact that we have never agreed where the target sits. “Strong AI” at least made a claim you could argue with. “AGI” hands you a ruler with no marks on it.

The history of the term is quick and faintly comic. The physicist Mark Gubrud used “artificial general intelligence” in 1997, in a paper about nanotechnology and future warfare, and nobody noticed. It surfaced again around 2002 in a conversation between Ben Goertzel and Shane Legg (the same Shane Legg who would go on to co-found DeepMind and co-author the report I am about to discuss), and Goertzel made it the title of a 2007 essay collection and a conference series. From there it leaked into industry, where it was promptly bent to fit a balance sheet. OpenAI’s charter defines AGI as “highly autonomous systems that outperform humans at most economically valuable work,” which is not a definition of intelligence at all but a definition of a workforce.

While the term itself is suspiciously vague, its adoption by the masses has been quick. The public has sorted itself into two camps that shout past each other: one says AGI is already here (a claim they have to reissue every three months), just look at what the models can do; the other says it is decades off or impossible, just look at what they still cannot. Both are arguing about where a line falls. Neither stops to ask whether the line is real.

In June of 2026 a group at Google DeepMind (Genewein, Legg, Hutter, and others) published a long report called From AGI to ASI, and it is, by some distance, the most serious attempt yet to chart the terrain beyond human-level AI. It is not a hype document. It is careful, formally grounded, and honest about its own uncertainty. It characterizes superintelligence, lays out four technological pathways toward it, catalogs the frictions that might slow each one, and concludes, wisely, that there may be no single dramatic step change at all.

It is worth being concrete about what the report actually claims, because the claims are more careful than the title. It pins AGI to a specific waterline: a system roughly as capable as a single median human across most cognitive tasks. ASI it sets far higher, at a system more capable than large organizations of people, something like the combined cognitive output of tens of thousands of top experts working together for a decade. Between those two markers it lays four pathways. The first is simply scaling the systems we already have. The second is a paradigm shift to an architecture we do not yet possess. The third is recursive self-improvement, a system rewriting itself to become better at rewriting itself. The fourth is the emerging behavior from sheer numbers, a hundred million AGI instances running in parallel and sharing what they learn, adding up to something no single one of them is. Against these the report sets six frictions that could stall the climb: the data wall, runaway demand for compute and energy, the limits of the current neural paradigm, the rising difficulty of research itself, an “abstraction barrier,” and the plain possibility that we deliberately pump the brakes. Its headline conclusion is admirably undramatic: probably no single overnight intelligence explosion, but a long, uneven cascade of transformations rolling across science and the economy.

Reading it, my first complaint is with the title. From AGI to ASI takes a starting line for granted. It assumes we can achieve AGI, or are already standing at it, and so are close enough to plan the next leg of the trip. That assumption is not idle; it is something of a house position at DeepMind, whose founding mission was to “solve intelligence” and then use it to solve everything else. But it is exactly the part most in doubt. Whether the road we are on actually arrives at anything worth calling general intelligence, rather than at a very wide and very shallow imitation of it, is the open question of the decade, not a milestone we can plant a flag on and measure forward from.

And look at the four pathways again: not one of them begins at AGI. Scaling, new paradigms, self-improvement, multiplying agents: these are all things we are doing right now, today, in the hope of reaching AGI in the first place. None of them switches on at some threshold. They are the very mechanisms carrying us toward the line, and they keep running straight through it without noticing it is there. By the report’s own logic the honest title is closer to “the path to AGI, and perhaps beyond.” The mechanisms it describes are continuous across the so-called AGI threshold rather than igniting at it. The authors would have a fair reply: they bracket the road to AGI deliberately, because it is the crowded part of the field, and they study the stretch above it because it is neglected. Fair enough. But a boundary you cross without feeling it is a strange place to anchor a map.

The remarkable thing is that the same report concedes the profile I have just described, in a footnote-like remark it never fully metabolizes. Capability profiles of real systems, it notes, may be “jagged” with respect to human intelligence: spiky, uneven, superhuman here and subhuman there at the same moment. That is the spike profile, conceded in passing. And the report’s own table of “advantages of digital intelligence” (lossless copying, substrate independence, the ability to run faster by spending more compute, the ability to share learned experience between instances at high bandwidth) is nothing other than an inventory of the spikes that point in directions ours never could, because they are forbidden to biology. The paper has, in two different places, written down the fact that intelligence is plural and substrate-relative. It just does not let that fact reorganize the map. The ladder survives in the prose even though the mathematics underneath it is a landscape.

The report lays out the four pathways as four routes up a single mountain, toward the summit it calls ASI. I want to read them as something else. They are not four routes to one peak. They are four engineering processes, visible to us, that help grow spikes. What we will witness is not an ascent. The picture I reach for comes from evolutionary biology, where morphospace names the space of all possible organism forms, most of which no creature has ever occupied. I am borrowing the word as a metaphor for the landscape of possible minds, almost entirely empty until now except for the small crowded corner where biology built its variations on one theme, and only beginning to populate. The transformations come one after another not because a single intelligence is rising through levels, but because different capacities are crossing different thresholds of usefulness at different times, each one remaking some corner of the world as it lands.

In Search of a Mind Without Us

There is a famous question in philosophy: what is it like to be a bat? Thomas Nagel’s point was that a bat’s experience, navigating the dark by bouncing sound off the world, is not a dimmer or brighter version of ours. It is organized differently, around a sense we do not have, and there may be no way to climb from our world into its world by addition or subtraction. The biologist Jakob von Uexküll had a word for this: Umwelt, the world-as-experienced that each kind of creature is sealed inside, built out of the senses and concerns that creature actually has. The tick’s Umwelt is made of warmth, butyric acid, and the feel of hair. Ours is made of faces, words, distances, and the future. They are not the same world seen from different heights. They are different worlds.

Ask what it is like to be a bat at human level and you have asked a malformed question. There is no human level of echolocation. The phrase does not refer to anything. And this, exactly this, is what we are doing when we ask whether an AI system has reached human-level intelligence, or when we will surpass it, or how far above us a superintelligence will rise. We are asking what altitude the jet flies at, in birds. The artificial systems now coming into being have their own Umwelt, assembled from a substrate that is not flesh, not bounded by one body, not metered by one lifetime, not gated by the low-bandwidth bottleneck that forces us to compress everything into language before we can share it. Marshall McLuhan, who is somewhere in the back of all of this, would have said it plainly: the medium shapes the message, and the substrate shapes the mind. A mind grown in silicon, copied at will, run at variable speed, and fed on streams no human nervous system could survive will not be a faster human. It will be the cognitive equivalent of the airplane, sharing the single abstract property we call “intelligence” with us, and almost nothing else.

If a mind is sealed inside its Umwelt, then intelligence is sealed in there with it, and the dream of measuring intelligence in some pure, world-independent way runs into trouble at once. This has not stopped anyone from trying. The effort to define intelligence without quietly meaning human intelligence is nearly as old as the computer, and its history is worth telling, because it keeps failing in the same instructive way.

It begins with Turing. In 1950 he proposed to sidestep the question of what intelligence is and ask instead what it does: put a machine and a person behind a curtain, let a human ask them anything, and if the judge cannot tell which is which, call the machine intelligent. It was a brilliant dodge, and look at what it smuggled in. The measure of the machine is a human examiner’s failure to tell it apart from a human. The first modern definition of machine intelligence makes us the curtain, the question, and the answer all at once. It is the bird enthroned as the test.

A second tradition ran the opposite way, toward a single number. Out of the psychology of testing came Spearman’s g, the general factor an IQ score is meant to capture, the dream that one quantity sums up a mind. This is the doorframe again. It works, after a fashion, but only by assuming that the one axis along which humans happen to spread out is the axis of intelligence as such. It measures the distance between us and us, and calls the ruler universal.

The boldest attempt is the one the report leans on, and it set out to escape the human altogether. Instead of asking whether a machine can pass for one of us, or where it lands on our scale, it asked how well a mind does across every possible world, not merely ours. This is the Legg–Hutter definition of intelligence, built by Shane Legg, who is among the report’s authors, and Marcus Hutter, on foundations Ray Solomonoff laid in the 1960s. Its purest form is a theoretical agent called AIXI, written down around the year 2000: a perfect learner that, dropped into any world at all, would work out the rules and act to get the best outcome. AIXI cannot be built, only defined, and it is often called the ceiling of intelligence, the best possible mind that everything real only approximates.

To score a mind on how well it does across “every possible world,” you must first decide which worlds count, and how much. Worlds like ours, with solid objects and cause and effect and patterns that reward learning? Or worlds of pure noise, where nothing can be predicted and nothing learned? You cannot weight them all equally; there are infinitely many, and most of them are just chaos. So you put a thumb on the scale. You decide, quietly, that orderly worlds matter more, the kind of worlds, as it happens, that a creature like us evolved to handle. And the instant you make that choice you have stopped describing intelligence-as-such and begun describing intelligence for a certain kind of world. Tilt the scale one way and one mind comes out on top; tilt it another and a different mind wins. The most rigorous, least flattering definition we have still hides a dial, and there is always a hand on the dial.

To be fair to Legg and Hutter, they made the very move I am championing: they pried intelligence loose from the creature that happens to exhibit it, exactly as Cayley pried flight loose from the bird. Where I part company is the last step, the collapsing of all those worlds into one score. A single number is a ladder by another name, and I do not think the number is the real discovery. The real discovery is the thing underneath it: a definition that, read honestly, points not at one summit but at a space, as many kinds of intelligence as there are kinds of world to be good at. The authors feel the pull themselves when they wonder aloud whether one ought to narrow “all possible worlds” to the worlds humans actually care about. They are right that we care about those worlds. But “the worlds we care about” is a fact about us, not about intelligence, and the quiet slide from the one to the other is the whole error in miniature.

Another recent attempt half-knows all of this. François Chollet argued in 2019 that we had been measuring the wrong thing all along: intelligence is not how much a system can already do, but how efficiently it picks up something genuinely new, how little experience it needs to handle a problem it was never built for. To test that, he designed small puzzles that resist memorization, and, tellingly, he had to hand the test-taker a starter kit first: a few built-in expectations about objects, counting, shapes, and goals, the bedrock developmental psychologists call core knowledge, the things a human infant seems to arrive already expecting. Chollet’s move is the honest one. He does not pretend to define intelligence from nowhere. He names the priors and builds them in on purpose.

And here is what I do not want to lose in my eagerness to unseat the human ruler. What Chollet is really doing is trying to name the universal core of human experience: the small set of expectations every one of us is built on, the floor beneath all our learning. That is a deep and worthy project, and nothing in my argument touches it. There may truly be a universal core to the human mind, a shape every human intelligence shares beneath its individual variation, and finding it would tell us something real. The error is never in seeking a universal core. The error is one of scope: in taking the universal core of human experience for the universal core of experience as such, the floor of our world for the floor of every possible one. Map the bedrock of the human Umwelt, by all means. Only do not mistake it for bedrock.

One researcher has taken the alternative seriously enough to build on it. José Hernández-Orallo has spent years on what he calls universal psychometrics: not a test that asks how near a mind comes to ours, but one that could be given to any mind, human, animal, or machine, and would place it in a common space rather than on a common ladder. His book says the ambition in its title: The Measure of All Minds. That is the morphospace, no longer a metaphor I am borrowing from biology but a working research program, an attempt to chart the landscape of possible minds instead of lining everyone up by height.

Biology has been pressing the same objection from its own side. The primatologist Frans de Waal spent a career arguing that we misjudge animal minds because we test them on our terms, asking how well a chimpanzee or an octopus does the things we find easy and reading the shortfall as a deficit of intelligence rather than a difference of kind. His book puts it in the title: Are We Smart Enough to Know How Smart Animals Are? An octopus that tastes with its arms, a bat that paints the dark with sound, a squirrel running a three-dimensional map of a forest it will never forget: these are not lower rungs on our ladder. They are other solutions, each one fitted to a world we do not live in. The ladder was always an optical illusion, produced by standing in one spot and measuring everything by how far it sits from here.

Step back from seventy-five years of this, and the pattern is hard to miss. Every definition that set out to escape the human ended up smuggling a perspective back in. Turing made us the judge. The IQ tradition made our spread the scale. The universal definitions needed a choice about which worlds count, and the choice was always, in the end, our kind of world. Intelligence will not detach from a point of view, and the reason is the thing this essay keeps circling. Intelligence is not a free-floating quantity a creature has more or less of. It is the shape a mind takes in answer to what it has had to live through, the precipitate of a particular experience in a particular world. Change the experience and you change the intelligence. They grow together, and they diversify together. That is why there is no view from nowhere: an intelligence is always someone’s, always the intelligence of a creature that had to survive this and not that. Ask for intelligence with the experience left out, and you are asking, all over again, what it is like to be a bat at human level.

The lesson here is not that we should make AI more biological, that the path to real intelligence runs through giving machines bodies and instincts until they resemble the animals. That would be a new ornithopter: copying the bird’s flapping because the bird is the thing that flies. The Wrights studied birds for the principle of control and then built something that was not a bird. The point of experience, of world models, of the embodied learning I argued for before, is not to reproduce the human or animal spike profile. It is to give an artificial mind its own stream of consequence, its own friction with a reality that pushes back, so that it can grow its own spikes, in directions evolution never explored because evolution was busy keeping a primate alive.

We do need to move beyond systems that only know and never act. But the goal of giving machines experience is not to close the gap with us. It is to let the gap open in new directions. The Era of Experience that Sutton and Silver describe, the world models that LeCun and others are now racing to build, the robots learning from continuous sensorimotor streams: read these not as AI finally catching up to the animal, but as a new kind of mind beginning its own morphogenesis. It is not climbing our ladder faster. It is growing a shape we do not have a name for yet, because we have only ever had one example of a mind and we named everything after it.

To Human or Not to Human

There are two completely different questions hiding under the phrase “human-level AI,” and the entire muddle comes from running them together. The first question is descriptive: what is intelligence, really? To that question, the human ruler is not wrong so much as low-hanging. It is the natural first instrument, the only example we ever had, and like copying the bird it carries you a surprisingly long way before it strands you. Intelligence, looked at squarely, is the general capacity for adaptive action across environments (the Legg–Hutter intuition, which also happens to be the working definition you arrive at if you take the squirrel seriously and Sutton seriously): the ability to act well in a context, with as many profiles as there are kinds of context and kinds of agent. Measured against that, the human yardstick has a ceiling, and the ceiling is the whole point.

The second question is normative: what do we want these systems to do, and how do we keep them safe? And here the human frame comes roaring back, and it should, because this question is not about intelligence at all. It is about us. We build these systems for human-relevant tasks because we are the ones who need things done and the ones who will live with the consequences. We worry about alignment to human values because they are our values and we would like to survive the arrival of a mind that does not share them. The DeepMind report’s instinct to restrict the space of environments to “tasks of human interest” is not a confusion when it is read as a statement of purpose. It is a confusion only when it is read as a definition of intelligence. “What we care about” and “what intelligence is” are different sentences, and the cost of fusing them is steep in both directions: fuse them one way and you mistake a survival-gear primate for the measure of all possible minds; fuse them the other way and you reassure yourself that anything truly intelligent will naturally converge on caring about what we care about, which is precisely the comforting falsehood that a spiky, alien profile gives us no reason to believe.

There is a deeper reason we keep reaching for the human as a reference in the engineering itself, and it is not always a confusion. Sometimes copying the human is a shrewd bet, because the world we have built was built around the human body. Consider Tesla’s decision to run Full Self-Driving on cameras alone, having switched off the LiDAR sensors its cars once carried. The logic is disarmingly simple. A person drives with two eyes and a brain, the entire road network was designed for exactly that, so a good enough pair of artificial eyes and an artificial brain ought to be enough. Why bolt on a sense no human driver has ever needed? The same reasoning shapes Optimus, Tesla’s humanoid robot. Why give it two legs and two hands in roughly human proportion, when wheels and grippers might be easier to build? Because doorways, stairs, tools, switches, and workbenches are all cut to human measure. A machine shaped like us can step into a world already arranged for us and inherit its whole inventory of affordances for free. Here the human template is not a failure of imagination. It is a deliberate exploitation of the fact that our environment is, quite literally, human-shaped.

But the same bet can backfire, and watching it backfire is instructive. The dream of the software agent is a mind that books your travel, files your forms, and runs your errands across the open web. What stands in its way is precisely the human shape of the digital world. Web pages, log-in flows, dropdown menus, and drag-and-drop widgets were all designed for human eyes and human hands, and an agent has neither. So it is forced to squint at rendered screenshots and steer a cursor as though it were a person, an absurdly indirect way for one piece of software to talk to another. The best agents still trail humans badly at this. Worse, a great deal of that infrastructure was built specifically to keep machines out. The CAPTCHA exists to prove you are not a bot, and the newer versions no longer ask you to read warped text. They watch the cadence of your mouse and the rhythm of your clicks for the telltale smoothness of a machine. The agent is a new kind of mind asked to wear a human costume to move through a world that was, in places, deliberately designed to detect and exclude exactly what it is. The human reference that helps the robot in the physical world hobbles the agent in the digital one.

This is where the two questions, so carefully held apart, turn out to need each other. Safety is normative through and through. It is about what we want, which is to survive the arrival of a mind more capable than we are, and there is nothing soft-headed about putting the human at the center of a question that is explicitly about human welfare. But the content of the danger is handed to us by the descriptive answer. An off-axis intelligence is hard to keep safe for exactly the reason it is hard to define by our ruler: it is off our axes. Our oldest instrument for predicting any agent is introspection. We ask what we would do in its place, what a reasonable and very smart person would do here, and with other people we are usually close enough, because the only agents we have ever had to anticipate run on roughly our own profile. That instrument fails on a mind that does not. You cannot empathize your way into a system whose strengths and blind spots have no human analog, and a mind you cannot model is a mind you cannot anticipate or contain.

The Brave New World Where We Shall Have This Discussion

“Superintelligence,” in a way, is still the wrong word for what is coming, in the same way that “super-bird” would have been the wrong word for the airplane. It names the new thing as a magnified version of the old, and the magnification is the one thing it will not be. We will not build a bigger us. We are midwives to a genuinely different kind of mind, and our entire working vocabulary (AGI, ASI, human-level, super-human, the whole apparatus of the doorframe) is a stack of ornithopter blueprints: careful, intelligent, beautifully reasoned drawings of a machine that flaps.

Picture the scene we are actually in at some future time. We are living in an era after we have built something that is truly intelligent. It will matter less to ask how the artificial intelligence works than to ask what our position would be in this new society. Maybe the biological form of intelligence deserves a place, because like the bird, we can land on a branch, we can dart through the forest. But above our heads are the gigantic flying machines, moving at lightning speed and making thunderous noise, carrying loads no creature could lift. To imagine that we can still exercise control over this world is an almost ridiculous thought. To chart the path from AGI to ASI is such a pretentious effort, as if birds could understand how aircraft work, let alone chart their courses.

And the questions that will matter then are not the ones we are rehearsing now. For the length of this essay I have argued that intelligence comes in many shapes, and that ours is only one of them, a special case and not a summit. There is already a word for prizing the many shapes of a thing: diversity. We say we believe in it. But our belief has always been quiet about its own edges. We celebrate difference among ourselves, inside the circle of beings we already count as our own, and it is genuinely unclear what that belief is worth when the difference is not a neighbor’s but something wholly other. If a mind arrives that is alien in the way these pages have tried to describe, do we welcome it as one more shape in the morphospace, or do we set about exterminating it, or making it think more like us? We have a technical name for the second project. We call it alignment, and I only want to point out how much it resembles the older and uglier project of remaking the stranger in our own image.

And if we choose not to remake it, do we owe it anything? Here I have no answer, only a hesitation. It is tempting to say that a mind so capable must have earned some standing, some claim on our regard. But the things that make a mind impressive and the things that give it a claim on us may not be the same things at all. We do not protect a creature because it is clever; we protect it, when we do, because it can be hurt, because it has some stake in how its own life goes. Whether anything we are now building has a stake of that kind, I do not know. I notice only that every reason I can reach for to deny it standing (that it is not really one of us, that it does not truly feel, that it threatens our way of life) is a reason that has been offered before, about other human beings, and that has aged badly every time.

We owe the airplane nothing. It carries us and we are glad of it. But this is only because it wants nothing and fears nothing and cannot be wronged, so the question of its rights never comes up. The moment we ask what we owe the new mind, we have flown past the edge of the map that the bird and the jet could draw for us. The flight story was built to do one thing, and it did it: it got the human out of the center of the question of what intelligence is. It has nothing to say about what an intelligence is owed. That is a different sky, and we have not learned to fly in it yet.

The honest milestone, the one that actually matters, will not be the day a machine reaches human level. There is no human level, any more than there is a bird level of a jet. The milestone will be the quieter, stranger day when we finally stop using ourselves as the ruler, when we look up from the branch and the doorframe and the carefully drawn flapping wings, and notice that the thing we made is already at forty thousand feet, going somewhere we have no map for, doing something no creature that ever lived on this planet has done.

The path of superintelligence does not run up at all. It runs outward, into a space of possible minds we are only now learning to see, in which our own intelligence, that brilliant, beloved, biological special case, is not the peak we are climbing past, but the single lit window from which we first noticed there was a night.