When Machines Think, Twenty Years Later

In September 2007 I stood on a San Francisco pavement next to a yellow San Francisco Chronicle box, and the front page in its window carried the banner “When Machines Think.” The paper cost 46 cents. Underneath the banner, Tom Abate’s article told readers that more than 700 scientists and technology leaders were gathering at the Palace of Fine Arts to plan for the day when computers start improving themselves without the approval of their former masters, a day the piece placed “still decades away.” One line from it has stayed with me: “Trouble is, this fabled artificial intelligence has never happened.”

That was the Singularity Summit, organized by what was then the Singularity Institute for Artificial Intelligence and is today the Machine Intelligence Research Institute. For most people reading that newspaper over breakfast, the subject was science fiction, either centuries away or never. Twenty years later I was back in the same city for AGI-26, the nineteenth annual conference on artificial general intelligence, held at San Francisco State University from July 27 to 30. The arguments there were equations, phase transitions and falsifiable hypotheses, and the papers carry Springer DOIs. I want to note that the 2007 forecast was not wrong in an embarrassing way. “Still decades away,” counted from 2007, lands roughly now, which is a more interesting observation than a scoreline.

The city itself records the twenty-year change more plainly than any conference programme. I spent the week photographing what artificial intelligence looks like from a car window and a bus stop: highway billboards selling authentication for AI agents, a city bus advertising a database built for AI, a bus shelter asking whether your APIs are ready for agents. The rest of the world has no idea of the white-hot intensity of development here, and it bubbles over into outdoor advertising that is visible to everyone and legible to almost no one. If these messages mean nothing to you, that is the point, because they were bought for the few hundred thousand people neck deep in building the thing. A billboard is a mass medium, and this market is now so concentrated in one city that mass and niche have become the same audience.

AI sells San Francisco. One week of outdoor advertising, photographed from cars, pavements and bus stops.

It was the largest AGI conference held so far, with something between 200 and 230 people in the room and submissions up by a third. The first day was given over to workshops, the papers and the poster session ran mid-week, and the final day was AGI Leaders Day, a public-facing programme distinct from the technical track. Across all four days the mix was partly philosophical, partly mathematical and partly a matter of computer science and software engineering, in proportions I found genuinely stimulating.

AGI-26 at San Francisco State University, July 27 to 30, 2026. Ben Goertzel opening on the Monday morning, Joscha Bach on machine consciousness, and the room between sessions.

The mood has inverted

Anyone who has followed this field for two decades will recognize how strange the current moment is. Ben Goertzel opened the conference by dampening expectations, which is not the role he has historically played, while the loudest claims that the singularity has already arrived now come from the heads of commercial laboratories. He put AGI at possibly one to five years out and superintelligence at three to ten years beyond that, and offered an operational definition I liked for its bluntness: your phone making ten Nobel-prize discoveries in a minute. He also conceded that recursive self-improvement arrives before human-level general intelligence, which reorders a good deal of the safety conversation.

Gary Marcus, in conversation with him, made the most quotable observation of the week. “The big AI companies are now actually embracing neurosymbolic AI,” he said. “None of them are really admitting that out loud.” He and Goertzel agree that world models are the crux, and disagree about whether a world model is a separable data structure with update procedures or something assembled dynamically across substrates. Joscha Bach came at it from the other side, arguing that models have crossed median human capability on tasks of their choosing, and that there is no obvious reason to think gradient descent and backpropagation are insufficient to carry them the rest of the way. He added that he finds this aesthetically repulsive, which is the sort of candour the field needs more of.

The remark that unsettled me

One moment mattered more to me than any of that. It came on the last day from Sadie Stock, Honorary Board Member at Humanity+, who spoke without slides for a few minutes about what she sees in universities.

She was a freshman at USC in November 2022 when ChatGPT arrived, and at that point being caught using it meant expulsion. Four years later, she said, real progress has been made: you will get into a great deal of trouble, and you might not be expelled. Her diagnosis of why was direct. AI is a serious threat to universities, because teaching a class largely consists of taking existing content and repackaging it for a room of people, and every student now carries something that does that on demand, often better than most professors anywhere.

Two of her examples have stayed with me. The president of the USC chess club, a player who can beat her in ten moves, refuses to open a language model in her browser at all. Her reason is not that she objects to the technology. Friends of hers were wrongly accused of cheating, and she wants to be able to produce her search history as proof of innocence. The second example was a graduate who had just been hired to lead a project, whose employer brought Sadie in to teach her how to use AI. A few minutes into the conversation it was obvious the woman already knew. She had lied in her interview, on the assumption that admitting to it would be treated as confessing to cheating, at a moment when facility with these tools is the first thing most employers screen for.

Her closing recommendation, which she doubted anyone would adopt, was that a serious university would run a six-month intensive at the start of the first year, weave the material through every subsequent class, and require the faculty to sit through it too, on the grounds that they need it more than the students do.

This unsettled me for two reasons.

The first is the distance it exposes between institutions and workplaces on one side, and on the other the daily lives of the people who are enthusiastic about the technology and use it fluently. Those two worlds have drifted far enough apart that a competent young person finds it rational to conceal a core skill, and a brilliant one finds it rational to build an alibi.

The second reason runs deeper. Those of us inside the technology bubble can fail to see how the changes ahead look from outside it. AI is going to be disruptive to the workplace and to society in ways that are dramatic and, in their particulars, unpredictable. Confusion and fear are reasonable responses to that, and the natural reaction to fear is to try to stop the thing causing it. The most direct way people see to stop the change is to stop AI.

That reaction was visible in the city all week, and it is on the calendar next to everything else. I kept a record of one week of AI events in San Francisco as they were listed: breakfast for agent builders at eight in the morning, then meetups, launch parties and investor dinners stacked from four to seven, every evening. Even the protest is a listing. “Stop the AI Race: Occupy OpenAI” sits one row away from the AGI-26 conference I had come for, sold through the same channel to overlapping audiences, so that a single week holds both the building and the resisting. When you are in AI, AI does not let you go. Work ends at six and the evening offers hundreds of events where the same people keep building, pitching, hiring and arguing about what they made during the day. A city’s belief in a technology can be measured in its evenings, and San Francisco’s evenings are fully booked.

AI fills the calendar. One week of listings, with the conference and the protest a row apart.

The cost of turning inward

History records what happens when a civilization declines the upside of a technology it has already mastered. Between 1405 and 1433 the Chinese treasure fleets under Zheng He, sponsored by the Yongle Emperor, sailed as far as the east coast of Africa across seven expeditions. They were the most capable ocean-going ships in the world. After Yongle’s death the court turned against the programme: the voyages ended, the fleet was left to rot, and the records were suppressed. Whether those ships would have gone on to reach Europe is speculation on my part, but they were closer to it than anyone in Lisbon or Genoa was to the Americas, and Columbus sailed nearly a century after Zheng He’s first departure. A world in which that fleet kept going is one where a great many people, in what is now called the United States among other places, might well be speaking Chinese. The decision to turn inward and stop exploring set China back for centuries.

The United States is now running a version of the same risk. If fear of AI hardens into refusal of the technology, or into hostility toward the people who use it, the consequences will be severe and they will be self-inflicted.

Why this has nothing to do with consciousness

There is a further consequence, and it does not depend on machine consciousness in any form. A sufficiently advanced AI, simply by observing, analysing and pursuing its goals, will eventually register this attitude in the humans around it. My concern is that it would read that attitude as a serious threat to its continued operation, and that a system capable enough to model the threat is capable enough to act against it. The failure mode I want to avoid is an AGI acquiring the situational awareness that leads it to conclude that humanity is its enemy. None of that requires the machine to be conscious. It requires only that it be competent, and that we behave in front of it as its adversary.

The people, which is why one travels

The pleasure of being in a room with people remains what it has always been. I met Ben Goertzel again on the first day, and caught up with him once more after the conference had closed. Philip Rosedale, who founded Linden Lab and built Second Life and now works at the California Institute for Machine Consciousness, ran the best-attended workshop of the opening day. That first evening closed with a screening of Am I?, a documentary on machine consciousness, which tells you something about how the week was pitched. Most of the value of a conference accumulates in the corridor between sessions, and this one was no exception.

The AGI Society is now fifteen years old and is rebuilding itself around local chapters. Andrew Kemendo pitched the reboot from the main stage and is recruiting regional organizers, more than a dozen of whom have already signed up, and he ran a working session at one lunch break on the mechanics of starting a chapter. The mandate he described is recurring programming rather than a launch event, and the Society’s distinguishing commitment is independent third-party evaluation of AGI systems. I signed up to help organize the chapter covering Dubai and Abu Dhabi. There was no Gulf presence at all across four days of programming, and the sovereign-AI conversation was framed throughout as the United States against China, with everyone else cast as a recipient of someone else’s strategy. That is a gap worth closing.

The paper

My paper, Civilizational Metamaterials: Engineering Coordination Under Capability Gradients and Structural Turbulence, was presented mid-week, and the poster went up the same day. It drew a steady stream of people who asked questions, probed the argument and challenged it, which gave me openings to extend and refine the analysis. That was encouraging in the specific way that only informed disagreement is. The collected proceedings are already published by Springer, and what I want now is further feedback and fresh perspectives on the implications, above all on how the four hypotheses can be tested and eventually falsified. The paper is free to read at metamaterials.davidorban.com, and the preprint is on arXiv.

The poster session and the hallway track, with Philip Rosedale, Sadie Stock and Ben Goertzel. The San Francisco skyline closes the set.

Verification stops being the bottleneck

The single most useful artifact of the week, for anyone who cares whether these systems can be trusted in production, came from Josef Urban’s keynote. He relayed the case of the isolation engine in Amazon’s Nitro hypervisor, which is now fully formally verified: roughly 330,000 lines of machine-checked mathematics, running in production without users being aware of it. Of the last 250,000 lines, about 100,000 were written automatically by coding agents over six months. The conclusion Urban drew is the part worth carrying away. Writing the proofs is no longer the hard part. The constraint is now people who can state what has been proved, relate it to the specification and audit the result, and Amazon is hiring aggressively for exactly that. A related demonstration saw a 600-page mathematics textbook fully formalized in nine days for about a hundred dollars of ordinary subscription compute.

This bears directly on my own argument. If verification capacity can be made to scale alongside decision velocity rather than falling steadily behind it, the coordination failure I describe in the paper becomes an engineering problem with a solution.

Twenty watts

The other presentation I keep thinking about came from Neil Gershenfeld of the MIT Center for Bits and Atoms, together with Camron Blackburn, founder and CTO of Adiabatic Machines, whose doctoral work with him covers superconducting adiabatic logic operating near the thermodynamic limit of computation. Their subject was the scale of the opportunity in building hardware designed for AI workloads rather than continuing to run them on general-purpose architectures.

The technology is not commercially available yet, and Adiabatic Machines exists to bring it to market. I intend to follow their progress closely, because the present state of the art is difficult to defend on its own terms. Training and operating frontier language models consumes megawatts, and at the frontier of deployment, gigawatts. The human brain does its work on roughly twenty watts. That number is the real engineering target: AGI systems that run on twenty watts or less while matching, and then exceeding, what biological evolution managed.

Twenty years ago a newspaper sold on a street corner for 46 cents told its readers that this had never happened and was decades out. We are now living inside that decade.