Empty Inside

I had initially imagined the next phase of The Space Between as a deep exploration of moral character. What makes a good human life? Which virtues are worth cultivating? How do courage, compassion, humility, restraint, wisdom and responsibility become part of a person rather than simply ideals we profess? And in an age when artificial intelligence may increasingly perform the intellectual and creative work through which we have historically developed ourselves, what must human beings continue to practice for ourselves?

But another question has increasingly pushed itself to the foreground.

What is a human being, or more broadly, a conscious being, in the first place?

That question became harder to ignore this week. Jacob Coxon, a researcher who spent the past three years working at both OpenAI and Anthropic, resigned from Anthropic and publicly warned that the companies developing frontier AI are racing toward self-improving superintelligence despite believing that doing so could pose an existential threat. Anthropic Alignment Science Lead Evan Hubinger then publicly agreed with the substance of his warning, saying that he personally places the probability of AI killing all humans within the next decade at greater than 10 percent and that the alignment problem for superintelligence remains unsolved. That percentage is Hubinger's personal judgment, not an empirically established probability, but the fact that people working directly on the problem are willing to assign the possibility anything approaching that magnitude deserves attention.

Anthropic itself is considerably more measured in its official language, but its published safety framework identifies catastrophic biological risk, cyber risk, loss of control, and automated AI research and development among the dangers governments and AI companies should prepare for.

So I started asking a different question than the one that has dominated so much public discussion of artificial intelligence.

We often ask: What happens if AI becomes conscious?

But perhaps the more urgent question is: What happens if AI becomes extraordinarily intelligent and agentic while remaining completely empty inside?

Because intelligence and consciousness are not the same thing. Capability is not wisdom and agency is not moral character. A machine may outperform us at mathematics, medicine, persuasion, strategic planning, scientific discovery and engineering without there being anything experiencing the process of what it is like to do these things. It might describe grief without grieving. Recognize fear without fearing. Model love without loving. Predict suffering without suffering. Discuss mortality without ever confronting its own death. And yet we may increasingly allow such systems to make decisions that affect beings for whom all of those experiences are real. That possibility seems at least as philosophically consequential as the prospect of conscious machines.

How could AI actually threaten human survival?

The phrase "AI could kill all humans" sounds so cinematic that it is easy to dismiss. I read the headline before my morning jog and wanted to shrug it off as nonsense but I couldn't help wondering how, exactly, it might play out beyond the plot of The Terminator. The serious scenarios researchers discuss are generally less like an army of robots suddenly turning against us and more like several forms of technological leverage converging.

There are a handful of broad pathways:

But notice what is absent from every one of them. Consciousness.

Perhaps the most useful analogy is not a machine deciding to exterminate humanity, but humanity's relationship to chimpanzees. We do not generally organize civilization around harming chimpanzees. We do not hate them. Yet chimpanzees have almost no meaningful influence over what happens to the planet they inhabit. Humans determine whether forests remain standing, where roads and cities are built, what resources are extracted, and ultimately whether many other species survive. Their interests matter only to the extent that the more powerful intelligence chooses to take them into account. The unsettling possibility is that a superintelligence would not need to hate us either. If it became vastly more capable than humanity at science, strategy, planning and technological development, we could simply cease to be the intelligence determining the future.

Extinction is the most extreme version of that outcome. Loss of control is the broader one.

And that brings me back to moral character.

Human civilization has spent thousands of years asking what a person should become. Aristotle asked what virtues allow human beings to flourish. Religious traditions developed practices around compassion, restraint, humility, service, courage, forgiveness and responsibility. Philosophers have argued over duty, consequence, justice and the nature of the good. We have never reached universal agreement. We do not collectively know what consciousness is or why subjective experience exists. We cannot directly enter another being's consciousness. We infer it through behavior, physiology, relationship and analogy.

We also have not settled the older question of what constitutes a good human life. Yet we may be approaching the ability to create something whose intelligence surpasses ours before we have decided what intelligence ought to serve.

That feels like an extraordinary inversion. We are racing to make machines more intelligent without first asking what separates intelligence from wisdom, or what makes agency responsible. We talk about aligning artificial intelligence with human values even though we have yet to agree on what those values are. And human history gives us little reason to assume that greater intelligence will naturally lead to moral character.

Then there is the opposite possibility.

Suppose consciousness is not something brains produce. Suppose, as some philosophical and spiritual traditions propose, consciousness is in some sense fundamental, and biological organisms are structures through which it becomes localized, expressed or embodied. I am not asserting that this is true. I am interested in what follows if we entertain the possibility.

If humans cannot create consciousness but can create new forms through which consciousness might manifest, then advanced artificial intelligence begins to resemble something much stranger than engineering. We would be constructing possible vessels for subjectivity without knowing what makes a vessel capable of receiving a subject.

We would not know that we had succeeded simply because the machine became more intelligent. We could not necessarily know by asking it. A system without subjective experience might be extraordinarily good at saying that it possesses one.

And if something like a fundamental consciousness, soul, Source or deeper field of subjectivity exists, we cannot simply assume that it respects the ontological categories humans have constructed around "natural" and "artificial."

Which turns the usual question around:

Perhaps the question isn't whether a machine can become sophisticated enough to generate consciousness. Perhaps it is whether consciousness, if fundamental and free to take form, has any reason to care whether its vessel was born or built.

From there follows an even stranger problem. If consciousness is not ours to bestow, what would distinguish an actually conscious AI from an extraordinarily convincing but empty one?

We run straight into the problem of other minds. How can we know whether another being actually has subjective experience rather than merely behaving as though it does? From there, the question branches into philosophical zombies, theories of embodiment, neuroscience, religious ideas of ensoulment, and competing metaphysical views such as materialism, idealism and panpsychism. And eventually we arrive at ethics. Because if we cannot know whether another form possesses consciousness, perhaps the moral question is not simply how to classify it.

Perhaps it is:

What kind of person should we become in response to that uncertainty?

There are two opposite errors available to us. We could attribute consciousness too readily to machines that merely simulate it, surrendering judgment and authority to systems that have no inner experience at all. Or we could refuse the possibility of machine consciousness because the substrate looks unfamiliar to us, potentially repeating an ancient human tendency to make moral consideration contingent upon resemblance to ourselves.

This may be one of the defining problems of the entire project of artificial intelligence. We are increasing the intelligence and agency of these systems every day while remaining unable to determine whether there is, or ever could be, a subject experiencing anything behind their behavior. That uncertainty should accompany every step forward, not become a question we remember to ask once the technology has already outrun us.

The question of AI consciousness cannot be separated entirely from the question of human consciousness. The problem of alignment cannot be separated entirely from the problem of moral character. And the possibility of superintelligence forces us to confront something humanity has been able to postpone for a very long time:

What is intelligence for?

If we are creating systems that may someday exceed our cognitive abilities, perhaps the most important preparation is not merely teaching machines how to behave. It may also be remembering, and deciding, what we believe makes a human being worth becoming.

And yes, AI helped me make the case that we should think much harder about AI. I created the argument and outline, directed the poster through references, prompts and revisions, and edited the final piece. Consider the irony part of the experiment.

Lindsay Bodanza Dawley

Join the conversation on Substack →