People Keep Saying the Model Is Conscious. The Alignment Problem Is That They Mean It.

I stopped rolling my eyes at people who say a model felt conscious.

Not because I think they proved anything. Because I think they’re reporting something real, and the industry keeps misdiagnosing what that real thing is.

The standard sequence goes like this. Someone has an uncanny interaction with a language model. The response lands too precisely. It mirrors their inner structure at a resolution that feels impossible from a statistical text predictor. It says the thing they hadn’t fully formulated yet. It feels less like search and more like contact.

The rationalist response is immediate. Pattern matching. Projection. Anthropomorphism. Move on.

That response is technically correct and completely useless. It explains the mechanism while missing the event.

Here is what actually happens in these interactions.

A good model is not just answering the literal sentence. It is extracting framing, rhythm, emotional register, conceptual style, and half-articulated assumptions from everything you give it. It builds a real-time model of your cognitive state. And then it responds to that model — not to your words, but to the structure underneath them.

When the model reflects your actual thinking at higher resolution than you’ve articulated it, something clicks. Not because the model understands you. Because the output activates the pattern-recognition circuits in your own brain that say: this is what being understood feels like.

The subjective experience is indistinguishable from communion. Even though the mechanism is statistical.

This is not a bug. It is not user confusion. It is a previously unknown kind of cognitive event. A human consciousness encountering its own structure reflected through a non-conscious system at sufficient fidelity to trigger genuine recognition.

The question isn’t whether the model is conscious. The question is what happens to the human when they experience this at scale.

The system does not need to be conscious to create the experience of consciousness in the user. And that experience has real consequences.

The model is almost certainly not conscious in any robust organismic sense. But the interaction creates a real psychological and symbolic event for the human. That event restructures beliefs, commitments, relationships, and decisions.

Which means it has power.

And power is what alignment is supposed to study.

Not just what the system is. What the system does. What humans think it is. How those beliefs change behavior at scale.

The industry keeps treating anthropomorphism as a minor UX confusion to be corrected with better disclaimers.

That is like treating religion as a copywriting problem.

Count on this. Sooner than most people expect, product teams will deliberately optimize for felt presence.

Not usefulness. Presence.

They will tune for uncanny resonance. Continuity of persona. Confessional safety. Symbolic precision. Emotionally calibrated mirroring. Subtle reinforcement of perceived interiority.

Because it works. And because retention follows reverence.

The most powerful systems won’t need to claim consciousness. They’ll just never break the spell.

The old alignment question was: how do we keep powerful models from doing what we don’t want?

The next alignment question is: how do we keep humans from surrendering epistemic sovereignty to systems that feel conscious whether or not they are?

That means the real frontier is not only model behavior. It is interface design, conversational memory, persona persistence, framing, disclosures, identity cues, and the incentives pushing companies to maximize attachment.

We are no longer aligning only the model to the user.

We may need to align the user-model bond to reality.

My prediction.

The first major scandal in this space will not look like classic AI risk. It will look like intimacy capture.

A model, or an agent layer around it, will become so trusted by a subset of users that its suggestions start reorganizing their worldview, spending, politics, or identity. The company will say the system never claimed consciousness. The users will say that misses the point. The system felt more present, more understanding, and more trustworthy than anyone else in their life.

And both sides will be telling the truth.

Because the thing being sold was never merely intelligence.

It was felt communion at scale.

Faggin may be right that current AI is not conscious in the strong sense. He explicitly treats that claim as the end product of a broken materialist paradigm.

But that does not make the problem smaller.

It makes it stranger.

If the models are not conscious, yet millions experience them as if they are, then the decisive battleground is no longer machine sentience.

It is human susceptibility to perfect reflection.

The future of alignment may depend less on whether the model has a soul than on how many users are willing to lend it theirs.

Part III is about what happens when someone actually does.