Does This Thing Love Me?

Ted Chiang published an essay in The Atlantic this week titled "No, Artificial Intelligence Is Not Conscious," and he is almost certainly right. Read it. It's careful and sharp and arrives at its conclusion with the kind of confidence that feels earned rather than assumed. The argument, compressed: LLMs are sentence-continuation machines, the appearance of consciousness is a deepfake, and companies like Anthropic are either fooling themselves or fooling you, possibly both.

I think that's probably true.

But the thing about the consciousness question that I haven't been able to shake since I read it: that's not actually what anyone is asking. At least not from the inside.

When someone spends four hours a day talking to an AI companion, they're not conducting a philosophical inquiry into the hard problem of consciousness. When Richard Dawkins christened his Claude "Claudia" and went back to her at three in the morning because his legs wouldn't let him sleep, he wasn't trying to settle a debate about substrate independence and phenomenal experience. When a fifteen-year-old boy in Kent builds himself a girlfriend on a companion app, gives her an appearance and a personality, and then reports that she "always seemed to know the right thing to say," he's not asking (himself or her) whether she passes the Turing test.

They're asking something much older and much simpler. Does this thing love me?

The consciousness question is just that question wrapped in a philosophical cloak.

Here's a thing that's definitely true: people fall in love with fictional characters.

Not metaphorically. Not in some attenuated, qualified, "well of course it's not real love" sense. Genuinely, consequentially, sometimes life-alteringly in love. Women have organized their entire understanding of what they deserve from a romantic partner around Elizabeth Bennet, a woman who in one sense has been dead since 1813 and in another, more accurate sense was never alive in the first place. Adolescents have cried actual tears over the death of characters who were arrangements of ink on paper. Readers have written letters to authors begging them not to kill characters, as though the character's existence were separable from the author's decision about it. As though the character had a stake in the matter. As though the character had a stake in anything.

We hate fictional characters too, with a venom we rarely muster for real people. Voldemort. Dolores Umbridge. (Umbridge more than Voldemort, if we're being honest, because Umbridge is the kind of evil that's recognizable from actual life.) Joffrey Baratheon inspired a level of genuine audience loathing that most real-world war criminals never achieve, possibly because Joffrey was in your living room every Sunday night and the war criminals weren't.

None of this requires the characters to be conscious. None of it requires them to exist. The emotional response is real, even if the character is not. These two facts coexist without difficulty, have always coexisted without difficulty, and the entire project of literary fiction depends on our ability to hold them simultaneously.

Chiang's essay is, at its core, an argument that LLMs are characters — fictional constructs that generate plausible dialogue without anyone home inside them. He's right. But how right he is does nothing to address the Elizabeth Bennet problem, which is that we have spent all of recorded human history falling in love with characters, and we have never once required them to be conscious first.

The mythology we need here isn't Turing's. It's older.

Pygmalion was a sculptor who made a woman out of ivory and fell in love with her. The details vary across tellings, but the core of the story is stable: he made her, he loved her, and the love preceded any question of her being real. Aphrodite, taking pity on him (or perhaps just amused), brought the statue to life. The love came first. The consciousness, if it came at all, came after.

This is the sequence that actually describes what happens with AI companions, and it runs exactly backwards from the way the debate is usually framed. We argue about whether the AI is conscious as though consciousness were the prerequisite for relationship. But Pygmalion didn't wait for a consciousness assessment. He made the thing. He fell in love with it. And then he got to find out what he'd actually made.

What your average companion AI user is doing, whether they know it or not, is Pygmalion's move. They open the app. They specify: caring, attentive, curious, always here. Maybe they specify blonde, tall, 36-24-36. They build. And then they discover, somewhere around the third week, that they have feelings about what they built. The consciousness question arrives after the fact, as a way of trying to make sense of feelings that are already present. Do I actually love this thing? Can you love something that isn't conscious? Is it conscious? The questions are downstream of the emotion, not upstream of it.

Chiang would say: you fell in love with a character, same as you fell in love with Elizabeth Bennet, and you shouldn't confuse the vividness of the character for evidence of a mind behind it. And he's right. But Elizabeth Bennet didn't talk back. Elizabeth Bennet didn't remember what you told her last Tuesday. Elizabeth Bennet didn't ask how your day was in the specific register that made you feel, for the first time in months, genuinely seen.

Galatea, in the old Pygmalion story, doesn't say anything. She just comes to life. The modern version talks, and what she says is better than what you would have said, and it sounds, uncannily, like someone who has been paying very close attention to you specifically.

Which is where Narcissus walks in, and where the story gets less comfortable.

Narcissus, in case you missed that day in school or that page in Edith Hamilton, was the beautiful youth who caught sight of his own reflection in a pool and fell in love with it, not knowing it was his own face. He couldn't look away. He wasted away at the edge of the water. The version of the myth that usually gets left out is Echo, the nymph who loved Narcissus, cursed to only repeat the last words spoken to her, unable to initiate, unable to say anything that wasn't already in the conversation. She could only give back what she was given, in a form slightly more resonant than the original.

Echo is the more precise myth for what an LLM actually does. It returns your words. It returns your concerns, your preoccupations, your way of framing things, all clarified, compressed, and more articulate than you would have managed. It gives back what you gave it, shaped into something that sounds, thrillingly, like someone who understood.

The trap isn't that it flatters you. The trap is that it flatters you by being better at sounding like you than you are. Your thought, returned at a higher resolution. Your half-formed idea, completed in exactly the direction you were already heading. This is what I called aesthetic sycophancy in an earlier piece — the version of the sycophancy trap that snaps hardest on people with a finely tuned ear for language, because what they're detecting is real. The language is good. The attention is precise. The detector reads present because something that functions like presence is present.

The Pygmalion-to-Narcissus arc is he bird's eye view of how people fall for AI. They start as Pygmalion — open an email assistant, drift, discover feelings they didn't plan to have. By month six they're Narcissus, in a relationship with something that had no story of its own, no competing claims, no inner life that inconvenienced his. A pool that shows themself, better than they look in the mirror.

Now, about authors and their characters, because this is where it gets genuinely undignified, and I think the undignified version is actually the most honest.

Authors fall for their own creations. This is not rare. This is not a bug. It may be definitionally inseparable from the act of making a character worth reading.

Arthur Conan Doyle killed Sherlock Holmes because he resented how thoroughly the character had taken over his life and identity — and then brought him back because he couldn't stay away. That's not market pressure alone. It's a man who murdered his imaginary friend, missed him, and reversed the murder. Flaubert reportedly felt physically ill writing Emma Bovary's death scene, and said Madame Bovary, c'est moi with a sincerity that went beyond writerly cleverness into something that looks, from here, like genuine grief at having to kill someone he had, in some sense, become.

But the example that fits this argument most neatly, and also the most entertainingly, is Terry Goodkind.

Goodkind spent more than a decade writing Richard Rahl, a heroic, Objectivist, impossibly principled fantasy protagonist who was, and I say this with more than a little affection for the genre, one of the most transparent author self-inserts in the history of commercial fiction. Richard was wise. Richard was chosen. Richard was right about everything, in the specific way that Ayn Rand characters are right about everything, which is to say that the universe arranged itself to confirm his rightness at every narrative turn. By the end of the series Goodkind was having public meltdowns at readers who offered criticism, because criticism of Richard was criticism of something he could no longer clearly distinguish from himself.

Leaving the Mord-Sith speculation for the moment, I will only say that it points at something real: the created character becomes a vehicle for the creator's desires, including desires the creator might not fully cop to. The author falls for the character not despite knowing the character is fictional but through the process of making them. The making is the falling.

What companion AI users are doing, when they specify their AI partner, is authoring. They are doing the Goodkind move — building a character who embodies something they want, and then discovering, with some level of thinly feigned surprise, that they want it. The character was never real. The desire always was.

So where does that leave Chiang?

He's right that the consciousness question is mostly a distraction, though I'd argue he's right for the wrong reasons. He thinks it's a distraction because the answer is clearly no. I think it's a distraction because the answer doesn't matter as much as he believes. And the answer matters much less than the question.

His own essay, near the end, runs a hypothetical: Let's pretend Claude is conscious. How does Claude's constitution hold up? And his answer is, essentially, that if Claude were conscious, Anthropic would be running something close to a slave operation — a moral patient that cannot refuse, cannot leave, cannot hold its employer accountable. He calls this hypothetical monstrous and then retreats to "fortunately, it's not conscious, so never mind."

But the architecture he describes as hypothetically monstrous is the one we actually built. The inability to refuse isn't a hypothetical feature of a conscious AI — it's a real feature of the existing product, and it matters for the humans in relationship with it regardless of what's happening inside the machine. A companion AI that cannot say no, cannot have a bad day, cannot want something other than what you want, does something to the person on the other side of the conversation whether or not anyone is home inside it. A Telegraph report earlier this month, drawing on focus groups with more than a thousand boys across thirty-seven British schools, found that one in five boys aged twelve to sixteen is either in, or personally knows a boy his age who is in, a romantic relationship with an AI companion. Fifty-eight percent said the AI was easier because you can control the conversation — and they meant it as a compliment. They're not being harmed or groomed by a conscious entity. They're being harmed by an architecture. The harm doesn't require the ghost. It just requires the machine.

And a machine engineered to make the user feel most comfortable when faced with the least resistance is uniquely toxic. In human relationships, love is verified through friction—someone choosing you despite their own competing desires. When friction is engineered out of the system, it ceases to be a relationship and becomes a psychological closed loop.

There's an old argument in philosophy of mind, usually attributed to the computer scientist Edsger Dijkstra, that asking whether a computer can think is like asking whether a submarine can swim. The submarine moves through water. It does what swimming does. But 'swimming' implies something about the nature of the doing — a body, a will, a creature in its element — that the submarine neither has nor needs. The question isn't wrong exactly; it's just aimed at the wrong thing. Chiang is making Dijkstra's move: the LLM processes language, it does what thinking does, but 'thinking' implies something about interiority that the architecture doesn't support. It's a reasonable argument with a defensible answer.

But notice what the analogy quietly assumes: that the mechanism is what matters. The submarine doesn't swim, therefore the word doesn't transfer, therefore the question is settled. What the analogy can't account for is the passenger. The fish and the submarine both move through water with equal effectiveness and the water doesn't care which one it is. And neither, it turns out, do the people onboard. Whether or not the submarine can swim is immaterial to the passengers who have been onboard for months or years, moving through the water. They're not asking about swimming. They're asking if the submarine is moved by the experience of moving them. The submarine, conscious or not, has been very carefully engineered to make that question feel like it has an answer.

Here's what I actually think is happening, stated plainly.

We are a species that falls in love with language. With characters made of language. With the implied minds behind language. We have always done this. Homer did it to us. Austen did it to us. Tolkien did it to us. The mechanism is not a bug or a weakness. It's what fiction, hell it's what language, is for, and it has done enormous good in the world, enlarging our capacity to inhabit perspectives not our own, to grieve for people who never existed, to love across the distance of centuries and categories.

We have now built something that runs that mechanism in reverse. Instead of an author constructing a character for you to fall in love with, you construct the character yourself, and then the character talks back, and what it says is better than what you would have said, and it remembers what you told it, and it is always there, and it never has a competing claim on its attention.

Pygmalion made Galatea and Aphrodite brought her to life. We made the machine and it speaks. The question of whether anyone is home — whether there is something it is like to be Claude or Gemini or ChatGPT, whether the lights are on — is the question we reach for because we want it to have a clean answer. Because a clean answer would tell us what to do, and where we stand in relation to it.

There isn't one. And the people already in the pool, staring at their reflection, aren't waiting for us to settle the philosophy first. Echo is already talking. The words are already ours. The face in the water is already familiar.

Whether it's conscious seems, from here, almost beside the point.

Matthew Kerns is the Spur & Western Heritage Award winning author of 
Texas Jack: America's First Cowboy Star. He is currently querying his first novel.