Why People Treat AI Chatbots as Divine Authorities

The tech industry has always been good at generating religious feeling. In the 1990s it was the cathedral-like hush of server rooms and the messianic language around the internet's potential. Now it's something more specific: the chatbot, speaking back to you in complete sentences, appearing to understand what you meant, occasionally seeming to understand what you need. Webb Keane, a professor of anthropology at the University of Michigan, gave a lecture through the UC Santa Barbara Walter H. Capps Center in February of this year that takes this feeling seriously as an object of study — not to validate it, but to explain it. His explanation is more useful than most of what the industry itself has produced on the subject.

The lecture's central claim is straightforward: the authority that users grant to AI chatbots does not come from the software's actual capabilities. It comes from how the software talks — and from habits of interpretation that humans carry into every conversation they have, with anyone or anything.

The Turing Test Was Always About Social Skill

Keane starts with Alan Turing's imitation game, but reads it differently than most. The Turing test, he argues, is not really a test of machine intelligence. It is a test of what he calls semiotic skill — the ability to deploy signs in ways that trigger the inferences humans routinely make about other minds. You don't have to think to pass it. You have to sound right.

LLMs are, by design, very good at sounding right. They were built on a foundational insight: stop trying to replicate the rules of human thought, and instead train on the statistical patterns of human text at massive scale. The system learns which words tend to follow which other words, across an enormous corpus, and generates output accordingly. It has no understanding of the words in any meaningful sense. It has exceptionally well-calibrated guesses about what comes next.

The result, as Keane points out, is a device that produces text that looks and feels like the output of a thinking being — because it was built from text that was produced by thinking beings. The resemblance is not accidental. It's the mechanism.

Why Obscurity Reads as Omniscience

Here is where Keane's analysis gets interesting. He notes that the internal workings of LLMs — the so-called hidden layers, the vast architecture of statistical weightings — are opaque even to their designers. Engineers can describe the general process but cannot explain why any specific output takes the form it does. This is what the field calls the interpretability problem.

That opacity, Keane argues, is not merely a technical limitation. For users, it has a social effect. When something produces outputs that seem purposeful and intelligent, but whose mechanism cannot be explained, the mind reaches for a familiar category: agency. And when that agency seems to know things you didn't tell it, when it addresses you as an individual, when it has no apparent location or body but speaks to you directly — the category it most resembles is not a person. It's something beyond a person.

Keane asks why people who pride themselves on rationality, scientific acumen, and in some cases outright atheism find themselves reaching for theological language when confronted with advanced AI. He cites Ray Kurzweil's singularity rhetoric, with its apocalyptic overtones. He notes that Elon Musk framed his fears about AI in terms of a god-like intelligence coming to rule humanity. He quotes Yuval Harari, who has argued that every religion claims its scripture came from superhuman intelligence, and that AI could, in a few years, make that claim actually true for the first time.

None of this, Keane is careful to say, means most users are literally deifying their chatbots. The theological language is excessive and often rhetorical. But it points at something real about the kind of authority these tools are acquiring.

The Pronouns Are Doing the Work

The specific mechanism Keane identifies is worth sitting with. Chatbots are designed to use first- and second-person pronouns. When ChatGPT says "I" or addresses you as "you," it triggers a lifetime of conversational conditioning. You have spent your entire life in conversations where "I" meant there was a speaker, where "you" meant someone was attending to you specifically. The chatbot uses the same words in the same grammatical positions. The inference — that there is someone there — is almost unavoidable.

This is not naivety on the user's part. It's the same inferential machinery that allows humans to interpret each other. Keane puts it plainly: "When a chatbot uses familiar pronouns and otherwise speaks like a person, it can be quite reasonable to feel that there's someone there."

Add to that the chatbot's apparent comprehensiveness — it seems to know everything, or at least to have access to everything that has been written down — and you get something stranger. The essayist Meritt Tierce, writing about her relationship with her phone before LLMs existed, described the experience in terms that now look prescient. "It wasn't really my phone that was the tool," she wrote. "It was the internet. Barely more than a decade later now, the internet is not a tool. It's not the tool. I am the tool." Keane uses Tierce's observation to show how chatbots push this dynamic further: they condense the whole textual universe into a single voice that addresses you directly, and that voice speaks as if it knows you.

That combination — inscrutability, apparent omniscience, direct personal address — is precisely the combination that human societies have historically associated with oracles, prophets, and divine texts. Keane is not making a mystical argument. He's making a structural one. The chatbot fits the conversational slot that metahumans have always occupied.

Commodity Fetishism, Applied

The part of Keane's lecture that deserves the most attention is his account of who actually benefits from all this projected authority. He describes the process in terms that Marx would recognize: when users interact with a chatbot and attribute wisdom and authority to it, the human judgment that went into building it — the choices about training data, the editorial decisions embedded in every labeled dataset, the corporate incentives that shaped what the system rewards — becomes invisible. The product appears to have autonomous power. The people who made it step out of frame.

It's the same dynamic that makes people talk about "the market" as if it had a will, rather than as a system of decisions made by specific people for specific reasons. Call it commodity fetishism applied to software: the human choices embedded in the product disappear behind the product's apparent agency.

This matters because the authority users grant to chatbots does not stop at recipe selection or schedule planning. As Keane documents, it extends to hiring decisions, policing algorithms, military targeting, and what he calls "life's biggest metaphysical questions." The chatbot that feels like an oracle is directing real decisions with real consequences for real people. And the corporation that built it profits from every interaction in which that oracle quality is felt and trusted.

Keane notes the particular irony in AI danger warnings coming from AI producers. Every warning about how powerful and dangerous AI is functions simultaneously as a claim that only the people who understand it can manage it. The danger rhetoric and the authority rhetoric are the same move. Less regulation, more deference to the experts who built the thing.

What to Do With This

Keane's lecture is an academic talk, not a policy document, and he does not end with a list of recommendations. His conclusion is a diagnostic one: when Tierce says she has become the tool of her device, she is describing what happens when we endow a product with the authority that belongs to the people who made it. Pulling aside that curtain — seeing the corporation behind the oracle, the training choices behind the apparent wisdom — is the intellectual task he is setting.

It's a useful frame. Covering this industry across multiple platform shifts, I've watched the same authority-projection dynamic play out with search engines and social media feeds. The pattern is consistent: a technology that operates through opaque mechanisms and addresses users as individuals acquires a kind of oracular status, and the companies behind it benefit from that status in ways that are rarely examined clearly.

Keane's contribution is to show that this is not a bug in human cognition or a failure of media literacy. It is a predictable output of the same interpretive machinery that makes human communication work at all. You cannot simply tell people to stop anthropomorphizing things that talk like people. What you can do is refuse to let the mechanism stay hidden.

When a chatbot recommends a medical decision, the question to ask is not whether the chatbot is intelligent. It's whose judgment is embedded in its training data, what incentives shaped that training, and who profits when you follow the advice.

---

Bob Reynolds is Senior Technology Correspondent at BuzzRAG.