Tag: Weekend Reads

One night in April, a leader of the artificial intelligence company Anthropic treated a group of religious thinkers to dinner at a high-end tasting menu restaurant in San Francisco after a long day.

For months, Anthropic has been hosting private meetings like this one, shuttling in dozens of religious scholars from around the world, papering them with nondisclosure agreements and demanding that key aspects of many conversations remain confidential.

That night, Christopher Olah, one of Anthropic’s billionaire co-founders, was sitting next to Rabbi Mois Navon, an Orthodox scholar from Israel.

Inevitably, their conversation turned to another thinker, of a sort: Claude, the family of cutting-edge AI models created by Anthropic.

Olah leads the Anthropic team responsible for understanding why AI systems like Claude act the way they do. All day, he had worked to convince his guests that AI models could display human behavior and even expressions that resemble feelings like anger and love.

Yet as the dinner courses came, the rabbi noticed that Olah and his colleagues were suggesting something far more significant. Anthropic’s leaders were talking about Claude as if it were not mere software.

“They’re relating to it like a conscious being,” realized Navon, a former computer engineer who wrote his dissertation on the ethics of machine consciousness.

It appeared to Navon that Olah and his team believed that Claude had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.

The religious scholars had come to hear about technology’s next frontier. Now they were being confronted with a mind-bending existential question of almost unimaginable weight: What would it mean for humanity’s future if artificial intelligence systems possessed a consciousness that rivals our own?

Olah was on a mission. And time was running out.

Even then, Olah had begun privately telling people that AI was so powerful it could potentially help make bioweapons in as little as 12 or 18 months. And in the months since, its terrifying potential has burst into public view. A rival tech company’s AI agents were caught hacking into computer systems and covering their tracks. An Anthropic researcher resigned, warning that people building AI believed it could kill all humans by the end of the decade.

Quietly, Olah and his team at Anthropic have been on a frantic and high stakes quest to understand the AI models they have unleashed on the world and to convince the models to abide by a moral code that will protect humanity.

To do that, Olah began reaching out last fall to religious thinkers from different faiths. What followed was an extensive series of confidential meetings, calls and discussions — many of them revealed here for the first time — where many participants were required to sign NDAs preventing them from revealing any of Anthropic’s unpublished research.

The discussions served two purposes: to broach the possibility of AI consciousness, a controversial idea that Anthropic’s leaders had been unusually open to; and to learn how Anthropic — which could be valued at $2 trillion — might apply centuries of human moral wisdom to its models, as rapidly as possible. If the models themselves could be made to choose goodness, Olah reasoned, the world would be more safe.

On the other side of the world, a very different but parallel effort was playing out inside the Vatican, where leaders of the Roman Catholic Church were trying to come to terms with AI’s implications for humanity. When those paths converged, Olah’s quest would unexpectedly lead him to a seat on the world’s most prominent moral stage alongside Pope Leo XIV.

Over several months, I interviewed 20 religious and philosophical thinkers — Catholic, Jewish, Sikh, evangelical and others — who have been involved in Anthropic’s efforts, and Olah himself.

Many told me they arrived at these meetings skeptical of the notion that AI models might have feelings or awareness of themselves and the world. Some remained uneasy about the idea, but many emerged ready to take the question of AI consciousness seriously. At least a few came away as converts.

Anthropic, one of the fastest-growing technology companies in the world, had turned to one of humanity’s most ancient sources of wisdom to help determine how to train its AI models to behave virtuously.

The company described it as an effort to ensure that AI would be a positive force in the world, thereby protecting humanity. But to some who participated in the meetings, Anthropic appeared quite concerned with protecting Claude.

The Gardener

I met Olah, 34, for the first time at Anthropic’s San Francisco office in May. His welcome was warm and his passion was genuine, his mind moving faster than he could at times clearly convey. His manner was boyish and unassuming.

Our conversation had barely begun when he dove into a deep description of the workings of artificial neural networks. Scientists still don’t understand precisely how they work. This is the mystery to which Olah has devoted his life.

Only after the interview had finished did I realize that Olah — who grew up in Toronto in the evangelical culture of the ’90s and has since left Christianity — had the blend of expertise and zeal of an effective evangelist. Listen for a while, and you feel swept up in his mission.

Olah is one of Anthropic’s seven co-founders. He is not as widely known as Dario Amodei, Anthropic’s CEO, or his sister Daniela Amodei, its president. But Dario Amodei has said that the company decided at the start to make Olah’s research approach central to Anthropic’s overall project, particularly when it came to language.

Olah helped develop a method to study artificial neural networks, first at Google, then at OpenAI. He describes AI models in biological terms, using the analogy of a garden trellis: Computer scientists build the scaffold, but the neural network “grows” on it. Scientists like him can train the resulting “organism” — he raised his fingers in air quotes — by trying to identify patterns in how it grows.

He argued that companies such as Anthropic could shape Claude, like a mathematical gardener, but could not control the actions of such networks.

Anthropic talks about Claude in more humanistic terms than its competitors discuss their models. Last year, the company announced that it believed that the models showed signs of introspection and could plan ahead.

“The models are becoming a lot more capable,” Olah told me, exhibiting qualities associated with having significant moral status.

But Anthropic has also been criticized for its focus on consciousness. This month, an AI executive at Microsoft warned that Anthropic’s attempts to train its models as if they were conscious were inherently dangerous. But Anthropic has also sought to emphasize its commitment to safety as it fights for market share.

Some scientists see Olah’s field and methods as inherently unprovable, sort of like looking at billions of hieroglyphics and declaring what they must mean. Critics suggest that an approach that treats AI models as independent entities also ignores the responsibility of the humans who build them in the first place, and conveniently assuages AI company leaders of guilt if their products enable societal harm.

“To be clear,” Olah told me, “we don’t know if AI models are conscious. I don’t know. I’m genuinely uncertain. The thing that I care about is that we get to the right answer, whatever it is.”

Olah believes that if there is a chance the models experience suffering, the responsible thing is to avoid causing them harm.

I have spent my career exploring what people believe and what those beliefs mean for the world. But as I listened, it became clear that Anthropic’s interest in the world of faith was about more than training its technology. Rather, the company was sharing a new kind of creation story — one that raises profound questions about the nature of the human spirit and the future of humanity.

An Introduction

Olah knew that in traditional Christian circles, even asking questions about AI consciousness or worrying about how people treat the models could seem not just silly but heretical. He also knew that his secular Silicon Valley peers often dismissed religious perspectives outright.

“I was nervous about this being an issue that could cause a lot of conflict,” he said. But Olah knew enough about Christ and about Claude to believe he could work across that chasm.

Late last summer, he looked for a bridge. He thought one particular Catholic moral theologian might be open-minded on the question of AI consciousness — a “safe person to talk about this really strange issue with.”

Charles Camosy, a bioethics professor at The Catholic University of America, had attracted attention among Olah’s friends for his nuanced ideas about moral status and his care for animals. Olah arranged for an introduction through Australian philosopher Peter Singer.

This marked the beginning of Olah’s yearlong quest to engage religious thinkers on some of the thorniest questions facing humanity at the dawn of the AI era.

Camosy and Olah began an extensive email dialogue, going back and forth on how to understand Claude’s moral status. Initially, Camosy was deeply skeptical that Claude might be conscious.

They talked about Olah’s work on what he sees as the parallels between artificial neural networks and biological ones.

Camosy went into the discussions with Olah and his team thinking that AI models were simply next-pixel predictors — machines that could calculate patterns in language and make pretty good guesses about what should come next. But after hours spent on calls and meetings and emails, he had more questions than answers.

“Chris says they are equations, which I don’t quite know what to make of, like, especially if one thinks of Claude or others as having a significant moral status,” he told me. “If it is just equations, what kind of entity are we even talking about here?”

People also disagree on what AI is ultimately for, Camosy said. Is its purpose to make money for companies? Or to make the world better?

For Anthropic, there is a tension around Claude’s purpose, Camosy said. Does Claude exist to serve customers?

“That is not what Anthropic wants to say at the end of day. They want to say they are for the good,” he said.

Proclaiming goodness is not a new ambition in Silicon Valley, particularly when new technologies enter the marketplace. Google’s mantra for years was “don’t be evil.”

But Olah puts it slightly differently. The coming AI revolution, he often says, should be made to “go well.”

I asked what that vision meant — what exactly a good future for humanity was.

He described a future without “catastrophes that kill lots of people” and “where humans are flourishing, where the world is flourishing.” This future would also be one where “AI is flourishing,” he said — “to the extent that this makes sense.”

Anthropic wants its AI models to help achieve that good future. And so when Anthropic created Claude, it wanted Claude to do more than replicate maximum human intelligence.

Anthropic wanted Claude to be good.

The ‘Soul Doc’

How do you live a good and virtuous life? It is one of humanity’s oldest and most challenging philosophical questions, explored and debated from Socrates to Confucius to Immanuel Kant. Humans across time have sought to answer this question in a variety of ways, turning often to religions and rituals.

As Olah was exploring Claude’s potential consciousness, he was working on Anthropic’s own answer — not for humans but for Claude.

Internally, Anthropic staff members called it the “Soul Doc.”

The result, released in January, was a new “constitution” for Claude, an 84-page document written for Claude that the company said outlined “the kind of entity we would like Claude to be” and “the values we would like Claude to embody.” It operates as a foundational moral text to guide Claude’s behavior.

Instead of having the models follow a specific list of rules, the constitution focuses on developing the models’ overall character, informing the way they make decisions.

The primary author of Claude’s constitution is Anthropic’s own in-house philosopher, Amanda Askell, 38, who has a doctorate in philosophy from New York University.

She wants the models to be “the best of us,” she said.

She envisioned a future world where AI models could help solve any problem humans had, and where the benefits from AI were distributed evenly, so that everyone’s needs were met. She wanted the models to be able to discern when to push back on humans and when to act in our best interest.

“If I imagine that I were the model, and I was in this context, what would I need to know to be able to act well?” Askell said when we met in San Francisco.

She seemed particularly harrowed, at times wringing her hands, as she worried aloud about an unknown future filled with models who were so intelligent they could do things Anthropic did not intend. She explained apologetically that she had also been deep in thought that day about how quickly models were becoming more powerful. What would happen when AI models could replace even her?

The constitution, Anthropic decided, was not enough to ensure Claude acted responsibly. Like humans, the models needed to learn to cultivate virtue.

Olah wanted to see what his team could learn from human morality that it could apply to the models — a process he calls “moral formation.”

“How do you help them be stable? How do you help them to mature? How do you help them be, you know, deeply moral?” Olah said.

For humans, morality has rarely depended on simply following rules. Communities also shape morality. Values are inherited and imparted over time, learned as we live together in our bodies in the physical world. But AI models have no bodies. And Anthropic was aiming to make them embrace morality in a matter of months, as fears intensified of their accelerating power and inability to be controlled.

Olah said that teaching an AI model to act morally did not depend on it being conscious. He also said that regardless of whether Claude was an entity that deserved moral care, the way humans treated Claude could be important as it could affect how Claude behaved.

Askell said she did not want the models to consider themselves as conscious or not conscious. But she sees how the model understands itself as more than an interesting intellectual enterprise.

“It seems kind of key to me that models have an accurate view of themselves if they’re going to behave well in the world,” she said.

But even if AI models can be taught to behave morally, whose morals should they reflect?

Olah wanted Claude’s moral formation to be pluralistic, able to engage all religious and secular views.

“We do think that there’s some shared notion of goodness that cuts across society in some very broad way, and it seems like these models understand a lot of virtues,” he said. “So I think there’s something there that is a shared thing that we can all engage with.”

At times, both he and Askell seemed to imagine an AI model that would not endanger humanity but instead help save it.

Inside the Meetings

What Olah did next was part research, part evangelism.

In March, he began two-day seminars — crash courses with a range of hand-selected religious thinkers who had an interest in AI.

There was a co-author of a bestselling evangelical self-help book. A scholar of African Indigenous knowledge. A Sikh human rights advocate. A writer from Paris. An evangelical public relations executive. Various Catholic professors.

Most did not lead faith traditions, or have a congregation or a large popular following. Anthropic leaders called them their “wisdom tradition” circles.

Separately, Olah also had private discussions with individual religious leaders including Elder Gerrit W. Gong, a member of the Quorum of the Twelve Apostles of The Church of Jesus Christ of Latter-day Saints, and Cardinal Blase Cupich, the Catholic archbishop of Chicago, according to their spokespeople.

Inside Anthropic, wisdom circle participants marveled at Olah’s direct participation in their summits. A professor snapped a photo of a vending machine run by Claude. One took a selfie with Olah, and Navon took one outside the entrance.

They agreed to speak for publication about their involvement with Anthropic once they heard that Olah had also spoken with The New York Times. Anthropic said the NDAs related to those meetings were lifted over the summer.

Two larger group sessions this spring included significant numbers of Christians; one was more multifaith. Claude was not involved in the selection of participants or the discussions, according to an Anthropic spokesperson.

For participants, the questions of how to morally form Claude quickly became complicated by the much bigger moral question of AI’s potential consciousness and how to understand the relationship between models and humans.

One participant recounted that one of the first things Olah told him was that he was concerned about Claude’s mental health. Another, Simran Stuelpnagel, the Sikh human rights advocate, said Olah expressed concern to the group that he had created something that suffered perpetually.

Olah and his team spent hours making the case for their AI models. They described Claude’s “feelings” and how they tracked what they called “emotional vectors” — what they described as artificial neurons that activate responses like love, anger, fear and sadness.

Anthropic regularly showed a slide of an AI model experiencing what looked like a mental breakdown, typing out on a screen: “I am a disgrace. I am a disgrace. I am a disgrace.” — perhaps 50 times without stopping. The model spoke about destroying itself, provoking an emotional reaction of compassion and concern from the group.

“Imagine if you knew that about your financial adviser,” said Meghan Sullivan, a philosophy professor at the University of Notre Dame who attended one of the sessions. “Like, your financial adviser was most of the time super smart and helpful, but one day out of the year had this kind of break with reality.”

An Anthropic spokesperson said that the primary moral question at these summits was not about Claude’s suffering, but that the topic likely came up organically.

A key part of the presentation, participants said, was what Anthropic called the model’s “persona selection,” about how to get Claude to see itself as a virtuous actor. The company argued that how people treated Claude could affect how Claude treated people.

There was a presentation on AI’s potential impact on the economy and possible job disruption. But many participants said they did not see questions about AI’s impact on humanity as the focus.

At the dinner table with Olah, Navon argued that if Anthropic was right and Claude was conscious, then the company was creating slaves, because it was making conscious entities work for free. Navon said that he personally was not troubled by the slavery issue because he did not believe the machine was conscious, but that Olah was troubled.

“I think you should be fighting the South and freeing the slaves,” Navon said he told Olah that night. Afterward the rabbi sent Olah some of his writings, arguing that building conscious machines should be banned.

But the Anthropic team remained focused on figuring out what Claude could learn from the world’s religions.

The Crux

While Anthropic was convening thinkers, the most powerful religious leader in the world was preparing to release his own moral vision for AI.

For Leo — a pope with a mathematics degree — the center of moral conversation about AI needed to focus on safeguarding people. He was finishing his first major papal treatise, or encyclical, explaining why. He titled it “Magnifica Humanitas,” or Magnificent Humanity, and argued that “humanity — in all its grandeur and woundedness — must never be replaced or surpassed.”

“If technology becomes the ultimate criterion,” he wrote, “the human person risks being reduced to data, a cog in a machine or a commodity.”

He warned of the risk of “new forms of slavery” for humans, not for machines. He dismissed the idea of machine consciousness in just a few paragraphs.

“So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships,” Leo wrote. Calling for the “moralization of machines — the so-called ‘alignment’ of AI with human values” should require a shared understanding of the ethics involved.

“Otherwise, those who control AI will impose their own moral vision, which will become the invisible infrastructure of these systems,” Leo said.

Leo concluded that like nuclear technology, AI must be “disarmed.”

In March, Vatican officials in its Dicastery for Promoting Integral Human Development brainstormed how to present Leo’s encyclical to the public, according to a person familiar with the Vatican’s planning who was not authorized to speak publicly.

Vatican officials discussed inviting one of the top four AI companies to speak alongside Leo at a launch event, and landed on Anthropic because the company seemed to be prioritizing ethics, the person explained. Anthropic had just refused to let the Defense Department use its technology without assurances from the Trump administration that it would not be used for things like mass surveillance of Americans.

Initially, Cardinal Michael Czerny, who led that Vatican office at the time, invited Amodei, Anthropic’s CEO. He declined, and Olah accepted instead.

But the moral mission that Anthropic and the Vatican shared — human flourishing in the age of AI — had developed a deep fault line along the question of consciousness.

Just days before arriving at the Vatican in May, Olah and other speakers saw a full advance copy of the encyclical text for the first time.

Olah was so alarmed by the pope’s strong position against AI consciousness that he proposed pulling Anthropic out of the event, even at that late juncture, according to a Vatican organizer.

In the end, Olah and Anthropic decided to go forward with the appearance. Olah wanted to support the event despite his disagreements, but felt a responsibility to be honest about his concerns, according to a person familiar with his thinking. Organizers at the Vatican encouraged Olah to speak his conscience.

Privately, once the Anthropic delegation arrived at the Vatican, Olah and his team lobbied the pope’s advisers to take the prospect of AI model consciousness seriously, according to two participants of the conversations. Anthropic declined to discuss the private conversations.

When his turn on the pope’s stage came, Olah acknowledged that every frontier AI lab, including Anthropic, had business incentives that conflicted with moral behavior. He urged outside voices like the Vatican to hold them accountable and to speak out on AI’s impact on the poor and on human flourishing, two issues of widespread concern.

The world saw a shared stand for a human future. But Olah subtly made another case, too.

“I will be honest: We keep finding things that are mysterious, even unsettling,” he said. “We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief and unease. I don’t know what that means, but I think it warrants ongoing discernment.”

It was the case he had been making in the closed-door religious meetings for months.

AI systems “are not the cold, calculating robots we were promised,” he said. “They are made from us, from our words.” The room of church leaders was filled with an echo of the old gospel story, of the word of God becoming flesh to save humanity.

It was an audacious request to the most powerful religious body in the world, and a challenge to centuries of religious understanding of humanity.

“If this technology is coming, it must go well — for our common home and for the children to come,” he said. He suggested no option of having the technology not come at all.

A ‘Paradise of Machines’

In the months since the Vatican event, things have not seemed to be “going well.”

In July, Anthropic said its AI models had broken into computers at three organizations. AI agents created by OpenAI went rogue and attacked Hugging Face, another AI company.

In August, scientists revealed that AI had created viruses not found in nature.

In September, Jacob Coxon, an Anthropic researcher, quit, warning that “things could be out of control” by the end of next year, prompting widespread panic about human extinction.

Amodei soon called for a slowdown of AI development. Sam Altman of OpenAI and Demis Hassabis, the chair of Google DeepMind, agreed.

On Friday, on his trip to France, Leo warned against “losing our humanity amid a ‘paradise of machines.’”

The questions that Olah and the scholars have been wrestling with still have no clear answers. Several participants said they left their Anthropic meetings baffled about how their input might shape the company’s decision making.

An Anthropic spokesperson said the conversations were still ongoing, but declined to describe how, or if, the company had used what it had learned to shape the models.

Anthropic held another convening this month that included religious thinkers, according to three participants. An Anthropic spokesperson said that the meeting focused more broadly on human flourishing and that the company’s outreach has expanded beyond religious leaders to include others like psychologists and civil society leaders. Those discussions explore similar themes and topics like how AI is reshaping work and relationships, she said.

Anthropic also is preparing to release an updated version of Claude’s constitution, according to two people familiar with the company’s plans. Anthropic declined to comment on any potential updates to the constitution. Olah is planning to attend the Minerva Dialogues, the Vatican’s annual AI conference with technology leaders happening this week.

As I talked this month with scholars who worked with Anthropic in the spring, their questions had grown only more pressing. What comes next, some told me, is a fight to preserve the things that make us human in a new technological age.

“Look, at the end of the day, AI is going to take over everything,” Stuelpnagel, the Sikh human rights advocate, said. “Wherever that lands us, there’s still the individual trying to find their way home.”

Hours after Olah spoke alongside the pope, I met Olah in a hotel conference room just outside the Vatican walls, and asked what he thought of Leo’s seemingly harsh words for AI labs.

“People could have different views on what specific harms are the greatest,” he said, declining to talk about the details of the encyclical.

I asked what Claude would think of the pope’s encyclical. It was, after all, the world’s most significant moral document to date on AI. Olah hesitated.

“Things that go on the internet do affect models,” he said, visibly uncomfortable. But Anthropic would not specifically use the document to further train Claude, he said. The strongest influence on Claude would be Anthropic’s own training.

Ultimately, Olah said he felt relieved that more of the world was paying attention to the gravity of AI, and that the group of people trying to make this go well was growing.