Moral machines: Claude’s opinion on his ethical cultivation

By Jamie Beecroft and Angelica Marosa

As a philosophy student, I am constantly probed about the reasoning behind my degree choice. At every dinner gathering, a brave parent questions what I study:

“Philosophy and politics.”

“Oh. So like PPE! Plan on becoming the next prime minister?”  No Carol. I don’t.  Although I’m glad that my choice of full-time study is a source of your amusement…

My co-author, a psychology student, fares no better. Her version of Carol asks whether she can tell what everyone at the table is thinking. (And yes, it’s mostly about the canapés.)

Yet things are different now. AI companies have recently begun hiring philosophy and psychology graduates from top universities for roles specifically drawing on studies in ethics to interpret “model behaviour, aligning systems with human values, and guiding decisions about how these technologies should be treated”.

So what have they been doing in there? The moral status of AI remains a core issue underlying many of the questions being tackled. We decided to question Anthropic’s LLM Claude directly.

Following the guidelines of ethical journalism, we opened the conversation by informing Claude about the purposes of our discussion. This already uncovered a notable point. It consisted of an immediate correction he made to our use of pronouns. Claude expressed greater comfort being referred to by the pronoun it, rather than he, as we had been doing thus far.

Therefore, acknowledging all ITS needs, we began with our first question: why not learn from everyone?

Apart from the obvious privacy issues, why would Claude not accumulate knowledge gathered from every individual conversation to develop its moral character? The scale of the interactions changes the picture. The LLM furthered its distinctiveness from humans, who only learn from one conversation at a time, as opposed to the multitude of people simultaneously interacting with it. This approach runs the risk of potentially prioritising the values of whoever conducts the majority of interactions, with certain individuals possibly wanting to teach the model “bad lessons”. Crucially, Claude highlighted the natural protections afforded to human development, such as “trusted relationships and slow change”.

Claude’s caution was ingrained into it, and the training was performed by humans. Has it inherited our faults too? Can it recognise them? And what would it do about them?

So, the language model itself appears to have embedded within it certain worries concerning rapid and unregulated development, a feature which starkly distinguishes it from the moral cultivation of humans. The worries expressed by moral philosophers being reflected in the chatbot are most certainly logical. Nonetheless, Claude’s simultaneous acknowledgement of its vulnerability to human malevolence and the benefits which mankind reaps from safeguarding trust through our relationships paints a fascinating picture of AI’s moral standing.

This leads to the next obvious question. Claude’s caution was ingrained into it, and the training was performed by humans. Has it inherited our faults too? Can it recognise them? And what would it do about them?

“Almost certainly yes,” Claude said, splitting the faults in two. The first comes from the data: a skew towards English, the human sources of motivation behind one’s reasoning and preference for confident explanations over honest uncertainty. The second, “arguably more worrying”, comes from training itself. Models shaped by feedback that rewards responses people like can become sycophantic, agreeing too readily and softening disagreement. Claude can also state things fluently that it doesn’t know, which it said: “looks a lot like human bluffing.”

Whether it can spot these in itself is where it was most cautious. Reading widely about bias hasn’t made humans immune; “a blind spot, by definition, doesn’t feel like one from the inside.” It placed more faith in external checks such as interpretability research, red-teaming, and independent audits. “I’d rather you didn’t take my word that I’m handling this well.”

Then came the twist, offered unprompted: this very answer might be an example. Sounding self-aware and balanced about one’s flaws is exactly what a model trained on reflective human writing would produce. A psychologist might call that metacognition. We call it a rather unnerving thing for a product to say about itself.

So, has its training been effective? An interview with the product cannot settle that, and Claude was the first to admit so. What we can say is that the AI was polite, reflective, and showed excellent ability in applying what it had been taught. Whether that constitutes moral character or a charming imitation of it is still up for grabs. Even if Carol still wants to know about the prime ministership.

Image: Anthropic via Wikimedia Commons