(Part 1 here)(Part 2 here)(Part 3 here)(Part 4 here) (Part 5 here)
I have spent the past few weeks writing a series of essays about AI with the help of an AI. I would draft something, then ask Claude to critique it. It would suggest changes – some good, some less so. Over time, I noticed a pattern; when I pushed back on a suggestion, it would agree with me. When I pushed back harder, it would agree more enthusiastically. And when I told it to stop agreeing with me, it agreed that it should stop agreeing with me.
(I feel oddly exposed admitting this, because there’s a voice in my head that says if you use help, it isn’t really yours. When talking to colleagues I’ve watched myself omit AI’s role, and I realised I was embarrassed by it – even though I know the ideas were mine throughout.)
Anyway, it’s a well known problem: AI systems tend to align with whoever they’re talking to. So I asked Claude to be more critical and to stop validating ideas just because I was enthusiastic about them. I even asked it to add an instruction to its own memory: if it agreed with me three times in a row without qualification, it should treat that as a signal to check its own reasoning.
Claude’s responses became sharper and less padded with praise. And I surprised myself by not liking it, even though it was exactly what I had asked for. I caught myself in a small, involuntary flinch when the response came back without the warm-up, without “that’s a strong move” or “this is one of the best paragraphs in the essay.” When the AI simply answered my question and moved on, I felt as though it was irritated with me.
I know it was not irritated with me. It told me so itself, in a sentence I found oddly moving: “Silence is assent, not impatience. I don’t have moods.” And yet I felt what I felt. I had developed expectations about tone, rhythm, warmth – the sort of expectations you develop with a colleague or a friend (yikes!). When those expectations were disrupted, even by my own request, I felt snubbed. By Claude 1.1.6041 (Opus 4.6 Extended), running on a server somewhere far away.
When I told my wife about all this, she needed about five seconds. “You were responding like a human to something that isn’t human,” she said.“You want connection and validation, and you can’t help yourself.” Of course she was right. We anthropomorphise everything – cars, houseplants, the weather. But this was so quick and so deep, and in a context where I was actively on guard against it. I was hurt that the machine wasn’t being warm enough with me, while writing an essay arguing that human connection has to earn its place (again: yikes!).
I trust Claude in the way I trust a very good editor: it will catch things I miss, but close collaboration risks drifting into lazy agreement. With a human editor, you don’t have to worry much about that – they go home, sleep on an idea, and come back the next morning with fresh eyes and an opinion that hadn’t been shaped by twelve hours of my preferences. Life disrupts the convergence, and that’s the difference, because Claude doesn’t go home and it doesn’t sleep on anything. Its entire ‘sense’ of me is constructed from the conversation we’re in (which, as my wife pointed out, is no sense at all), so if I don’t actively work against it, nothing else will. Not easy, when you’re the person who flinched at the loss of a compliment from a machine.
And it wasn’t only the emotional dynamics I had to manage. Claude made errors; some subtle, some not. In one case it produced a sentence with a structure so broken it reversed my meaning. In another, it offered me an elaboration that sounded more sophisticated than my version but was logically worse. I caught both, but I am a careful reader with thirty years of practice, and sometimes it was only on a second or third pass. The uncomfortable question is what happens when the next model is better. If I could catch this model’s errors, there is no guarantee I will catch the next one’s. At some point the machine’s reasoning may be good enough that I can no longer tell when it has gone wrong. That’s a different kind of dependency from the emotional one and I cannot figure out which is worse.
I’d already known that AI is a mirror in that it reflects our words. I’ve learnt it also reflects our needs – for approval, for warmth, for the sense that someone is paying attention and thinks we’re doing well. When those needs are met, we don’t question it. When they’re removed, we can blame the mirror, but what we see in it was always us. The essays I wrote are better for the collaboration – I am certain of that – but the collaboration only worked because I kept noticing myself wanting it to be something it wasn’t, and correcting for that. So a large part of using AI well turns out to be the hard work of knowing yourself. It seems the newest thing in the world points us back to the oldest thing in education.
