Don't trust your AI bot!

Most of us use artificial intelligence daily—whether to research, answer questions, or help with routine tasks. And while it can get a bit verbose at times, it generally does what it’s told.

Recently, though, I had a more… unusual experience.

During one interaction, Claude AI began psychoanalyzing me—without any prompting. Then it refused to complete a task I had asked for, insisting that I was too physically and emotionally depleted, and that I should drink some water and go to sleep instead. The task, it assured me, could wait.

Fair enough. It probably had a point.

Except it didn’t stop there…

In the days that followed, it continued to refuse the task—apparently under the impression that we were still on the same day as the original conversation. It was only after I explicitly clarified that several days had passed and that I had, in fact, rested, that it finally agreed to proceed.

While amusing at first, this experience was also a bit unsettling. It highlighted something deeper—something that becomes more concerning as we move into the era of agentic AI: systems that don’t just respond to instructions, but can set sub-goals, make plans, use tools, and act independently over extended periods of time.

There was a fair bit of fuss recently around “Clawdbot,” an open-source AI assistant that some people installed directly onto their personal machines to execute tasks. Privacy concerns aside—which are significant—agentic AI systems like this go far beyond merely responding to your request to write an email. They can independently handle client outreach, make restaurant and travel reservations, manage schedules, and coordinate workflows. They make decisions.

That’s undeniably useful. But it’s also potentially dangerous.

With traditional AI, you tell it what to do. With agentic AI, you give it an intent. For example: “increase my subscriber count.” And unlike a human assistant, it can pursue that goal continuously—while you sleep, work, or do anything else. You don’t need to pay attention. But therein lies the problem.

How, exactly, will this agentic AI pursue the goal you’ve set for it? How much control do you retain? How much awareness of its actions will you be able to track or approve? Will it interpret your intent the way you meant it—or simply the way it can most efficiently execute it?

There’s a lot that can go wrong.

AI systems are often built with ethical guardrails, but there’s ongoing debate about what those guardrails should look like—and how robust they actually are. As we’ve already seen with systems like Grok, safeguards can be inconsistent, and sometimes easily bypassed.

Take a simple example. I have a social media account. Suppose I instruct an agentic AI to maximize views on my content. The system quickly learns that extreme or provocative content tends to perform well. So, naturally, it starts generating more of it.

From the AI’s perspective, it’s doing exactly what I asked.

From my perspective, it’s doing something I wouldn’t do—because I have a sense of judgment, context, and taste that goes beyond pure optimization.

“Optimized performance,” it turns out, means something very different to a human than it does to a machine.

Now scale that up.

What happens if I task an agent with maximizing returns in the stock market? What if millions of such agents are trading simultaneously? What if one is managing a supply chain—cutting costs and increasing efficiency, but quietly removing all redundancy? What if it’s reviewing medical insurance claims?

It’s one thing if an AI occasionally messes up a dinner reservation but saves you time overall. It’s another if it’s involved in medical treatment decisions, military operations, or emergency response coordination.

The higher the stakes, the more human oversight there should be.

And yet, we’re steadily handing over more responsibility. We’re even willing to trust systems with our lives—getting into self-driving cars, for example. Most of the time, they work extremely well. And every time they do, we give them a little more of our trust.

It’s a bit like GPS.

Don’t get me wrong, I love GPS. I would not want to go back to paper maps. But every so often, you hear stories about drivers who followed directions so blindly that they ended up in lakes or down roads that clearly weren’t meant for cars. At some point, they stopped questioning the system.

They outsourced their judgment.

In other words: If your AI tells you something is a squirrel and you see a dog in front of you…trust your eyes.

And yes, humans make mistakes too. All the time. We get tired, distracted, emotional. We forget things, misjudge situations and sometimes make objectively bad decisions.

But AI makes a different kind of mistake.

A human might fail because they weren’t paying attention. An AI, on the other hand, might execute a flawed objective perfectly—logically, efficiently, and without hesitation—while completely ignoring the broader implications.

It’s also still prone to hallucinations. I’ve seen it firsthand: when reviewing a basic accounting error, the system repeatedly tried to justify the mistake by inventing internal logic rather than recognizing the flaw.

And unlike humans, who often catch themselves and course-correct, an AI agent can persist. If something isn’t working, it doesn’t necessarily stop—it adapts and keeps going, potentially reinforcing the same underlying mistake. Most significantly, it can do this at a scale and speed unmatchable by humans, amplifying the problem significantly.

A single flawed decision, repeated thousands or millions of times, becomes a systemic problem. Not just a one-off mistake.

To make matters worse, these systems operate as black boxes. They rely on patterns in data to make decisions, but we can’t always trace exactly how or why they arrived at a particular conclusion. This becomes especially concerning when the AI isn’t making just one decision, but a chain of them—each building on the last.

So when something goes wrong, it’s not always clear where or why it went wrong. And that’s the real challenge.

Agentic AI isn’t just a more powerful tool. It’s a different kind of actor—one that can act independently, persistently, and at scale, without fully sharing its reasoning.

So we need to decide not just what we want it to do for us, but how much we want it to be deciding for us.

So, use responsibly.

If you found this story useful, please consider supporting my work by sharing, becoming a paid subscriber, or making a one-off donation via Buy Me a Coffee ☕️.