AI agents at work: what actually breaks and how to fix it before it costs you

A logistics company in the Netherlands deployed an AI agent in early 2025 to handle supplier communications automatically: reading emails, drafting responses, updating an ERP system, flagging exceptions. Six weeks in, the agent had quietly been marking certain invoices as "resolved" when they were actually in dispute. No one caught it until a supplier escalated. The agent had not hallucinated in the traditional sense. It had followed its instructions perfectly and made a reasonable inference that turned out to be wrong. That distinction matters enormously for anyone building or overseeing these systems.

This kind of story is now common enough to constitute a pattern. AI agentsAI agentsAgentic AI refers to AI systems that pursue goals autonomously by planning, taking actions through tools, and adapting based on results, with minimal step-by-step human direction.View full definition → are in production across finance, legal, operations, and customer service. The technical capability is largely there. The operational readiness, in most organizations, is not.

The shift from copilots to agents: what's actually changing

For most of 2023 and 2024, the dominant use of LLMs in business was assistive: a human asks, the model answers, the human decides what to do with the output. That model is being replaced, at least partially, by agentic architectures where the model takes actions, calls tools, reads from and writes to external systems, and sometimes orchestrates other models to complete a task.

The practical definition of an agent, as used in 2026, is a system that combines an LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → with some form of memory, a set of tools or APIs it can invoke, and a loop that allows it to plan and act across multiple steps without a human in the middle. OpenAI's Operator product, Anthropic's approach with Claude's tool use, and Google's Agentspace all reflect this direction. So does Microsoft's Copilot Studio, which allows enterprise teams to build agents that connect to internal data sources and take actions inside Microsoft 365 and beyond (Microsoft is a vendor here, so their framing of capability should be read with that context in mind).

What has changed most in the last 18 months is the reliability of the underlying models on multi-step reasoning tasks, and the maturation of frameworks like LangGraph and CrewAI that give developers more control over how agents are structured, what they can access, and when they should stop and ask a human.

What has not changed: agents still fail silently. They still struggle with ambiguous instructions in ways that are hard to anticipate. They still lack genuine judgment about when a situation is outside their competence. And they still have no intrinsic incentive to flag uncertainty, unless you build that in explicitly.

What this means for the AI user

The operational question is not whether to use agents. For many workflows, the efficiency argument is strong enough that the question is already settled. The question is how to deploy them without creating a new category of operational risk.

A few things stand out from what practitioners have learned the hard way.

Scope is the most important design decision. An agent with access to read data and draft outputs is a different risk profile from an agent with access to send emails, update records, or execute transactions. The instinct to give agents broad access to make them more useful is understandable, but it compounds failure modes. Start with read and draft, not read and act. Expand only when you have observable evidence that the agent handles edge cases correctly.

Logging and observability are not optional. This sounds obvious and is consistently underbuilt. Every action an agent takes, every tool call it makes, every decision branch it follows should be logged in a way that a non-engineer can audit. If you cannot answer "what exactly did the agent do between 9am and 11am Tuesday," you are operating blind. LangSmith (from LangChain, a vendor) and Weights and Biases both offer tracing tools for this purpose, though the logging architecture should be designed independently of any single vendor's solution.

Human-in-the-loop checkpoints need to be designed, not improvised. The default assumption in most agentic deployments is that humans will catch problems through normal business processes. They won't, reliably. The agent that marked invoices as resolved was not being monitored because no one had defined what monitoring meant in practice. Decide in advance which actions require human sign-off, build that requirement into the agent's workflow, and treat any attempt to bypass it as a system failure, not a feature.

Prompt engineeringPrompt engineeringPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.View full definition → for agents is a different discipline than prompt engineering for single-turn queries. Agents need explicit instructions about uncertainty: when to stop, when to ask, what to do when a tool returns an unexpected result. A system prompt that works well for generating a summary does not transfer cleanly to an agent managing a multi-day workflow.

Before you expand your agent footprint

• MapMapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → every external system the agent can touch, and for each one, ask what happens if the agent makes the most plausible wrong call. Invoice marked resolved, email sent to wrong recipient, record updated with stale data: these are not edge cases, they are the scenarios your design should be stress-tested against.
• Define failure explicitly. An agent that stops and says "I don't know how to handle this" is succeeding, not failing. Build your evaluation criteria to reward appropriate escalation, not task completion rate alone.
• Assign a human owner for every agent in production. Not a team, a person. Someone whose job it is to review logs, spot anomalies, and decide when the agent's behavior has drifted enough to warrant intervention.
• Treat agent outputs in high-stakes domains (legal, financial, compliance) as drafts requiring review for longer than feels necessary. The pressure to remove the human review step will come early. Resist it until you have enough production data to justify it.
• Run periodic adversarial tests: give the agent ambiguous or contradictory inputs on purpose, and document how it responds. This is more informative than benchmarking on clean, well-formed tasks.

The organizations getting the most out of AI agents in 2026 are not the ones that deployed fastest. They are the ones that treated the first deployment as a learning instrument rather than a finished product, and built the observability infrastructure to actually learn from it. That infrastructure is boring to build and unglamorous to maintain. It is also the difference between an agent that scales and one that quietly causes problems for six weeks before anyone notices.