Skip to content

Why an AI agent that used to work stops working

AI agents get worse over time without warning: the model is retired, a system changes, the context goes stale. What causes it and how to prevent it.

Léo ConterCEO at Elev5 min read
Line chart of GPT-4 accuracy on the same task, falling from 84% in March to 51% in June 2023. Source: Chen, Zaharia and Zou.
GPT-4 accuracy on the same task: 84% in March and 51% in June 2023. Source: Chen, Zaharia and Zou, 2023.
In short
  • An AI agent does not break like ordinary software: it keeps answering, only worse, and nobody notices until a customer complains.
  • There are four common causes of AI agent degradation: the model is retired or updated, a system connected to the agent changes, the information it consults goes stale and the volume grows.
  • Research from 2025 and 2026 shows that the quality of the context an AI agent receives predicts its reliability better than the model chosen.
  • The solution is to treat the AI agent as a continuous operation: test it with real cases every month, fix what got worse and plan model changes ahead of time.

An AI agent that worked well in March can be making mistakes in September without anyone having touched it. No error message appears and nothing freezes on the screen. It keeps answering, with the same assurance, only worse.

This post explains why that happens and what to do so you do not find out from a customer.

AI agents don't break, they get worse

An ordinary system, when something goes wrong, stops. The invoice is not generated, the screen shows an error, someone calls support.

An AI agent fails in a different way. It always returns an answer. If the information it consults has gone stale, it answers with the stale information. If the system next to it changed a field, it fills in the wrong field. And it does so in the same confident tone as always.

Ordinary automation breaks and tells you. An AI agent gets worse and doesn't.

That is why the decline takes time to be noticed. The first to notice is usually the customer who was given the wrong deadline or the finance team that found a discrepancy at closing.

The four most common causes

The model changes or is retired. AI vendors change models all the time. Anthropic retired Claude Sonnet 4 and Opus 4 on June 15, 2026, and Opus 4.1 on August 5, 2026. The documentation is direct: after the date, calls to the old model fail. OpenAI shut down the Assistants API, used by many agents, on August 26, 2026. Even when the model stays online, a new version behaves differently. A study from Stanford and Berkeley showed that GPT-4 fell from 84% to 51% accuracy on the same task between the March and June 2023 versions.

A system connected to the agent changes. The ERP is updated, the CRM gets a new field, the pricing spreadsheet changes format. The agent keeps reading and writing as before, only now in the wrong place.

The context goes stale. The agent answers based on the documents, rules and tables it was given. When the return policy changes and the document does not, it keeps stating the old rule.

The volume grows. An agent tested with twenty conversations a day behaves differently with two hundred. Longer histories and more mixed information make the model more prone to error.

The problem is usually in the context, not in the model

When an agent starts making mistakes, the common reaction is to switch models. Most of the time, the problem is in what it is given to read.

Two recent studies point in the same direction:

  • Long context makes the answer worse. In July 2025, Chroma tested 18 models, among them GPT-4.1, Claude 4 and Gemini 2.5, and concluded that performance drops as the amount of input text increases, in ways that are hard to predict. A single irrelevant passage in the middle already gets in the way.
  • Context quality predicts reliability. A study published in July 2026 kept the model fixed and varied only the context. With weak context, the agents made up information, used tools the wrong way and ignored rules. The title of the study sums up the idea: the context fails first.

Another study, from May 2026, followed agents for up to 200 consecutive sessions. Even with the model fixed, accuracy fell over time, because the agent's memory grows, gets summarized and gets mixed up. And the behavior tests kept passing while factual accuracy got worse.

In practice, this means a good agent depends on someone taking care of what it knows: up-to-date documents, clear rules and unnecessary information out of the way.

Building got easy. Keeping it running is the work

Today you can put an agent together in a few days with accessible tools. That is why the market is full of projects that worked in the demo and stopped a few months later.

Gartner predicts that more than 40% of AI agent projects will be canceled by the end of 2027, because of high cost, unclear value or lack of risk controls. Building is usually not the problem. With nobody accountable for the operation, the agent keeps losing value until someone turns it off.

What to do so the agent keeps working

  1. Put together a set of test cases. Twenty to thirty real situations from the operation, with the right answer known. It is the yardstick for knowing whether the agent got better or worse.
  2. Run the tests every month. And also whenever a system connected to the agent changes. Compare with the previous month.
  3. Take care of what the agent reads. A document that changed needs to be updated. A document nobody uses anymore needs to come out.
  4. Follow the model calendar. Vendors publish retirement dates in advance. Switch ahead of time, test with the same cases and only flip the switch when the result is the same or better.
  5. Have someone accountable. Someone needs to receive the alert, understand what got worse and fix it. Without that, the rest does not happen.

At Elev, that is how the agents we build work: the build comes together with the operation. We test every month with your company's cases, fix what got worse and change the model when it makes sense. If you have an agent running with nobody watching, or want to build one that keeps working, see how we work with AI for Operations or talk to us.

Sources

Frequently asked questions

Why did my AI agent start making mistakes if nobody touched it?

Because everything around it changed. The vendor updates or retires the model, the system it consults changes a field, the documents it uses go out of date. The agent keeps answering with confidence, only based on something that is no longer true.

How often does an AI agent need maintenance?

The reasonable minimum is one test a month with real cases from the operation, and a review whenever a system connected to it changes or the vendor announces a model change. Anthropic, for example, gives at least 60 days' notice before shutting down a model.

What happens when an AI model is retired?

Calls to that model stop working on the scheduled date. If the agent was not switched and tested beforehand, it simply stops. It happened with several Anthropic and OpenAI models between 2025 and 2026.

Is it worth paying only for the build of an AI agent?

Rarely. Building has gotten cheaper every year; what costs is keeping it running. An AI agent with nobody accountable for its operation tends to get worse every month, and the loss shows up in customer service or in finance, not on the dashboard.

How do I know if my AI agent is getting worse?

With a fixed set of test cases: real questions and tasks, with the right answer known. You run the same set every month and compare the result. Without that, the decline only shows up when someone complains.

Published on September 23, 2026

Want to see this in your operation?

Tell us what only works today because someone remembers. You get a direct read of what AI can already take over and what needs to exist first.

Book a conversation