AI Agents for On-Call Engineering Teams: Trusted to Investigate, Not Yet to Fix

It is 3 a.m. and a page goes off. The on-call engineer opens a laptop, half awake, and starts the same ritual as always: check the dashboards, scan the logs, look at what deployed in the last few hours, search for the last time this alert fired. Ten or twenty minutes later they finally know enough to decide what to do.

That first stretch of the incident is where AI agents are winning trust fastest. The step after it, actually changing production, is where most teams still say "not yet." New survey data from two observability vendors puts numbers on that line, and it is a useful map for any team deciding how much to hand an agent.

The trust line, in numbers

Grafana Labs and research firm ETR surveyed 300 IT and engineering decision-makers (director level and above) across the United States, Japan, Latin America and Western Europe in June 2026. The 2026 State of Observability report, released October 19, shows comfort with AI falling step by step as the stakes rise:

• 85% are comfortable with AI doing read-only investigation.
• 84% are comfortable with AI recommending a fix.
• 56% are comfortable with AI making code or configuration changes.
• 52% are comfortable with fully autonomous remediation.
• 50% are comfortable with AI deploying changes to production.

Adoption is already well past the experiment stage: 68% have moved AI for observability past investigation into pilots or production. And it is helping the people who carry the pager. 67% say AI tooling has improved the on-call experience, including 28% who say it has significantly reduced on-call burden and alert fatigue.

Riverbed's global survey of 1,200 business decision-makers, IT leaders and technical specialists across seven countries, released October 6, lands in the same place from a different angle. 77% say their organization is hesitant to let AI make operational decisions without human approval, 69% favor human-supervised AI, and only 24% extensively trust AI recommendations with limited human review. On average, just 19% of IT operations are automated today.

Both companies sell observability and AI operations tools, so they have an interest in the topic. The value here is less in any single number and more in how consistent the pattern is across two independent samples: investigation is broadly trusted, autonomous change is not.

Why the line falls exactly there

Reading data is reversible. Changing production is not, or at least not cheaply. An agent that pulls the wrong log or misreads a graph wastes a few minutes. An agent that rolls back the wrong service or scales the wrong cluster can turn a small incident into a large one.

There is also a human reason investigation is the right place to start. Google's SRE book, in its chapter on being on-call, warns that stress pushes on-call engineers away from deliberate reasoning and toward shortcuts, such as assuming an alert has the same cause it had the last three times it fired. An agent that has already gathered the evidence, including the parts that contradict the obvious guess, gives a tired engineer something better to reason from.

The surveys also say what would move the line. In the Grafana data, the top barrier to adoption is that the technology is not yet reliable enough (26%), followed by regulatory and compliance constraints (23%). The two things respondents say would most increase trust are a proven accuracy track record and better explainability, cited by 45% each. Riverbed's respondents name security and compliance concerns (55%), the risk of operational disruption (45%) and lack of trust in AI decisions (39%) as the top obstacles.

Both lists point to the same conclusion: trust is earned with evidence over time, not granted on day one.

A four-rung ladder for on-call agents

A practical way to act on this is to give the agent a ladder and promote it one rung at a time, based on its record.

1. Gather context. When a page fires, the agent pulls the relevant dashboards, recent deploys, error logs and similar past incidents, and posts a short summary into the incident channel before the engineer has finished logging in. This is the 85% zone. Almost no one objects.
2. Propose a fix with its reasoning. The agent suggests a likely cause and a specific next step, and shows the evidence that led there. The engineer decides. This is still the 84% zone.
3. Run pre-approved actions with a click. For a short list of well-understood runbook steps, such as restarting a service, scaling up or rolling back the last deploy, the agent prepares the action and a human approves it. This is where comfort starts dropping toward the mid-50s, so keep the list short.
4. Act alone on narrow, proven cases. Only after the agent has a documented record on a specific action, in a specific system, does it get to take that action without a click. Most teams will have very few of these, and that is fine.

The promotion rule matters more than the ladder. Log every summary, suggestion and action, and review them in postmortems. When an agent's suggestions for a given alert type have been right consistently over many incidents, promote that alert type, not the agent as a whole.

Sign up for our newsletter

Get the latest updates, product launches, and helpful tips delivered straight to your inbox.

Email addressSubscribe

What to measure

Token counts and "AI usage" numbers say little about whether on-call is better. A few measures say much more:

• Time to useful context. How long from the page until the engineer understands what is going on?
• Summary accuracy. In postmortems, how often was the agent's first read of the incident right?
• Rejected suggestions. How often did the engineer override the agent, and why?
• Pager load. Google's SRE practice caps on-call work at no more than 25% of an engineer's time. If agents are helping, that number should become easier to honor.

The two gaps that will slow you down

The first is data. An agent investigating an incident is only as good as what it can see. Riverbed found that while 96% of organizations say visibility across domains is important for AI-driven operations, only 17% have fully unified visibility across networks, applications, endpoints and cloud. Only 21% rate their data quality as excellent. If your logs, metrics and traces live in five disconnected tools, fix that before expecting an agent to connect the dots.

The second is watching the agents themselves. Grafana found that 80% of organizations are now building or operating products powered by large language models, yet 45% have pieced together their AI observability from open-source projects and only 25% use a purpose-built product. An on-call agent is one more production system. It needs its own logs, its own review and its own owner.

What this means for on-call teams going forward

The surveys describe a sensible middle ground, not a stalemate. Teams are letting AI do the part of incident response that is tedious, reversible and easy to check, and holding back on the part that is risky and hard to undo. That is not a lack of ambition. It is how trust in any new teammate gets built.

The teams that move fastest up the ladder will be the ones that treat every incident as evidence: logging what the agent saw, what it suggested and whether it was right, and promoting it one narrow action at a time.

Frequently asked questions

Should an AI agent ever page a human? Yes. One of the most useful early jobs is enriching or grouping alerts so that the human who gets paged starts with context rather than a bare alert.

Is it safe to let an agent roll back a deploy? It can be, for systems where rollbacks are routine and well tested. Start with human approval on every rollback and remove the click only after a clean record.

Who owns the agent? The same team that owns the on-call rotation. If no one is responsible for reviewing what the agent did, it is not ready to touch production.