When an incident starts we first collect information spread across several systems. We find the affected service from an alert and check traces and logs. We also compare recent deployments and code changes.
Experienced engineers know this routine. Customer impact still continues while they collect the information.
A member of my team developed an Incident Agent to reduce that time. It gathers information needed to investigate incidents. This article draws on that development and operational experience to examine which tasks AI can handle during incident response.
The agent was configured to collect the context a person needed to make a decision.
When the agent chose its own tools the sequence changed from one run to the next. Some runs started with logs and others started with code. It also selected deployments unrelated to the incident as possible causes.
The rules were then updated to follow the sequence SREs normally use to investigate incidents: start with the alert, check traces and logs, then compare deployment history and code.
The operating procedure had to be clear before AI could follow a consistent investigation sequence. AI could not repeat a decision consistently if people could not explain how they made it.
An early version invented trace IDs, which identify requests, even though those IDs were absent from query results. It also blamed deployments made after the incident. Natural wording made the wrong answers easy to miss.
Rules were added to prevent identifiers absent from tool results from appearing in answers. Changes made after the incident were excluded from cause candidates. The initial support scope was also kept narrow.
The output separated confirmed facts from guesses. It also showed what had not been checked. Operators needed to see the evidence before they trusted the answer.
Google SRE describes troubleshooting as repeated observation and hypothesis testing. It also notes that a general process slows down without system context. Effective Troubleshooting
In one production incident the Incident Agent found a cause candidate before the manual review reached it. In another case it failed to send the request to the right analysis agent.
One correct answer is not enough to entrust an investigation to the agent. Calls and data access must work consistently. A person also needs to see why a run failed.
False-positive cases were incorporated into the rules and knowledge base. The investigation procedure could also be changed without changing code. The sequence changes as the system changes.
AI can suggest a cause but a person decides whether to roll back or add capacity. Judging customer impact and change risk requires service context.
The Incident Agent showed where errors started and connected recent changes with relevant logs. It also reduced the number of hypotheses that people had to check.
Evaluating a tool like this requires measuring the time to the first reviewable hypothesis alongside accuracy. Its evidence also needs to be checked against query results. Reviewing a wrong answer first can delay recovery actions.
What I consider important in this case is reducing the time people spend finding information so they can decide sooner.
That requires a clear investigation sequence and reliable data access. It also requires controls that check the answer.
This agent could gather information in a consistent order only after the rules reflected the SRE investigation sequence.