After becoming an SRE team lead I was repeatedly asked by senior leadership to provide direction during incidents. Even when we had never seen the service before we were expected to join the channel and assess the impact and say what needed to happen next.
It was difficult to recommend the next action without knowing the code or recent changes. Yet even in that situation SRE was expected to provide direction during the response.
Joining more incident channels still meant spending time collecting service context. Increasing participation alone did not expand the range of support we could provide.
Service teams understand their code and business flow. They know what changed and which functions affect customers.
An SRE team has different information. It can see shared infrastructure and connect signals that appear across services. It is also better positioned to find dependencies and platform bottlenecks that are difficult to see from one service.
When SRE joined an incident channel we first had to check the service architecture and recent changes. Our initial support covered shared infrastructure and narrowing the affected path through logs or traces. Without service context we had limited grounds for recommending the next action.
We participated broadly but the depth of support depended on the service context available to us.
An incident requires more than technical analysis. Someone must determine which customers are affected. Someone must also decide whether to disable a function or return to an earlier version.
The team that operates the service should make those decisions. SRE can inspect logs and traces but cannot take responsibility for business impact and change risk on behalf of that team.
Leaving every incident to the service team was not the answer either. When an incident spreads across services each team has limited visibility beyond its own system. Looking only at service code will not establish the direction when shared infrastructure or the network is involved.
We divided responsibility using the following criteria.
- The service team owns customer impact and service recovery decisions.
- SRE connects shared signals and dependencies and provides evidence for decisions.
- The Incident Commander manages progress and decisions.
I concluded that incident response should not depend on a particular person’s experience or attention. I thought we should address it through systems and processes so that the outcome would not depend on who noticed the incident or joined the response.
We could not address every abnormal condition at once. We assessed business impact and prioritized the problems with the greatest impact.
We configured alerts around those priorities and started with early incident detection. The first step was to let teams begin responding to an alert without waiting for someone to notice the problem.
Incident outcomes should not depend on who is on call.
Priorities should follow business impact and alerts should make early detection possible. Responsibility for recovery and response coordination should be defined before an incident occurs.
We did not only support individual incidents. We also devoted substantial resources to configuring alerts and developing processes and policies so that service teams could start responding first.