When an incident occurs the team starts looking for the cause. Customer impact continues during that analysis.
A hotfix requires finding the cause and changing the code before validation and deployment. When a rollback is safe, the team can reduce customer impact without waiting for that work.
Recovery time changed depending on when the team acted to reduce customer impact. Finding more details about the cause during the incident did not lead directly to recovery.
A hotfix requires the team to narrow down the cause first. The team then needs to change the code and validate and deploy it again. It also needs to assess whether the new change could introduce another problem.
A rollback returns the system to its state before the problem. It can be executed without a complete explanation of the cause. The outcome is also easier to predict because the previous state has already run in production.
This difference in time matters when customers are affected. Failed requests and delays continue to accumulate while the team waits for a complete explanation.
Analysis is still necessary during an incident. But setting complete root cause analysis as the goal delays the recovery decision.
Analysis during an incident should go far enough to answer one question.
What is the fastest safe action that can reduce customer impact now?
If a recent change is a likely cause the team should first check whether it can roll back. If a resource is saturated the team can gain time by adjusting capacity before tracing the cause to the end. After the action the team checks whether customer impact has actually decreased.
If customer impact does not fall after the first action, the team considers another one. It checks whether failed requests and delays have decreased rather than how much of the cause it can explain.
A rollback can be riskier when a data change cannot be reversed or when the previous version has a serious problem. If an external dependency caused the incident there may be no internal change to roll back.
In those cases the team looks for a reversible mitigation such as limiting traffic or adjusting capacity. The name of the action is not the criterion. The team considers how long it takes to execute and whether it can recover if the action fails.
Recovering first does not end the analysis. After the incident is resolved the team reviews the timeline and metrics again. It verifies the root cause and defines actions to prevent recurrence.
A hypothesis formed during an incident is a decision made under time pressure. After recovery the team must check whether that hypothesis matches the evidence. Otherwise a fast recovery can leave the organization with the wrong explanation.
Once customer impact has stopped, the team continues investigating until it understands the cause.
Within the team we called actions such as rollbacks and capacity increases No Brain Actions. An incident right after a deployment calls for a rollback. High resource load calls for more capacity. We defined these responses in advance so that engineers could act reflexively without having to work out the response again during an incident.
Without that agreement an on-call engineer has to explain the need for the action and wait for approval during the incident. Execution is also delayed when the rollback procedure exists only in documentation and has not been tested. Those delays increase customer impact.
The rollback procedure needs to be tested in advance so that the team can execute it as soon as the agreed conditions are met.
During an incident, what matters most is which action reduces customer impact. Automating rollbacks or capacity increases requires defined conditions for when each action should run.