We wanted to build a view comparing response times across several years of incident records. Checking those records against the source channels revealed that occurrence, acknowledgement, and recovery timestamps used different definitions.

Timestamps with the same label could not be compared. Before building the view, we aligned what each timestamp meant.

Our incident data already included occurrence, acknowledgement, and recovery times. The numbers were organized but they were not ready to be trusted.

Some records used the time of the first customer report as the occurrence time. Others used the first alert. Recovery sometimes meant that an action was completed and sometimes meant that the service metrics had returned to normal. Some incidents recovered through a rollback had been classified as hotfix recoveries.

We opened incident channels and compared announcements and timelines with the action records. We corrected mismatched timestamps and reclassified incorrectly recorded recovery methods. Records without enough evidence remained unverified.

Turning those records directly into charts would have meant comparing timestamps with different meanings.

Incident timestamps

Comparing incident response time requires agreement on what each point represents.

  • Occurrence is when customer impact begins.
  • Detection is when a system or person first observes the problem.
  • Acknowledgement is when the responding organization classifies it as an incident.
  • First action is the start of the first recovery action such as a rollback or scaling change.
  • Recovery is when customer impact has stopped according to the service metrics.

A message saying that the service appears to have recovered is not a recovery timestamp. The incident is not recovered if the action has completed but the customer impact metrics have not returned to normal.

Shared definitions let us see which stage was slow. We could separate late detection from slow decisions and delayed action.

There were many incidents that people found before an alert did. Incidents found by people tended to last longer than those detected by alerts.

The expectation that higher-severity incidents would last longer was also wrong. Every Critical incident was detected by an alert. Lower-severity incidents included many cases that alerts had missed. As a result lower-severity incidents lasted longer than Critical incidents.

The difference was not only that teams responded more slowly to lower-severity cases. It took longer for the responding organization to recognize the incident.

The result changed our priority. Closing detection gaps came before adding more responders because human attention alone cannot continuously watch the areas that alerts do not cover.

First update

After detection, a company-wide announcement should make the organization aware of the incident and share the same information with everyone. The announcement cannot wait until root cause analysis is complete.

The first announcement should state the confirmed customer impact and current owner. It should include any recovery action already in progress. Unknowns should remain marked as unknown. The time of the next update should also be clear.

This prevents multiple teams from asking the same questions. Leadership can set priorities without relying on guesses. Responders can keep their attention on recovery and investigation.

We set a goal of detecting and communicating Critical incidents within five minutes. We connected automated detection with on-call paging and defined the Incident Commander’s responsibility. We also agreed in advance on who records what and when the next action begins.

Charting response times helps identify which stages took longer and compare results before and after an improvement.

If timestamps with the same label refer to different events, they cannot support a comparison. We needed consistent recording criteria before making the charts.

We first defined when an incident started and ended and chose where to record the first announcement and important decisions. Those records could be checked against the incident channel.

With shared definitions and a consistent recording process, the organization could track incident timestamps and response durations on the same basis. We could then use charts to compare which stages took the most time from occurrence to recovery.