When Does a Batch Failure Become an Incident?

A batch job did not run. No customer impact was confirmed.

Should we declare an incident immediately?

While working on reliability I often had to distinguish system failures from service incidents. Systems can fail. But not every failure affects customers.

A failure means that a system did not behave as expected. An incident means that the failure affected the service we are trying to protect.

Without this distinction the organization responds to every anomaly at the same level. If the standard is too loose the organization can miss important signals.

The criterion is customer impact.

What impact did it have on the user experience or business outcome we are trying to protect?

In our operating procedure we managed anomaly detection as a stage for assessing impact. We kept that stage separate from a low-severity incident.

When an anomaly occurs the first step is to assess its impact. Confirmed customer or business impact turns it into an incident. If there is no impact and a rerun or backfill restores normal operation the event is closed as an anomaly.

Technical failures remain tracked to prevent recurrence. The level of organizational response follows the actual customer impact.

An organization has limited resources and cannot respond to every anomalous state as a whole. Priorities need to be aligned with senior leadership around customer experience. Defining those criteria before an incident helps different responders identify failures that require an organization-wide response.