Can Service Tiers Capture Dependency Risk?

A high-priority service had an incident. Tracing the cause led to a dependency classified at a lower tier. A fallback was supposed to keep the request working when that call failed, but the fallback did not work either.

A service managed at a lower tier had stopped a major customer function. Its tier alone did not reveal how important the requests calling it were to customers.

The existing dependency view grouped relationships by service. It showed that one service called another, but not which requests made those calls.

Functions within the same service have different dependencies. Treating a dependency used by one function the same as a dependency on every request path makes the consequences of failure difficult to judge.

We considered how to examine request-level dependencies starting from a customer function. The view needed to show which services it called and whether the request could continue if a call failed. We also considered a query API so other tools could use the same information. At that point we were checking what could be provided.

Raising every dependency to a higher tier has a cost. It expands the operating requirements and response responsibilities of the owning team. Reviewing customer impact and fallback behavior first helps raise requirements where needed without adding unnecessary work elsewhere.

A fallback can fail under the conditions it was designed to handle, as it did in this incident. If a function can work without some information, the fallback needs to be tested for whether it lets the request continue. If that information is essential, the dependency’s operating requirements need to reflect its customer impact.

In this incident, a lower tier did not mean lower customer impact. A major function calling the service stopped, and the fallback did not protect it. Assigning a tier requires examining which customer functions stop when the service fails, alongside its own role. Reducing the same risk requires either reviewing the tier against that impact or changing the caller to tolerate the failed call.