Operators of an observability platform often hear that logs are slow or that metrics are missing.

Users can experience failure while every platform server is running. Delayed ingestion can prevent someone from finding logs from a recent incident. A successful query is not useful for incident response if the result takes too long to arrive.

We decided to build service level objectives (SLOs) around whether users could query logs and metrics rather than around server uptime.

We first monitored collectors, storage systems, and query components. If each component was healthy and errors were low we considered the platform healthy.

In a distributed log pipeline the status of one component cannot establish end-to-end success. The platform may accept an ingestion request while a downstream buffer delays availability. A query can also succeed while returning incomplete data.

The same was true for metrics. Storage could succeed while the query path timed out. Average response time could look healthy while users running heavy queries continued to fail.

We needed to separate internal component health from user success.

Ingestion and query metrics

The write path and read path have different characteristics for both logs and metrics. Combining them into one availability score makes it difficult to see which path failed.

We applied the following questions to the log platform.

  • Was the submitted log collected without loss?
  • Did it become queryable within the expected time?
  • Did the log query succeed?
  • Did the result arrive within a time the user could accept?

We also separated ingestion success from query success for metrics. The query path included both success rate and response time.

A success rate alone misses slow requests. Response time alone can make fast failures look healthy. We needed both to determine whether a user had completed a query.

Query load on an observability platform is not evenly distributed. Most queries cover a short time range while a smaller number scan much wider ranges.

Average response time is dominated by the large number of light queries. It can remain healthy while some users wait for tens of seconds.

We used P99, the latency at which 99% of requests finish, as the primary signal for log queries. When we reduced a range above 20 seconds to single-digit seconds the number of user reports also fell. The average alone could have hidden that delay.

We did not set the P99 target as low as possible. Serving every wide-range query within the same time would require substantial capacity. The target was based on providing the speed needed for incident response across common query patterns.

Caller information

Our first metric showed the outcome but did not tell us what to change. Knowing that response time had increased did not tell us whether to change queries or add capacity.

The missing caller identity for heavy queries was a particular problem. We could not tell which service or user repeated the same pattern.

We changed an internal component to record caller information. We could then group slow queries by pattern and work with users to adjust ranges and filters. We could also distinguish capacity problems from queries that needed to change.

To investigate an SLO miss we needed to find the failed or slow requests again. Caller information helped us check what those requests were used for.

We shortened the measurement interval to one hour so the current service level indicator (SLI) was visible. When a target was missed we could immediately see whether success rate or response time had degraded and on which path.

The report needed a weekly availability number. Operators needed a threshold that caught query failures before users reported them. We used the same SLI at intervals that matched each purpose.

An observability platform exists to inspect other systems. When it fails the ability to investigate an incident shrinks with it.

Checking whether servers are running is not enough. Logs must arrive on time. Queries must succeed and return within the time in which they are needed. Metrics require the same separation between ingestion and queries.

When reviewing an SLO, we check one final question.

Can we explain what the user becomes unable to do when this metric degrades?

If we cannot answer it the metric is probably describing an internal component. When it connects to a user action the SLO can guide an operational priority.