I operate a platform that processes hundreds of terabytes of logs each day. As ingestion grew, we had to prevent log loss while keeping costs under control.

Ingestion and queries

Our platform ingests far more logs than it reads. We rarely query the entire dataset.

The previous Elasticsearch platform indexed the full log content. Indexing costs and storage capacity grew with ingestion. We reduced costs by separating storage according to retention period, but older logs then required a separate restore step before we could query them.

We chose Loki to match this usage pattern. Loki indexes labels instead of log content and stores compressed logs in object storage. Being able to scale the write path and read path independently also influenced the decision.

The trade-off was potentially slower full-text search than Elasticsearch. Engineers had to learn its query language, LogQL, and there were few large-scale operating examples at the time. We accepted those constraints and prioritized cost and scalability.

Autoscaling

Logs pass through collectors into Loki, stay in memory and on disk, and then reach object storage. If a stage slows down, unprocessed logs accumulate in the buffer ahead of it. Once the buffer fills, we start losing logs.

We therefore needed to watch how quickly the buffer filled, not just storage capacity. Otherwise, we could lose the logs needed to investigate an incident at the moment it occurred.

We initially scaled pods with Kubernetes Horizontal Pod Autoscaler (HPA), using average CPU and memory utilization. But a falling average did not always mean less load.

Some ingesters, the components that receive and process logs, restarted after out-of-memory (OOM) failures. The restarted pods held less data and used less memory. They pulled the average down, so HPA did not scale out. The remaining pods stayed under load.

We added the upstream buffer utilization to the scaling conditions. Using KEDA, an event-driven autoscaler, we added pods when log processing fell behind.

Then we ran into excessive scaling. An ingester can take several minutes to become ready, and the buffer kept filling during that time. The autoscaler saw the growing buffer and requested more pods.

We limited how many pods could be added at once and adjusted stabilization periods to suit the metrics. We also connected node autoscaling: fewer pods do not reduce costs if the nodes remain.

New pods had to become ready before the buffer filled. We also needed to avoid adding more pods without waiting for those already starting.

Cost

After the migration to Loki, ingestion reached 1.6 times the previous volume. Compression reduced log size by about 65 percent, and total cost fell by about 23 percent compared with the previous platform.

We did not reduce costs by accepting fewer logs. Scaling ingestion and queries separately and using object storage let us retain the capabilities we needed.

We now ingest hundreds of terabytes of raw logs per day. Following an agreement, we are reducing retention for some logs from one year to six months.

With one-year retention, dividing average daily infrastructure cost by average daily raw ingestion over the same period gives a cost of no more than five cents per gigabyte. Labor is excluded; retention and the query infrastructure are included. This is not the cost of storing one gigabyte for one day.

Public managed-service ingestion rates provide a reference.

Retention periods and query volume can add costs, while volume discounts and contract terms change the actual rates. Our infrastructure cost is lower than these public rates, but excluding labor means this is not a comparison of total operating costs.

Removing spare capacity to save money increases the risk of log loss when ingestion spikes. Keeping capacity fixed at peak load leaves resources idle during normal operation.

Choosing Loki reduced storage and indexing costs, but preventing log loss also required changing our scaling conditions. We no longer size capacity based only on average utilization. We also check whether new pods can become ready before the buffer fills. Before reducing costs, we first check whether that time margin will remain.