Write requests failed while we were scaling down Loki. An instance receiving logs had not crashed. The problem was that the write path excluded a departing instance without selecting a replacement.
We keep the replication factor, or RF, at two in our environment. Each log is replicated to two instances. With RF set to two, both write targets are required, leaving no spare target if one is removed.
The minimum replica count needed for a write is called a quorum. Other instances can still be healthy while a particular request lacks enough write targets. In the path we investigated, Loki returned an error when a departing instance left the write set instead of selecting another healthy instance.
This matters when autoscaling removes instances. Reducing capacity after load falls can trigger client write retries. Testing only scale-up misses failures during scale-down.
The proposed change selects another healthy instance to replace the departing one and preserves the required replica set. I submitted Loki PR #23908 to make this behavior available as an opt-in setting.
The proposal leaves the default behavior unchanged and extends the write set only when the setting is enabled. Mimir already exposed a setting for the same purpose.
Sending writes to another instance also requires the read path to find that data. The PR checked the read path’s handling of departing instances and extended targets. Removing write errors would not be enough if the change left logs out of query results.
As of this article’s publication date, September 23, 2026, the PR is under review. This describes the process of proposing an improvement, not a change already accepted into Loki.
Keeping RF at two requires preserving both write targets during scale-down. Alongside checking graceful shutdown, we need to check whether another target can be selected at the moment a departing instance leaves the write set. Otherwise, writes can still fail after load has fallen.