Reducing log retention from one year to six months lowers storage cost. Deleted logs cannot be recovered.

Retention has a direct effect on cost when a platform ingests hundreds of terabytes per day. We needed to consider cost and the ability to investigate past events together.

We first checked who used each type of log and why. That answer let us separate logs that needed one year from logs that could be deleted after six months.

Error logs are needed to trace what happened to an affected user. An issue reported late may require logs from months earlier.

Informational logs were different. Many were repetitive records of operating state. They helped with recent incident analysis but became less useful over time.

Access records and audit logs needed separate treatment. Incident analysis alone could not determine their retention. Security policies and audit requirements could also apply.

Keeping every log for one year makes the policy simple. It also means paying the same retention cost for data with different purposes. Reducing everything to six months saves money but removes logs that need to remain available longer.

Rare queries do not prove that old logs can be deleted. A log may be needed only once for an audit or to investigate a user issue.

We checked who queried logs older than six months and what they resolved. Query counts informed the retention review. Even rarely queried logs needed to stay if they supported an audit or a user investigation.

We also needed to ask the following questions.

  • Can a metric or trace establish the same fact?
  • Is the raw log required to reconstruct a user issue?
  • Do legal, security, or audit requirements specify a period?
  • Can the smaller set of logs that needs long retention be separated?

Retention periods

We decided not to apply one retention period to every log.

Logs needed to trace affected users remain available longer. Repetitive informational logs have shorter retention. Access records follow the established security requirement. We agreed on audit log retention separately with the responsible organization.

In Loki, logs can be distinguished by the labels used in queries. Retention periods can differ by those labels, which made it easier to apply separate policies to different logs.

We needed criteria that separated logs safe to remove from logs that had to remain available.

Security and audit requirements

A retention change permanently deletes data. It could not end with a storage configuration update.

Before making the change we needed to confirm the retained data with the security organization. The agreement on what to retain and for how long also needed to be recorded in a ticket. The reasoning behind the decision had to remain available for a later audit.

If a platform team reduces retention for cost alone it may not be able to explain the decision when a problem appears. If every log is kept indefinitely in the name of security there is no room to review the cost.

Longer retention increases the period available for investigation. It does not improve reliability at the same rate.

Reliability needs data that can establish what happened during an incident. An audit needs data that proves changes and access over a defined period. When these purposes are not separated the longest requirement tends to apply to every log.

We therefore reviewed queries of older logs alongside security and audit requirements. We retain logs needed for investigations or audits and are reducing retention to six months for logs where a shorter period is acceptable.