Writing on reliability, observability, and engineering leadership https://17earlgrey.pages.dev/en/ en-US Considering CI checks for known failure patterns so teams can address them before production incidents. https://17earlgrey.pages.dev/en/writing/moving-incident-prevention-into-development/ https://17earlgrey.pages.dev/en/writing/moving-incident-prevention-into-development/ Tue, 13 Oct 2026 00:00:00 GMT How comparing database outbound traffic after a query change separated applying a fix from checking its effect https://17earlgrey.pages.dev/en/writing/verifying-incident-prevention-actions/ https://17earlgrey.pages.dev/en/writing/verifying-incident-prevention-actions/ Mon, 05 Oct 2026 00:00:00 GMT An incident where a lower-tier dependency stopped a major customer function and prompted a review of request-level dependencies https://17earlgrey.pages.dev/en/writing/service-criticality-and-dependency-risk/ https://17earlgrey.pages.dev/en/writing/service-criticality-and-dependency-risk/ Fri, 02 Oct 2026 00:00:00 GMT Investigating write failures when scaling down Loki with a replication factor of two and proposing an improvement. https://17earlgrey.pages.dev/en/writing/loki-write-failures-during-graceful-scale-down/ https://17earlgrey.pages.dev/en/writing/loki-write-failures-during-graceful-scale-down/ Wed, 23 Sep 2026 00:00:00 GMT Why completed cost improvements left the team goal unmet and how that should guide the next priority https://17earlgrey.pages.dev/en/writing/measure-sre-outcomes-not-activity/ https://17earlgrey.pages.dev/en/writing/measure-sre-outcomes-not-activity/ Mon, 14 Sep 2026 00:00:00 GMT How requiring progressive deployment for roles that could directly affect customers reduced rollbacks after full rollouts https://17earlgrey.pages.dev/en/writing/a-reliability-policy-needs-enforcement/ https://17earlgrey.pages.dev/en/writing/a-reliability-policy-needs-enforcement/ Fri, 11 Sep 2026 00:00:00 GMT What an Incident Agent developed by a member of my team reveals about gathering incident context and validating AI analysis https://17earlgrey.pages.dev/en/writing/what-should-ai-do-during-an-incident/ https://17earlgrey.pages.dev/en/writing/what-should-ai-do-during-an-incident/ Wed, 09 Sep 2026 00:00:00 GMT What I learned about the hidden limits of a managed database when load-test headroom did not hold under production traffic https://17earlgrey.pages.dev/en/writing/can-public-metrics-show-database-capacity-limits/ https://17earlgrey.pages.dev/en/writing/can-public-metrics-show-database-capacity-limits/ Sun, 06 Sep 2026 00:00:00 GMT Observability platform SLOs based on log and metric query success rates and response times https://17earlgrey.pages.dev/en/writing/how-to-build-slos-for-an-observability-platform/ https://17earlgrey.pages.dev/en/writing/how-to-build-slos-for-an-observability-platform/ Thu, 03 Sep 2026 00:00:00 GMT How I separated incident responsibilities between a central SRE team and service teams to turn individual response work into organizational capability https://17earlgrey.pages.dev/en/writing/should-sre-teams-solve-incidents-for-service-teams/ https://17earlgrey.pages.dev/en/writing/should-sre-teams-solve-incidents-for-service-teams/ Mon, 31 Aug 2026 00:00:00 GMT How we aligned occurrence, acknowledgement, and recovery timestamps to compare incident response times https://17earlgrey.pages.dev/en/writing/an-incident-dashboard-is-not-the-first-thing-to-build/ https://17earlgrey.pages.dev/en/writing/an-incident-dashboard-is-not-the-first-thing-to-build/ Sat, 29 Aug 2026 00:00:00 GMT Why we needed different retention periods based on purpose and audit requirements instead of one policy for every log https://17earlgrey.pages.dev/en/writing/does-longer-log-retention-improve-reliability/ https://17earlgrey.pages.dev/en/writing/does-longer-log-retention-improve-reliability/ Wed, 26 Aug 2026 00:00:00 GMT Why we chose Loki for a large log platform and used buffer utilization to adjust autoscaling and cost https://17earlgrey.pages.dev/en/writing/operating-a-log-platform-at-hundreds-of-terabytes-per-day/ https://17earlgrey.pages.dev/en/writing/operating-a-log-platform-at-hundreds-of-terabytes-per-day/ Sun, 23 Aug 2026 00:00:00 GMT Why incident response should reduce customer impact first and finish root cause analysis after recovery https://17earlgrey.pages.dev/en/writing/separate-recovery-from-root-cause-analysis/ https://17earlgrey.pages.dev/en/writing/separate-recovery-from-root-cause-analysis/ Fri, 21 Aug 2026 00:00:00 GMT How to distinguish a failure from an incident and match the organizational response to actual impact https://17earlgrey.pages.dev/en/writing/when-does-a-failure-become-an-incident/ https://17earlgrey.pages.dev/en/writing/when-does-a-failure-become-an-incident/ Tue, 18 Aug 2026 00:00:00 GMT