Writing
Decisions from reliability and observability work
I write about problems encountered while operating large-scale systems and the decisions made at the time.
2026
Why SRE Focuses on Incident Prevention During Development
Considering CI checks for known failure patterns so teams can address them before production incidents.
When Is an Incident Prevention Action Complete?
How comparing database outbound traffic after a query change separated applying a fix from checking its effect
Can Service Tiers Capture Dependency Risk?
An incident where a lower-tier dependency stopped a major customer function and prompted a review of request-level dependencies
Why Writes Failed When We Scaled Down Loki
Investigating write failures when scaling down Loki with a replication factor of two and proposing an improvement.
Can Ticket Counts Measure Team Performance?
Why completed cost improvements left the team goal unmet and how that should guide the next priority
A Progressive Deployment Policy Based on Customer Impact
How requiring progressive deployment for roles that could directly affect customers reduced rollbacks after full rollouts
What Should AI Do During an Incident?
What an Incident Agent developed by a member of my team reveals about gathering incident context and validating AI analysis
Can Public Metrics Show a Database's Capacity Limit?
What I learned about the hidden limits of a managed database when load-test headroom did not hold under production traffic
How Should We Build SLOs for an Observability Platform?
Observability platform SLOs based on log and metric query success rates and response times
Should SRE Teams Solve Incidents for Service Teams?
How I separated incident responsibilities between a central SRE team and service teams to turn individual response work into organizational capability
Comparing Incident Response Times Starts with Consistent Records
How we aligned occurrence, acknowledgement, and recovery timestamps to compare incident response times
Does Keeping Logs Longer Actually Improve Reliability?
Why we needed different retention periods based on purpose and audit requirements instead of one policy for every log
What I Learned Operating a Log Platform at Hundreds of Terabytes per Day
Why we chose Loki for a large log platform and used buffer utilization to adjust autoscaling and cost
Why Recovery and Root Cause Analysis Should Be Separate
Why incident response should reduce customer impact first and finish root cause analysis after recovery
When Does a Batch Failure Become an Incident?
How to distinguish a failure from an incident and match the organizational response to actual impact