Reliability · Observability · Engineering Leadership
Building reliable systems is both a technology and an organizational problem.
I write about incidents, observability, cost, and organizational challenges from operating large-scale services. The focus is on decision criteria and outcomes rather than the technology itself.
Why SRE Focuses on Incident Prevention During Development
Considering CI checks for known failure patterns so teams can address them before production incidents.
Read articleA Progressive Deployment Policy Based on Customer Impact
How requiring progressive deployment for roles that could directly affect customers reduced rollbacks after full rollouts
How Should We Build SLOs for an Observability Platform?
Observability platform SLOs based on log and metric query success rates and response times
Comparing Incident Response Times Starts with Consistent Records
How we aligned occurrence, acknowledgement, and recovery timestamps to compare incident response times
When Does a Batch Failure Become an Incident?
How to distinguish a failure from an incident and match the organizational response to actual impact
Why SRE Focuses on Incident Prevention During Development
Considering CI checks for known failure patterns so teams can address them before production incidents.
When Is an Incident Prevention Action Complete?
How comparing database outbound traffic after a query change separated applying a fix from checking its effect
Can Service Tiers Capture Dependency Risk?
An incident where a lower-tier dependency stopped a major customer function and prompted a review of request-level dependencies
Why Writes Failed When We Scaled Down Loki
Investigating write failures when scaling down Loki with a replication factor of two and proposing an improvement.
Can Ticket Counts Measure Team Performance?
Why completed cost improvements left the team goal unmet and how that should guide the next priority
A Progressive Deployment Policy Based on Customer Impact
How requiring progressive deployment for roles that could directly affect customers reduced rollbacks after full rollouts