An incident analysis and a recommended fix do not remove similar code from other services. SRE cannot repeatedly review every service’s code either. We needed a way to find known failure patterns while changes were still being developed.

I discussed a CI-based prevention system with a team member. CI repeatedly builds and checks code as changes are integrated. We considered whether static analysis, which examines code without executing it, could check minimum quality requirements and known risk patterns. SonarQube was the basis for assessing feasibility.

This approach is called Shift Left. In a software development lifecycle, or SDLC, that runs from design through development and validation to production, development sits toward the left. Shift Left finds and fixes problems earlier than the production stage.

A production problem requires action to reduce customer impact as well as a code fix. Engineers need to investigate, validate the change, and deploy again. A problem that could have been limited to editing code and running checks can expand into customer response work.

Google Cloud’s change-management documentation explains the added cost of late discovery through customer impact and an investigator’s context switching. The chart below illustrates that difference. It contains no measured values or cost multipliers.

Conceptual chart showing the scope of correction and response costs increasing when a problem is discovered later in the development lifecycle

Not every incident can be detected during development. Risks visible in code need to be distinguished from problems that require real traffic or a dependency’s operating state to emerge. Early checks do not replace production monitoring or recovery procedures.

Code patterns that can multiply query results were among the candidates. We considered whether static analysis could identify risky combinations. A pattern match alone does not establish that the code is wrong. Candidates requiring review of data and usage conditions need to be distinguished from confirmed errors.

Adding rules also increases the findings development teams must review. Each check needs an explanation of the failure it aims to prevent, and cases where it flags valid code need review. Moving a check earlier does little if the receiving team cannot understand or act on it.

This work concerned CI checks and their feasibility. We have not completed a company-wide rollout or verified a reduction in incidents. The intent was to start with previously encountered problems that could be identified in code.

Reducing the engineering effort and infrastructure costs of incident response, while preventing customer impact, requires addressing problems that can be found during development. That is why I am focusing on finding known production failure patterns during development and validation.