We reviewed database capacity before an expected increase in traffic. The database handled twice the normal request volume in load tests so we considered that enough headroom.
The database failed under production traffic. The public metrics did not show that it had reached a limit. After the incident the vendor confirmed saturation through an internal metric.
The information available to us made it difficult to assess the remaining capacity.
A load-test result is valid for the conditions reproduced by the test. The same request volume can create a different load when it arrives steadily or in short bursts.
Production traffic included micro-bursts. The averages appeared to leave enough headroom while the database crossed its limit during those short periods.
We interpreted the tested request volume as production headroom.
After the incident we reviewed both request volume and traffic shape. We also checked peak load over short intervals.
Managed databases publish metrics such as network traffic and CPU. Low readings alone did not establish that we had room before an internal limit.
AWS tells Aurora users to add NetworkThroughput and StorageNetworkThroughput and compare the result with the network limit of the instance class. Public values must be combined to estimate whether an instance is approaching its limit. Evaluating Aurora DB instance usage
The vendor had an internal metric that confirmed saturation. We could not use it in our dashboards or alerts.
We cannot alert on a metric we cannot observe. We also cannot treat public metrics within their usual range as proof that enough headroom remains.
A managed database reduces the work required to operate the infrastructure. Running our own database hardware was impractical at our scale and cost.
Operational visibility is limited to what the vendor exposes. Diagnosing an internal state can require vendor support.
We had to include this constraint in capacity decisions. Before important traffic we needed to ask the vendor which limits were not observable.
We reviewed the scale-out criteria after the incident. Average CPU or throughput could miss micro-bursts.
We needed to compare database metrics with application latency and request failures from the same period. Requests can slow down or fail even when the internal limit is absent from database metrics.
Managed services do not expose every internal state. Capacity decisions based only on visible numbers can repeat the same failure.
We began examining application latency and request failures from the same period in more detail.