Checkout outage traced to one connection pool
Checkout was down for 47 minutes on June 18 because one deploy halved the payment-service connection pool while doubling its connection hold time. Every other system stayed green, which is exactly why it took 22 minutes to find. The fix is a config guard, not a heroic rewrite.
source → Grafana — checkout golden signals, 2026-06-18+1 more
source → Grafana — checkout golden signals, 2026-06-18+1 more
The shape of the outage
Checkout errors went from 0.6% to 14% inside eight minutes of the deploy while the platform-wide rate never left its baseline. A failure this local to one service, this correlated with one deploy, should have taken minutes to attribute — the divergence between the two lines is the entire diagnosis.
5xx rate by surface, % of requests, June 18
View source data
| t | Checkout | All other traffic |
|---|---|---|
| 13:50 | 0.4 | 0.3 |
| 14:00 | 0.6 | 0.3 |
| 14:10 | 9.8 | 0.4 |
| 14:20 | 14.2 | 0.5 |
| 14:30 | 11.6 | 0.4 |
| 14:40 | 1.1 | 0.3 |
| 14:50 | 0.5 | 0.3 |
source → Grafana — checkout golden signals, 2026-06-18
Timeline
Detection was fast; attribution was not. The 13 minutes between the page and the pool discovery were spent ruling out the usual suspects on dashboards that were, correctly, all green.
| Time | Event | State |
|---|---|---|
| 14:02 | deploy 2026.6.18-3 ships payment-service v241 | healthy |
| 14:07 | checkout error rate crosses 2%, first alert fires | degraded |
| 14:11 | on-call paged; platform dashboards green, checkout red | degraded |
| 14:24 | pgbouncer pool for checkout identified at 100% saturation | down |
| 14:31 | deploy rolled back; pool drains in four minutes | degraded |
| 14:49 | error rate back under SLO; incident closed | healthy |
source → PagerDuty incident #4211 timeline
The moment it surfaced
The root cause did not come from a dashboard. It came from asking why the database looked idle while checkout was starving — the answer was in pgbouncer, one layer above every graph we were staring at.
DB CPU is at 30%, no lock pileup, no slow queries. Why is checkout timing out against an idle database?
Because it never reaches the database. The pgbouncer pool for checkout is pinned at 100/100 connections since 14:02 — v241 halved the pool to 100 and doubled hold time by moving payment capture inside the transaction. Requests are queueing at the pooler, not the DB.
So the DB graphs were always going to be green. Rolling back.
source → PagerDuty incident #4211 timeline
What we ruled out
Traffic was flat — no spike, no bot surge, checkout volume within 4% of the previous Tuesday. The database was healthy the entire time: CPU under 35%, zero lock contention, p99 query time unchanged. And it was not a bad payment provider — provider latency held at baseline before, during, and after. Three green systems, one saturated pooler between them.
source → Grafana — checkout golden signals, 2026-06-18
Evidence trail
Everything below is linked for the next reader — the RCA carries the full connection-math walkthrough, the diff shows the two config lines that did it.
source → PagerDuty incident #4211 timeline
One guard, this sprint. Pool limits are still deployable without review — the same two-line change would take checkout down again today. The config guard (block any deploy that shrinks a pool below observed peak concurrency) is small, owned by platform, and closes the class, not the instance.
Action items
- Platform ships the pool-size deploy guard — blocks any config lowering a pool below trailing 7-day peak concurrency (owner: Iris, June 27).
- Alert on pgbouncer pool saturation above 80% for five minutes, paging the owning service, not platform (owner: Marion, June 24).
- Canary stage gains a concurrency-proportional check so pool-class regressions surface at 5% traffic (owner: Théo, July 4).
- Payment capture moves back out of the transaction; hold time returns to the v240 baseline (owner: pay team, June 25).