valhaCheckout outage traced to one connection pool
Checkout was down for 47 minutes on June 18 because one deploy halved the payment-service connection pool while doubling its connection hold time. Every other system stayed green, which is exactly why it took 22 minutes to find. The fix is a config guard, not a heroic rewrite.
Source 1Grafana, checkout golden signals, 2026-06-18Source 2PagerDuty incident #4211 timeline
The shape of the outage
Checkout errors went from 0.6% to 14% inside eight minutes of the deploy while the platform-wide rate never left its baseline. A failure this local to one service, this correlated with one deploy, should have taken minutes to attribute. The divergence between the two lines is the entire diagnosis.
5xx rate by surface, % of requests, June 18
View source data
| t | Checkout | All other traffic |
|---|---|---|
| 13:50 | 0.4 | 0.3 |
| 14:00 | 0.6 | 0.3 |
| 14:10 | 9.8 | 0.4 |
| 14:20 | 14.2 | 0.5 |
| 14:30 | 11.6 | 0.4 |
| 14:40 | 1.1 | 0.3 |
| 14:50 | 0.5 | 0.3 |
Timeline
Detection was fast; attribution was not. The 13 minutes between the page and the pool discovery were spent ruling out the usual suspects on dashboards that were, correctly, all green.
| Time | Event | State |
|---|---|---|
| 14:02 | deploy 2026.6.18-3 ships payment-service v241 | healthy |
| 14:07 | checkout error rate crosses 2%, first alert fires | degraded |
| 14:11 | on-call paged; platform dashboards green, checkout red | degraded |
| 14:24 | pgbouncer pool for checkout identified at 100% saturation | down |
| 14:31 | deploy rolled back; pool drains in four minutes | degraded |
| 14:49 | error rate back under SLO; incident closed | healthy |
The moment it surfaced
The root cause did not come from a dashboard. It came from asking why the database looked idle while checkout was starving. The answer was in pgbouncer, one layer above every graph we were staring at.
DB CPU is at 30%, no lock pileup, no slow queries. Why is checkout timing out against an idle database?
Because it never reaches the database. The pgbouncer pool for checkout is pinned at 100/100 connections since 14:02: v241 halved the pool to 100 and doubled hold time by moving payment capture inside the transaction. Requests are queueing at the pooler, not the DB.
So the DB graphs were always going to be green. Rolling back.
What we ruled out
Traffic was flat: no spike, no bot surge, checkout volume within 4% of the previous Tuesday. The database was healthy the entire time: CPU under 35%, zero lock contention, p99 query time unchanged. And it was not a bad payment provider, whose latency held at baseline before, during, and after. Three green systems, one saturated pooler between them.
Traffic
Flat. Checkout volume within 4% of the previous Tuesday, no bot surge, no campaign.
Database
Idle. CPU under 35%, zero lock contention, p99 query time unchanged through the whole window.
Payment provider
Healthy. Latency at baseline before, during, and after; every call that reached it succeeded.
The canary ran at 5% traffic for ten minutes. Pool saturation is a function of concurrent held connections, not error rate, so at 5% of load the halved pool never filled, so the canary saw a healthy service and promoted.
pgbouncer saturation was tracked but never alerted: the metric predates the alerting migration and the port was missed. The dashboard existed; nobody was looking at it, because nothing had ever pointed there.
Evidence trail
Everything below is linked for the next reader: the RCA carries the full connection-math walkthrough, the diff shows the two config lines that did it.
Action items
- Platform ships the pool-size deploy guard, blocking any config lowering a pool below trailing 7-day peak concurrency (owner: Iris, June 27).
- Alert on pgbouncer pool saturation above 80% for five minutes, paging the owning service, not platform (owner: Marion, June 24).
- Canary stage gains a concurrency-proportional check so pool-class regressions surface at 5% traffic (owner: Théo, July 4).
- Payment capture moves back out of the transaction; hold time returns to the v240 baseline (owner: pay team, June 25).
Sources
- 1Grafana, checkout golden signals, 2026-06-18
https://grafana.example.com/d/checkout
cited in The shape of the outage · What we ruled out
- 2PagerDuty incident #4211 timeline
https://example.pagerduty.com/incidents/4211
cited in Timeline · The moment it surfaced · Evidence trail