System status
Degraded performance 2026-05-07, 03:42–03:56 UTC 14 minutes
Webhook redelivery backlog — EU-west region
Resolved — postmortem published
A worker rollout briefly left the redelivery scheduler with a stale connection pool. Inbound webhooks were unaffected; outbound deliveries queued and drained within 14 minutes once the pool was rotated.
Timeline
- 03:42 UTCA scheduled decision-worker rollout begins in EU-west.
- 03:44 UTCOutbound webhook delivery success rate dips; the redelivery scheduler is holding a stale connection pool from the pre-rollout process.
- 03:47 UTCAlert fires. On-call identifies the stale pool as the cause; inbound webhooks confirmed unaffected.
- 03:51 UTCConnection pool rotated. Queued outbound deliveries begin draining.
- 03:56 UTCBacklog fully drained; delivery success rate back to baseline. Incident resolved.
Root cause
During a worker rollout, the redelivery scheduler retained a connection pool bound to the outgoing process. The new process came up healthy, but the scheduler's pool was not rotated on handoff, so outbound redeliveries queued against stale connections.
Impact
Outbound webhook deliveries to EU-west endpoints were delayed for 14 minutes. Inbound webhooks were unaffected, and no events were dropped — deliveries queued and drained in order once the pool rotated.
Resolution
On-call rotated the connection pool, after which the queued deliveries drained within minutes. No customer action was required and no events were lost.
What we changed
- Connection-pool rotation is now an explicit, asserted step in the worker handoff sequence.
- Added a rollout smoke check that verifies outbound delivery success before the old process is retired.
// system status
Every incident gets a postmortem.
Root cause, mitigation, what changed in the runbook — published within seven days, not buried. The same discipline we apply to the agents, applied to ourselves.
99.98% uptime · 90 days · Postmortems within 7 days · Updated every 60s