Incident Analysis: Elevated 5xx and Timeouts Across Multi-Provider Traffic

Incident Timeline (UTC)

• 14:10 UTC
  First anomaly detected: timeout ratio jumped from 0.7% baseline to 6.1% within a single check interval.
• 14:18 UTC
  Confirmed broad impact on model inference endpoints. Degraded
• 14:32 UTC
  5xx burst observed in two regions. Automatic fallback routing policy activated.
• 15:05 UTC
  Provider-side partial recovery announced. Error rates start trending downward.
• 15:42 UTC
  Latency returns near normal for most requests, but residual timeout pockets remain.
• 16:05 UTC
  Stability threshold restored for 3 consecutive checks. Incident moved to resolved state.

Root Cause and Contributing Factors

Primary root cause: transient upstream capacity saturation during overlapping model rollouts and elevated request concurrency.

Contributing factors:

• Retry storms from clients using aggressive timeout thresholds.
• Uneven traffic distribution across regions during the first 30 minutes.
• Insufficient circuit-breaker cooling interval in one routing tier.

Mitigation Steps Taken

• Activated cross-provider failover for high-priority traffic classes.
• Temporarily reduced parallel retries and increased jitter windows.
• Shifted non-urgent batch jobs to delayed queue processing.
• Pinned critical workloads to historically stable model routes.

Preventive Actions and Current Status

• Introduce dynamic retry budgets by region and provider class. Status: in progress.
• Raise alert sensitivity for sudden p95 latency slope changes. Status: completed.
• Add monthly game-day simulation for failover runbooks. Status: scheduled.
• Publish public reliability changelog for major routing policy changes. Status: planned.