Network & Infrastructure / Telemetry, Capacity & Incident Analysis
Retry storms after a transient network or infrastructure fault
A short outage ends, but recovery remains slow because clients aggressively retry and overload the restored service. The retry workload can exceed the original demand.
Environment and inputs: Network and infrastructure observability; Prometheus examples describe metric semantics rather than a universal exporter schema. Diagnostic/design example; not tested on production equipment.
Collect the evidence
Record original request rate, retries per request, timeout distribution, backend queues and resource saturation. Correlate retry amplification with the point when basic connectivity returns.
Explain the result
Retries can turn one failed operation into many competing attempts. Without bounded attempts, backoff and suitable idempotency, they can prolong an outage or duplicate state changes.
Plan the fix
Use supported bounded retry behavior, jitter and admission control consistent with request semantics. Coordinate client and proxy retries so independent layers do not multiply attempts unexpectedly.
Check the result and rollback
Simulate a brief outage and verify useful throughput recovers without sustained overload or duplicate effects. Restore previous policy if retries hide errors while increasing total work beyond available recovery capacity.