Network & Infrastructure / Telemetry, Capacity & Incident Analysis
Build an infrastructure incident evidence pack before state changes
An emergency restart restores service but erases the only useful queue, routing and ownership state. The team cannot explain the cause or evaluate whether recovery is durable.
Environment and inputs: Network and infrastructure observability; Prometheus examples describe metric semantics rather than a universal exporter schema. Diagnostic/design example; not tested on production equipment.
Collect the evidence
Collect a bounded snapshot of service symptoms, exact topology context, counters, recent changes and relevant logs with timestamps and uncertainty. Exclude credentials and unnecessary payload data.
Explain the result
Incident evidence should connect the user-visible failure to a testable mechanism. A large unfiltered log archive can be less useful than a small time-aligned record of the right state transitions.
Plan the fix
Define a lightweight collection procedure and prioritize restoration when the service objective requires it. Preserve before/after state for any corrective action so its actual effect can be assessed.
Check the result and rollback
Verify service recovery, remaining redundancy and recurrence signals, then document unresolved hypotheses. Avoid declaring root cause solely because a restart helped; retain the evidence needed to distinguish plausible explanations.