Network & Infrastructure / High Availability & Recovery Design
Fault injection with a specific hypothesis and stop condition
A broad resilience exercise creates several simultaneous failures without a clear success criterion. The result is disruption but little evidence about which dependency broke the service.
Environment and inputs: Routed and clustered services; define failure ownership, fencing and application recovery requirements before testing. Diagnostic/design example; not tested on production equipment.
Collect the evidence
Define the single failure mechanism, expected system response, affected scope, metrics and rollback trigger before execution. Confirm the recovery path and responsible operators are available.
Explain the result
An experiment is useful when its observation can confirm or reject a concrete hypothesis. Uncontrolled compound faults make causal attribution difficult and can exceed the tested safety envelope.
Plan the fix
Start with a bounded nonproduction scenario and expand only after the expected response is demonstrated. Use the platform's supported failure simulation where available and preserve baseline measurements.
Check the result and rollback
Stop at the predefined limit, restore normal state and verify service and redundancy. Do not proceed to a larger fault if the first exercise leaves an unexplained recovery or consistency failure.