Network & Infrastructure / High Availability & Recovery Design
Connection draining for services with long-lived sessions
A maintenance drain stops new requests but leaves persistent sessions indefinitely active. An immediate shutdown then interrupts work despite a dashboard showing the backend removed from new selection.
Environment and inputs: Routed and clustered services; define failure ownership, fencing and application recovery requirements before testing. Diagnostic/design example; not tested on production equipment.
Collect the evidence
Inventory connection lifetimes, protocol multiplexing, active work and the proxy's drain semantics. Measure whether new work can still arrive over an already-established connection.
Explain the result
Removing a backend from new connection selection does not necessarily stop new requests on existing sessions. Drain behavior is protocol- and implementation-dependent.
Plan the fix
Use the application's supported graceful shutdown and proxy drain mechanism with a documented deadline. Define what happens to sessions that cannot finish before the maintenance window ends.
Check the result and rollback
Verify new-work admission stops, in-flight work completes and deadline behavior is acceptable. Abort shutdown if essential work remains outside the agreed interruption policy, restoring readiness if maintenance is deferred.