Network & Infrastructure / Storage Networks & Distributed Data
Recovery traffic competing with production storage latency
After an OSD failure, client latency rises sharply while the cluster rebuilds protection. Recovery is necessary, but its resource demand can exceed the service's spare capacity.
Environment and inputs: Ceph and storage transport dependencies; match health identifiers and recovery behavior to the installed release. Diagnostic/design example; not tested on production equipment.
Collect the evidence
Measure recovery/backfill progress, client latency, device utilization, CPU and network queues. Record the installed release's scheduler and supported controls rather than assuming older settings have identical meaning.
Explain the result
Recovery and foreground I/O share finite resources. Slowing recovery may help clients but lengthens the exposure window, while maximizing it can overload the surviving cluster.
Plan the fix
Choose a release-appropriate recovery policy from the business risk and measured resource bottleneck. Address insufficient capacity rather than permanently reducing protection work to conceal overload.
Check the result and rollback
Validate client objectives and time to restore the desired redundancy during a controlled exercise. Revert if tuning stalls recovery or creates unacceptable application latency without the expected risk reduction.