Network & Infrastructure / Storage Networks & Distributed Data
Ceph heartbeat failures and shared-network contention
OSDs are marked down during backup bursts even though their hosts remain powered on. Shared transport congestion or processing delay may prevent timely heartbeat exchange.
Environment and inputs: Ceph and storage transport dependencies; match health identifiers and recovery behavior to the installed release. Diagnostic/design example; not tested on production equipment.
Collect the evidence
Record heartbeat latency, OSD events, interface drops, queue utilization and host scheduling pressure. Compare the same events with backup traffic and any separate public/cluster network paths.
Explain the result
Heartbeat failure indicates lost timely communication, not definitive proof of a dead disk. False down events can trigger additional recovery traffic that amplifies the original congestion.
Plan the fix
Fix transport capacity, traffic scheduling or host pressure based on evidence. Review network separation and supported timing only as part of a complete capacity and failure-detection design.
Check the result and rollback
Repeat a bounded backup workload and an actual OSD failure separately. Restore prior settings if detection becomes too slow or congestion still causes cascading membership changes.