Network & Infrastructure / Storage Networks & Distributed Data
MTU mismatch on a Ceph replication network
Client connections appear normal while replication and recovery are unreliable. The affected traffic may traverse a different VLAN, bond or jumbo-frame path from the client network.
Environment and inputs: Ceph and storage transport dependencies; match health identifiers and recovery behavior to the installed release. Diagnostic/design example; not tested on production equipment.
Collect the evidence
Inventory the actual OSD-to-OSD path, interface MTUs, bond members and size-related drops. Test every required peer path at the intended packet size rather than checking only gateway reachability.
Explain the result
Separate logical networks can have different transport constraints. Small health probes may succeed while larger recovery traffic exposes a size mismatch on one segment or alternate member.
Plan the fix
Apply a consistent supported size budget across the replication path and its redundancy. Do not assume that enabling jumbo frames on the host configures the switch or every intermediary.
Check the result and rollback
Validate replication, recovery and client latency across member and link failures. Restore the previous MTU if any required path cannot support it, avoiding a partially upgraded cluster transport.