Network & Infrastructure / Storage Networks & Distributed Data
Storage incident timelines when hosts disagree on time
Logs appear to show recovery completing before a failure occurred. Clock offsets between clients, switches and storage nodes can make a plausible incident narrative causally wrong.
Environment and inputs: Ceph and storage transport dependencies; match health identifiers and recovery behavior to the installed release. Diagnostic/design example; not tested on production equipment.
Collect the evidence
Record time-source state, measured offsets and timestamp precision on each system. Correlate sequence numbers or same-point observations where trustworthy clock alignment is unavailable.
Explain the result
Wall-clock timestamps carry uncertainty and can jump during synchronization or VM restore. Comparing them directly without that uncertainty can blame the wrong dependency.
Plan the fix
Repair the approved time service and annotate the incident timeline with known offsets. Preserve original timestamps rather than rewriting evidence to fit an assumed order.
Check the result and rollback
Validate time stability over a representative operational period and compare an intentionally generated event across systems. Revert a source change if it introduces larger offsets or unacceptable clock steps.