Network & Infrastructure / Storage Networks & Distributed Data
Storage queue depth and latency under shared concurrency
A benchmark gains throughput with deeper queues, but production tail latency deteriorates when many clients use the same setting. Aggregate outstanding work exceeds the backend's useful concurrency.
Environment and inputs: Ceph and storage transport dependencies; match health identifiers and recovery behavior to the installed release. Diagnostic/design example; not tested on production equipment.
Collect the evidence
Measure outstanding I/O, service time, queue wait, backend utilization and per-client concurrency. Compare throughput and percentile latency across the actual mixed workload.
Explain the result
Queue depth controls how much work can wait or execute; it does not increase the physical service rate indefinitely. Large aggregate queues can turn short overloads into long application delays.
Plan the fix
Tune concurrency at the appropriate client and storage layer based on measured saturation. Preserve fairness and overload behavior rather than optimizing only a single-client sequential benchmark.
Check the result and rollback
Validate mixed reads/writes, recovery load and peak client count. Restore previous limits if higher throughput comes with unacceptable latency or one workload monopolizes the shared storage path.