Network & Infrastructure / Telemetry, Capacity & Incident Analysis
Why averaging p99 latency across nodes is misleading
A dashboard averages each server's p99 and reports an apparently acceptable fleet value. A busy slow node or a low-volume outlier can make that calculation unrelated to the actual request distribution.
Environment and inputs: Network and infrastructure observability; Prometheus examples describe metric semantics rather than a universal exporter schema. Diagnostic/design example; not tested on production equipment.
Collect the evidence
Inspect metric type, sample counts, bucket definitions and per-instance request volume. Determine whether the system exports aggregatable histograms or already calculated quantiles.
Explain the result
Percentiles are not generally composable by averaging. A global percentile requires suitable distribution data and consistent semantics, not an arithmetic mean of node-level p99 values.
Plan the fix
Aggregate compatible histogram observations using the monitoring platform's supported method and label the scope clearly. Retain per-node views to locate concentrated latency even when the global measure is valid.
Check the result and rollback
Compare the query against a controlled uneven workload and known distributions. Revert if bucket mismatch, missing instances or volume weighting produces a result that does not represent the intended service population.