Data Monitoring / Alerting & Observability Design
Observability: Histogram latency percentiles
A dashboard averages per-instance p95 values into a misleading global p95. Inspect histogram bucket compatibility and the aggregation order.
Environment and inputs: Platform-neutral monitoring design with Prometheus and OpenTelemetry examples; syntax and signal availability depend on the deployed stack.
Failure scenario
A dashboard averages per-instance p95 values into a misleading global p95. Confirm the cause before choosing a fix.
Evidence to collect
Inspect histogram bucket compatibility and the aggregation order. Record the time, affected scope and a healthy comparison interval.
Worked diagnostic example
Illustrative PromQL for compatible classic histogram buckets; units follow the metric definition.
histogram_quantile(
0.95,
sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
)Fix the confirmed cause
Aggregate suitable distributions before estimating the percentile and document approximation limits. Write the expected result first. Change one relevant setting at a time.
Check the result and hand over
Repeat the check for histogram latency percentiles using the same scope and a representative interval. Check that the symptom is resolved, expected records are present and downstream results are correct. If coverage, correctness or latency worsens, restore the recorded configuration through the agreed rollback process.
Scope and precautions
Prefer read-only access and limit queries to the relevant period or dataset. This example has not been tested on your systems. Test changes in an approved environment. Keep credentials and sensitive data out of shared evidence.