AI Agents / Agent evaluation and observability
AI Agents: Latency percentiles
Track end-to-end p50 and p95 by task type alongside success rate.
Environment and inputs: Framework-independent agent evaluation; OpenTelemetry illustrates trace correlation.
When to use it
Choose an observable task outcome before optimizing a model or prompt. Keep inputs and scoring rules stable enough to compare candidates, and inspect critical failures individually rather than hiding them inside an average quality score.
Practical example
Small Python example using synthetic durations. A production benchmark should use substantially more observations and state its percentile method.
from statistics import median, quantiles
seconds = [0.9, 1.1, 1.2, 1.4, 1.5, 1.7, 1.9, 2.2, 2.8, 4.0]
print('p50', median(seconds))
print('p95', quantiles(seconds, n=100, method='inclusive')[94])Check the result
Compare a new prompt on the same workload and check tail latency as well as average.
Save and compare results
Save the test input, configuration and result. Fix the failing boundary, repeat the same test and compare it with the saved working configuration.
Scope and limits
These exercises have not been tested on your systems. Check feature support and configuration names against the installed release and linked documentation.