Local LLM / Local inference performance
Local LLM: Time to first token
Measure request arrival to first generated token separately from total completion time.
Environment and inputs: Ollama, llama.cpp or vLLM serving; tune only features supported by the selected release and hardware.
When to use it
Compare performance with a fixed workload and a documented quality gate. Prompt processing, generation, queueing and model loading contribute different delays; isolate the stage that dominates the user-visible experience before changing configuration.
Practical example
Illustrative timing math using synthetic monotonic-clock readings. Instrument the client to capture these events for real requests.
arrival = 100.0
execution_start = 100.2
first_token = 101.1
completion = 104.8
print('queue_seconds', execution_start - arrival)
print('ttft_seconds', first_token - arrival)
print('total_seconds', completion - arrival)Check the result
Compare short and long prompts to reveal prompt-processing overhead.
Save and compare results
Save the test input, configuration and result. Fix the failing boundary, repeat the same test and compare it with the saved working configuration.
Scope and limits
These exercises have not been tested on your systems. Check feature support and configuration names against the installed release and linked documentation.