Local LLM / Local memory and quantization
Local LLM: Weight memory estimate
Estimate parameter storage from parameter count and precision before accounting for overhead.
Environment and inputs: Local inference capacity and quantization; supported precision and offload behavior are runtime- and hardware-specific.
When to use it
Model file size is only part of the memory budget. Measure the serving process with realistic context lengths and active sequences, and keep a quality baseline so a memory saving does not silently change required task behavior.
Practical example
Lower-bound example for raw weight storage only. Quantization metadata, runtime workspaces and context caches add memory.
parameters = 8_000_000_000
bits_per_weight = 4
raw_bytes = parameters * bits_per_weight / 8
print('Raw weight GiB:', raw_bytes / 1024**3)Check the result
Compare the estimate with actual resident memory and record the unexplained difference.
Save and compare results
Save the test input, configuration and result. Fix the failing boundary, repeat the same test and compare it with the saved working configuration.
Scope and limits
These exercises have not been tested on your systems. Check feature support and configuration names against the installed release and linked documentation.