Model Quantization Savings Calculator

Quantization stores network weights at reduced numerical precision, cutting memory in direct proportion to the bits used. Model size equals parameters times bytes per parameter, so a 7 billion parameter model occupies about 28 GB in FP32, 14 GB in FP16 or BF16, 7 GB in INT8, and roughly 3.5 GB at 4 bits, with the compression ratio being simply original bytes per parameter divided by target bytes per parameter. Formats are not all clean powers of two: llama.cpp GGUF variants store weights in blocks with a shared scale and sometimes a zero point, so a scheme like Q4_K_M lands nearer 4.5 effective bits per weight than 4. Speed gains come from a separate mechanism. Low-bit inference is faster mainly because token decoding is memory-bandwidth bound, so halving the bytes read per forward pass nearly halves the time, which is why measured speedups often trail the theoretical compression ratio on hardware without native low-precision arithmetic units. Quality is the binding constraint. Perplexity degradation is negligible at FP16 and BF16, typically under half a percent at 8 bits, and roughly 1 to 3 percent at 4 bits depending on the method used. Larger models tolerate compression better, so a 70B model at 4 bits usually outperforms a 13B model at FP16 on the same memory budget. The percentages shown here are representative rather than measured; calibration data, layer-wise sensitivity, and whether the key-value cache is quantized alongside the weights all shift real-world results.

Reduced-precision arithmetic reached neural networks through embedded and mobile work well before large language models existed. Researchers demonstrated 8-bit integer inference for convolutional networks around 2015 and 2016, and the quantization-aware training scheme published by Benoit Jacob and colleagues at Google in 2017 became the basis for integer inference in TensorFlow Lite, establishing the vocabulary of scales, zero points, and per-channel parameters still in use. Applying it to transformers with billions of parameters required handling outlier activations that break naive scaling. Tim Dettmers and co-authors published LLM.int8() in 2022, isolating those outlier dimensions in higher precision so that 8-bit inference held quality at scale, and the method shipped in the bitsandbytes library integrated with Hugging Face Transformers. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh published GPTQ the same year, a one-shot post-training method using second-order information that made accurate 4-bit weights practical. Dettmers followed with QLoRA in 2023, introducing the NF4 data type and enabling fine-tuning of large models on a single consumer GPU. In parallel, Georgi Gerganov released llama.cpp in March 2023 with the GGML format, superseded by GGUF in August 2023, bringing block-wise quantized inference to ordinary CPUs and laptops.