Model Vram Calculator

GPU memory for a transformer is the sum of four terms, and knowing which one dominates tells you what to change. Weights cost parameters times bytes per parameter: 4 bytes in FP32, 2 in FP16 or BF16, 1 in INT8, and 0.5 in 4-bit formats, so a 7 billion parameter model needs about 14 GB in FP16. The key-value cache holds attention keys and values for every prior token and equals 2 times layers times hidden dimension times sequence length times batch size times bytes per element, which is why long contexts and large batches, rather than the weights, are usually what exhausts memory during serving. Activations hold intermediate tensors and are far larger in training than inference because they must be retained for the backward pass. Training additionally carries gradients at weight precision plus Adam optimizer moments, conventionally 8 bytes per parameter in FP32, producing the familiar figure of roughly 16 bytes per parameter for mixed-precision training before activations. Allow about 1 GB for the CUDA context and framework overhead, and plan to stay near 85 to 90 percent of nominal VRAM, since allocator fragmentation makes the final slice unusable in practice. The layer and hidden-dimension values used here are interpolated from common model shapes, so mixture-of-experts models, unusual head configurations, and architectures using grouped-query or multi-query attention, which shrink the cache several-fold, will deviate from the estimate. Parameter-efficient methods such as LoRA change the arithmetic entirely by freezing the base weights and training only small adapter matrices.

This memory accounting became necessary when models outgrew single accelerators. Mixed-precision training was formalised by Paulius Micikevicius and colleagues at NVIDIA and Baidu in 2017, pairing FP16 compute with an FP32 master copy of the weights plus loss scaling, which is the origin of the extra optimizer-state bytes that dominate training budgets. Google introduced bfloat16 for its TPUs, and NVIDIA added support in the Ampere generation in 2020, keeping the FP32 exponent range and removing the need for loss scaling altogether. Samyam Rajbhandari and co-authors at Microsoft published ZeRO in 2019 alongside DeepSpeed, itemising the 16 bytes per parameter of mixed-precision Adam training and showing that sharding optimizer states, gradients, and parameters across devices removes the redundancy; the same accounting underpins PyTorch fully sharded data parallel. Attention memory received similar treatment. Noam Shazeer proposed multi-query attention in 2019 and Joshua Ainslie and colleagues at Google generalised it to grouped-query attention in 2023, both cutting key-value cache size by sharing heads. Woosuk Kwon and co-authors at Berkeley then published PagedAttention with vLLM in 2023, borrowing operating-system paging to eliminate cache fragmentation. Community references such as the EleutherAI Transformer Math writeup collected these formulas into the estimation practice used today.