GPU Memory Calculator

GPU memory for a transformer breaks into four accountable pieces. Weights dominate: parameter count multiplied by bytes per parameter, which is 4 for FP32, 2 for FP16 or BF16, 1 for INT8, and 0.5 for 4-bit formats produced by GPTQ or AWQ. A 7-billion-parameter model therefore needs about 13 GiB in FP16 and roughly 3.3 GiB at 4 bits. The key-value cache is second: attention stores one key and one value vector per token per layer, so it grows linearly with batch size and sequence length and is what makes long-context serving expensive. Activations for a single forward pass are comparatively small, and framework overhead - the CUDA context, allocator fragmentation, and cuBLAS workspaces - adds roughly 10 percent on top. Training is a different regime. Alongside the weights you must hold gradients of equal size and Adam or AdamW optimizer state, which keeps a first and second moment per parameter, conventionally 8 bytes each in FP32. That is about 18 bytes per parameter before activations, so a 7B model that serves in 14 GB can demand well over 100 GB to train naively. The estimates here use representative layer counts and hidden dimensions rather than a specific checkpoint config, so treat them as sizing guidance and confirm against the actual model card. Grouped-query attention shrinks the cache far below what multi-head arithmetic predicts, and gradient checkpointing, FlashAttention, LoRA, and ZeRO sharding each move the training figure substantially.

Sizing models against video memory became a discipline once general-purpose GPU computing matured. NVIDIA released CUDA in 2007, and the 2012 AlexNet result by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton was itself shaped by memory limits: the network was split across two GTX 580 cards with 3 GB each. The transformer architecture introduced by Ashish Vaswani and colleagues in 2017 made the constraint sharper, because attention memory scales with sequence length as well as model size. The reduction techniques arrived in a recognisable sequence. Paulius Micikevicius and co-authors from NVIDIA and Baidu published mixed-precision training in 2017, halving weight and activation storage by keeping an FP32 master copy alongside FP16 compute. Google Brain contributed the bfloat16 format through its TPU work, trading mantissa bits for FP32 exponent range and avoiding most loss-scaling headaches. Samyam Rajbhandari and colleagues at Microsoft introduced ZeRO in 2019 as part of DeepSpeed, partitioning optimizer state, gradients, and parameters across data-parallel ranks. Tri Dao and co-authors released FlashAttention in 2022, cutting attention activation memory by never materialising the full score matrix. Post-training quantization followed with GPTQ from Elias Frantar and colleagues in 2022 and AWQ from Ji Lin and co-authors in 2023, while Georgi Gerganov llama.cpp project, started in March 2023, brought 4-bit inference to ordinary consumer hardware.