Training Time Estimator

Wall-clock training time for a dense transformer is estimated from a compute budget rather than from a simulation. The standard approximation is that a full training step costs about 6 floating-point operations per parameter per token: roughly 2 for the forward pass and 4 for the backward pass, which computes gradients with respect to both activations and weights. Total work is therefore 6 times parameters times tokens, and dividing by the aggregate throughput of the cluster gives seconds. A 7-billion-parameter model on 1 trillion tokens is about 4.2 times 10 to the 22nd FLOPs; on 64 H100 accelerators at 989 TFLOPS of FP16 tensor throughput each, that is roughly three weeks of continuous running. The correction factor that matters is Model FLOPs Utilization, the fraction of peak arithmetic throughput a real training loop achieves. Published values for well-tuned large runs generally sit between 30 and 55 percent; this page assumes 40 percent. The shortfall comes from memory-bandwidth limits, data loading, gradient all-reduce, kernel launch overhead, and attention operations that are not matrix-multiply bound. Scaling adds a second discount: data parallelism holds 85 to 95 percent efficiency to around 64 devices, then degrades as tensor and pipeline parallelism introduce more communication, dropping toward 65 to 75 percent at thousand-GPU scale. The estimate also ignores restarts from hardware failure, checkpoint writes, and evaluation passes, which is why real runs commonly take 1.5 to 3 times the theoretical figure. The 6ND rule assumes dense architectures and understates attention cost at very long sequence lengths.

The compute-first way of reasoning about training runs comes from the scaling-law literature. Jared Kaplan and colleagues at OpenAI published Scaling Laws for Neural Language Models in January 2020, establishing that loss falls as a smooth power law in parameters, data, and compute, and popularising the approximation that training compute is about 6ND for N parameters and D tokens. In March 2022 Jordan Hoffmann and co-authors at DeepMind published the Chinchilla paper, showing that models of the era were substantially undertrained and that for a fixed compute budget parameters and training tokens should be scaled in roughly equal proportion - a result that shifted the field toward smaller models trained on far more data and made token-count estimates central to planning. Model FLOPs Utilization as a reported metric was introduced by Aakanksha Chowdhery and colleagues in the 2022 PaLM paper, giving a hardware-independent way to compare training efficiency across systems. The software that made high utilization attainable arrived alongside it: Megatron-LM from Mohammad Shoeybi and co-authors at NVIDIA in 2019 brought tensor parallelism, Microsoft DeepSpeed and the ZeRO optimizer from Samyam Rajbhandari and colleagues in 2019 and 2020 partitioned optimizer state across ranks, and FlashAttention from Tri Dao and co-authors in 2022 removed a major bandwidth bottleneck. Hardware moved in parallel, from the NVIDIA V100 in 2017 through the A100 in 2020 to the H100 in 2022 and H200 in 2023, alongside Google TPU generations, each roughly multiplying achievable throughput per accelerator.