Batch Inference Cost Calculator

Batch inference trades latency for throughput. Serving an interactive request means keeping a model resident and answering within a few hundred milliseconds, which leaves accelerators idle between arrivals and forces the provider to hold spare capacity for traffic peaks. A batch job carries a deadline measured in hours instead, so the provider can queue the work, group requests into large batches that keep the GPU busy through the memory-bandwidth-bound decode phase, and schedule it into troughs in demand. That efficiency is passed back as a list discount, currently 50 percent at the major API vendors. The arithmetic is direct: total tokens equal requests times average tokens per request, all-real-time spend prices those tokens at the standard per-1000-token rate, and the mixed strategy applies the discounted rate only to the batchable share. The binding constraint is how much traffic tolerates delay. Chat replies, autocomplete, and interactive search do not; classification backlogs, embedding generation, document summarisation, catalogue enrichment, and scheduled reports do. Most teams find 40 to 70 percent is shiftable. Note that blended savings equal the discount multiplied by the batchable share, so a 50 percent discount applied to 70 percent of traffic yields 35 percent overall and never more. The model also ignores retries on failed requests, storage of input and output files during processing, and the operational reality that a job may return near the end of its window rather than promptly, which matters when downstream steps have their own deadlines.

Batch processing is the older of the two serving modes by decades. Commercial computing began with overnight runs on mainframes, where operators queued card decks and jobs were sequenced by systems such as IBM's job control language from the mid 1960s, precisely because the machine was too scarce to sit waiting on humans. Interactive time-sharing was the later innovation, but the underlying economics never changed: expensive hardware is cheapest when it is never idle. Machine learning inference recapitulated the pattern. Cloud platforms offered offline scoring first, with Amazon SageMaker Batch Transform and Google Vertex AI batch prediction letting teams score large datasets against a deployed model without standing up a live endpoint. Large language model APIs initially shipped only synchronous endpoints, so the same bulk workloads were run as scripted loops at real-time prices. That changed in April 2024, when OpenAI introduced its Batch API with a 24-hour completion window at half the standard token price. Anthropic followed with the Message Batches API later in 2024 at the same 50 percent discount, and Google offered batch prediction for Gemini on comparable terms. The discount has held near 50 percent since, which suggests it reflects a real scheduling gain rather than an introductory promotion.