AI Token Calculator

A token is the atomic unit a transformer language model reads, attends over, and is billed for. Modern models do not split text on spaces; they use subword tokenizers such as byte-pair encoding (GPT family, via the tiktoken library) or SentencePiece unigram models (many Gemini and Llama derivatives), which learn a fixed vocabulary of fragments from a training corpus. GPT-2 shipped a byte-level BPE vocabulary of 50,257 entries; GPT-4o class models use around 200,000. English prose lands near 4 characters or 1.3 tokens per word, which is the ratio this page applies, but the figure is a corpus average, not a rule. Source code, JSON, and long URLs run 2 to 3 tokens per word because punctuation and identifiers fragment; Hindi, Thai, and Japanese text can cost 2 to 4 times more tokens per character than English on Latin-optimised vocabularies. Billing multiplies that token count by a published per-million rate, quoted separately for prompt and completion. Input rates in the table above span two orders of magnitude, from about 0.075 US dollars per million for Gemini 1.5 Flash to 3.00 for Claude 3.5 Sonnet, and output is routinely 2 to 5 times the input rate because decoding is autoregressive and cannot be batched across positions the way prefill can. The main caveat is that a word-count heuristic is an estimate: exact counts require running the model tokenizer. Published rates also change, and prompt caching or batch endpoints commonly cut the effective input price by 50 percent.

Byte-pair encoding began as a data-compression scheme described by Philip Gage in the February 1994 issue of The C Users Journal. It reached natural language processing in 2016, when Rico Sennrich, Barry Haddow, and Alexandra Birch of the University of Edinburgh presented Neural Machine Translation of Rare Words with Subword Units at ACL, using BPE merges to give translation systems an open vocabulary without an unknown-word token. Related lines developed in parallel: Mike Schuster and Kaisuke Nakajima published WordPiece for Japanese and Korean voice search in 2012, and Taku Kudo and John Richardson released SentencePiece at Google in 2018, which tokenizes raw text without pre-segmentation and so handles languages that do not use spaces. Token accounting became a commercial concern when OpenAI opened the GPT-3 API in June 2020 and metered usage in units of 1,000 tokens rather than requests or characters. Anthropic, Google, and the open-weight hosting providers adopted the same unit, and by 2023 the industry had shifted to quoting prices per million tokens as volumes grew. Splitting the price into separate prompt and completion rates followed once providers observed how differently the prefill and decode phases load a GPU. OpenAI later published tiktoken as open source so developers could count tokens locally, and Anthropic added a token-counting endpoint, replacing rule-of-thumb estimates with exact figures where precision matters.