Token Counter

A token is the atomic unit a language model reads and writes, and it is neither a character nor a word. Tokenizers learn a fixed vocabulary of subword strings from a training corpus, so frequent words map to a single token while rare ones fragment into pieces. For ordinary English prose the ratio settles near 4 characters per token, or about 0.75 words, which is why a 1,500-word article lands close to 2,000 tokens. That ratio is a property of the vocabulary, not a universal constant: the cl100k_base vocabulary used by GPT-4 holds roughly 100,000 entries tuned for English and code, while Llama models use a SentencePiece vocabulary of about 32,000 entries, so identical text yields different counts on different models. Token counts govern both cost and feasibility. Input and output are billed separately, with output typically two to four times dearer because each output token requires its own forward pass through the network, whereas the prompt is processed in parallel. Counts also meter the context window - 8,192 tokens on Llama 3 8B, about 128,000 on GPT-4 class models, roughly 200,000 on Claude - and exceeding it produces a hard request error rather than silent truncation. A character-ratio estimate of the kind used here typically lands within 10 to 20 percent for plain English, but degrades badly on source code, mathematical notation, and non-Latin scripts, where Chinese, Japanese, Korean, and emoji characters often consume several tokens each. Production billing should use the real tokenizer.

Subword tokenization began outside machine learning entirely. Byte pair encoding was described by Philip Gage in 1994 as a simple data compression scheme that repeatedly replaces the most frequent adjacent byte pair with an unused byte. It was repurposed for language in 2016, when Rico Sennrich, Barry Haddow, and Alexandra Birch showed at ACL that applying BPE merges to text let neural machine translation systems handle rare and unseen words with a small fixed vocabulary, removing the out-of-vocabulary problem that had constrained earlier word-level models. A parallel line, WordPiece, was developed by Mike Schuster and Kaisuke Nakajima in 2012 for Japanese and Korean speech search and later carried into BERT. The practical tooling followed the models. Google released SentencePiece, an implementation by Taku Kudo and John Richardson published in 2018, which tokenizes raw text without pre-tokenizing on whitespace and therefore works for languages that do not use spaces; it remains the basis for the Llama and Mistral families. OpenAI adopted byte-level BPE with GPT-2 in 2019, so that any byte sequence is encodable, and later published the tiktoken library exposing the exact vocabularies used for billing. Because each laboratory trained its own vocabulary on its own corpus, no cross-model token count is authoritative, and estimating tools like this one exist precisely because the counts diverge.