Rag Chunk Size Calculator

Chunking is the step that decides what a retrieval system is actually able to retrieve. A document is cut into passages, each passage is encoded into a single fixed-length vector, and search compares a query vector against those. The chunk count follows a sliding-window formula: with an overlap fraction, the window advances by chunk size minus overlap tokens, so 512-token chunks at 10 percent overlap step 461 tokens at a time and a 10,000-token document yields 22 chunks rather than 20. Overlap exists to protect sentences and definitions that straddle a boundary, at the price of storing the duplicated span twice; 10 to 20 percent is the usual band, and beyond about 30 percent the redundancy ratio rises faster than recall does. Size is a trade-off between specificity and coherence. Short chunks under roughly 128 tokens produce embeddings that are sharp but brittle, matching only near-literal queries. Long chunks past about 1,024 tokens average too many topics into one vector and dilute the signal, and they also consume the generation budget: retrieving top-k chunks costs k times chunk size tokens of context, which is why practitioners keep retrieved material to roughly 30 to 60 percent of the window and leave room for the system prompt and the answer. Storage scales with chunk count, at 4 bytes per dimension for float32 vectors, so a 1,536-dimension embedding is 6,144 bytes per chunk. The caveats: embedding models have their own maximum input length and silently truncate past it, and fixed-size splitting ignores structure, so tables and code blocks are best kept intact.

Retrieval-augmented generation was named and formalised in 2020 by Patrick Lewis and colleagues at Facebook AI Research, whose NeurIPS paper coupled a dense retriever to a sequence-to-sequence generator for knowledge-intensive tasks. It built on Dense Passage Retrieval, published the same year by Vladimir Karpukhin and co-authors, which showed that learned dense vectors could beat the long-dominant BM25 keyword ranking on open-domain question answering. Both systems inherited a passage-splitting convention from earlier open-domain QA work on Wikipedia, which cut articles into fixed 100-word blocks - the first widely used chunk size, chosen mainly because the reader models of the day had short inputs. The engineering conventions in use today formed after 2022. Nils Reimers and Iryna Gurevych had released Sentence-BERT in 2019, making it practical to embed passages rather than whole documents, and OpenAI released the text-embedding-ada-002 model with 1,536 dimensions in December 2022, which set the default dimensionality for a generation of systems. LangChain, started by Harrison Chase in October 2022, and LlamaIndex, begun by Jerry Liu the same year, shipped recursive character text splitters with configurable chunk size and overlap; their defaults, commonly 512 or 1,000 units with roughly 10 to 20 percent overlap, became the de facto starting point. Later work moved toward structure-aware and semantic splitting, and toward evaluation harnesses such as RAGAS that measure retrieval recall and precision instead of relying on defaults.