Vector Database Storage Calculator

Capacity planning for a vector store starts with chunk count, not document count. Each document is split into passages using a sliding window, so chunks per document is the document token length divided by the effective step of chunk size minus overlap. Every resulting chunk carries four costs: the embedding itself at 4 bytes per dimension for float32, meaning 6,144 bytes for a 1,536-dimension vector and 3,072 bytes at 768 dimensions; the chunk text, roughly 4 bytes per token for UTF-8 English; a metadata record holding identifiers, source, and filter fields, typically 128 to 512 bytes; and the approximate-nearest-neighbour index. HNSW graphs store outgoing links per node and commonly add 15 to 30 percent on top of raw vector bytes, which is the 20 percent assumption used here. Replicas multiply the entire total. The figure that governs cost in practice is how much of this must sit in RAM. A million 1,536-dimension vectors is about 5.7 GiB of raw embeddings before index and payload, so a graph index kept resident typically calls for a 16 to 32 GiB memory instance to serve sub-millisecond queries. Compression is the main lever: scalar quantization to int8 cuts vector bytes by 75 percent while usually retaining above 95 percent of recall, product quantization goes further, and binary quantization reaches 32 to 1 at a real accuracy cost that generally requires a reranking pass. Caveats worth building in: index construction needs transient headroom above steady-state size, deletions are often tombstoned rather than reclaimed until compaction, and rebuilding a large index after restore can take hours.

Similarity search over high-dimensional vectors long predates language models. Product quantization, the compression scheme underneath many modern indexes, was published by Herve Jegou, Matthijs Douze, and Cordelia Schmid in 2011 in IEEE Transactions on Pattern Analysis and Machine Intelligence. Spotify open-sourced Annoy, written by Erik Bernhardsson, in 2013 for music recommendation. Yury Malkov and Dmitry Yashunin described Hierarchical Navigable Small World graphs in 2016, published in the same IEEE journal in 2018, and HNSW became the default index in most vector databases because it offers high recall at logarithmic search cost. Facebook AI Research released FAISS, by Jeff Johnson, Matthijs Douze, and Herve Jegou, in 2017, putting GPU-accelerated billion-scale search into general use. The database category formed later. Pinecone was founded in 2019 by Edo Liberty and launched a managed vector service in 2021; Zilliz released Milvus as an open-source vector database in 2019; Weaviate and Qdrant followed, and pgvector brought vector columns to PostgreSQL. Microsoft published DiskANN in 2019, showing that a graph index could serve most vectors from SSD while keeping only navigation structures in memory, which reshaped the cost model for very large collections. Demand exploded after the December 2022 release of OpenAI text-embedding-ada-002 and the retrieval-augmented generation boom of 2023, when storage sizing shifted from a research concern to a routine infrastructure budgeting exercise and quantization options became standard configuration knobs.