Etl Throughput Sizing Assistant

Sizing a batch pipeline is a queueing problem stated as arithmetic. Required raw throughput is volume divided by the window, so 100 GB in a four-hour window is about 7.1 MB per second. Transformation work is charged as a multiplier over that baseline: 1.2 for type casts and filters, 2.0 for joins and aggregations, 3.5 for heavy enrichment or model scoring, and 6.0 for the extreme case. The multiplier exists because a join materializes intermediate data that appears in neither the input nor the output volume. Dividing across workers assumes near-linear scaling, which holds only while the serial fraction stays small; source connection limits, target commit contention, and coordination form the ceiling that Amdahl's law describes, and most pipelines stop scaling somewhere between 16 and 32 workers. IOPS are derived at a 64 KB block size, so 7.1 MB per second is roughly 112 reads per second, trivial for NVMe but meaningful on network-attached storage. The memory model gives each worker a 64 MB base plus buffered batches scaled by complexity. The number worth watching is peak rather than average, because hash joins and sorts hold state proportional to the larger input; once that exceeds the heap the engine spills to disk and effective throughput can fall by an order of magnitude. Cost figures assume steady on-demand pricing and ignore spot interruption, retries, and data skew, where one partition holds most of the keys and a single worker becomes the entire critical path.

Extract, transform, load emerged as a named discipline alongside the data warehouse. Bill Inmon's Building the Data Warehouse in 1992 and Ralph Kimball's The Data Warehouse Toolkit in 1996 set out the two competing architectures, and the staging and transformation layer between operational systems and the warehouse became a product category in itself. Informatica was founded in 1993, and tools including Ardent DataStage and Ab Initio served the same market through the decade. Sizing at the time meant fitting a nightly window on a single symmetric multiprocessing server, so the binding constraint was usually disk throughput rather than parallelism. Distributed processing changed the arithmetic. The MapReduce paper by Jeffrey Dean and Sanjay Ghemawat at Google in 2004 described batch processing across commodity clusters, and Hadoop, started by Doug Cutting and Mike Cafarella, made an open implementation available from 2006. Apache Spark, begun at the Berkeley AMPLab around 2010, moved the working set into memory and made memory per worker the central sizing question rather than an afterthought. Cloud warehouses that separate storage from compute, Amazon Redshift from 2012 and Snowflake from 2014, then pushed the transform step into the warehouse itself, the pattern usually written ELT, which is why modern capacity questions are often expressed in credits or DPU-hours rather than server counts. Little's Law, published by John Little in 1961, still underlies the throughput, latency, and concurrency relationship in all of these designs.