Etl Throughput Sizing Assistant Calculator
Calculate etl throughput sizing assistant with our free tool. Get data-driven results, visualizations, and actionable recommendations.
Reviewed by Daniel Agrici, Founder & Lead Developer
Formula
Throughput = (DataVolume * ComplexityFactor) / (Window * Workers); Memory = Workers * (64MB + 2 * BatchSize * Complexity)
Calculates required throughput by dividing data volume by the processing window, then multiplying by a complexity factor that accounts for transformation overhead (1.2x for light, 2x for medium, 3.5x for heavy). Memory is estimated per worker based on base allocation plus batch buffer and transform state. Parallelism divides the workload across workers with near-linear scaling.
Worked Examples
Example 1: Nightly Data Warehouse Load
Problem:Load 100GB of transactional data with medium-complexity transforms in a 4-hour nightly window using 4 parallel workers.
Solution:Total records = (100 * 1024 * 1024) / 2 = 52,428,800 records\nRaw throughput = (100 * 1024) / (4 * 3600) = 7.11 MB/s\nEffective throughput = 7.11 * 2.0 (medium) = 14.22 MB/s\nRecords/sec = 52,428,800 / 14,400 = 3,641 rec/s\nPer worker: 3.56 MB/s, 910 rec/s\nMemory: 4 workers * ~196MB = ~0.8 GB\nCores: 4 + 1 = 5
Result:Feasible with 4 workers, 5 CPU cores, 0.8GB RAM. Estimated cost: ~$1.05 per run on cloud compute.
Example 2: Real-Time Micro-Batch Pipeline
Problem:Process 10GB of event data per hour with heavy transforms (ML scoring) using 8 workers.
Solution:Raw throughput = (10 * 1024) / 3600 = 2.84 MB/s\nEffective with heavy transforms (3.5x) = 9.96 MB/s\nRecords (0.5KB each) = 20,971,520 records/hour = 5,826 rec/s\nPer worker: 1.25 MB/s, 728 rec/s\nMemory: 8 * ~288MB = ~2.3 GB\nCores: 8 + 2 = 10\nIOPS: ~91 read + ~73 write = 164 total
Result:Requires 10 CPU cores, 2.3GB RAM, 164 IOPS. Heavy ML transforms are the bottleneck — consider pre-computing features.
Frequently Asked Questions
How do you calculate ETL throughput requirements?
ETL throughput is calculated by dividing the total data volume by the available processing window. For example, 100GB of data in a 4-hour window requires 100GB / (4 * 3600 seconds) = 7.1 MB/s raw throughput. However, transformations add significant overhead — a medium-complexity transformation (joins, aggregations, type conversions) typically doubles the effective data moved, requiring 14.2 MB/s of processing capacity. Heavy transformations involving machine learning scoring, complex lookups, or data quality checks can multiply this by 3-6x. Always size for peak throughput plus a 20-30% buffer to account for variability in data distribution and system load.
How many parallel workers should an ETL pipeline use?
The optimal number of parallel workers depends on data volume, transformation complexity, and available resources. As a rule of thumb, start with one worker per 25GB of data for medium-complexity transformations. Adding workers provides near-linear scaling up to the point where I/O or network becomes the bottleneck. For most setups, 4-16 workers handle typical enterprise workloads efficiently. Beyond 16 workers, coordination overhead starts to offset parallelism gains. Key constraints include: source database connection limits, target system write capacity, available CPU cores (at least 1 per worker), and memory (each worker needs its own buffer space). Monitor actual utilization to fine-tune.
How much memory does an ETL pipeline need?
ETL memory requirements depend on worker count, batch size, record size, and transformation complexity. Each worker needs a base allocation (typically 64-128MB for framework overhead) plus buffer memory for read/write batches, plus transformation state (hash tables for lookups, aggregation buffers, sort memory). For medium-complexity transforms, plan for 200-500MB per worker. Heavy transforms with large dimension lookups can need 1-2GB per worker. A 4-worker pipeline with medium transforms typically needs 2-4GB total. The biggest memory consumers are hash join operations and in-memory lookup tables — consider switching to disk-based approaches if these exceed available memory.
What IOPS are needed for ETL workloads?
ETL workloads are typically sequential I/O heavy, which is more forgiving than random I/O. Read IOPS can be estimated by dividing throughput by block size: at 7 MB/s with 64KB blocks, you need about 112 read IOPS. Write IOPS are typically 60-80% of read IOPS due to aggregation reducing output volume. For SSDs, these numbers are easily achievable (modern SSDs deliver 50,000+ IOPS). For HDDs, sequential throughput is the limiting factor — a single HDD delivers 100-150 MB/s sequential. Cloud storage (S3, GCS) has different characteristics: high throughput but higher latency per request, making larger batch sizes more important. For data lakes, partition your data to enable parallel file reads.
References
Reviewed by Daniel Agrici, Founder & Lead Developer · Editorial policy