Log Storage Calculator — Plan Retention & Volume
Estimate how much storage your logs need based on daily volume, retention period, and compression ratio.
Reviewed by Daniel Agrici, Founder & Lead Developer
Formula
Total Storage = (Lines/sec x LineSize x 86400 x Retention / Compression) x (1 + IndexOverhead) x Replicas
Daily raw volume equals total lines per second multiplied by average line size multiplied by seconds per day (86,400). The retention period determines total raw volume. Compression reduces this by the compression ratio. Index overhead adds a percentage for search indexing structures. Replication multiplies the final total by the replica count for durability and query performance.
Worked Examples
Example 1: Mid-Size SaaS Platform
Problem:A SaaS platform with 10 servers each generating 500 log lines per second, 250 bytes average line size, 90-day retention, 5x compression ratio, 15% index overhead, and 2 replicas.
Solution:Total lines/sec = 500 x 10 = 5,000\nRaw bytes/sec = 5,000 x 250 = 1.25 MB/s\nRaw per day = 1.25 x 86,400 = 105.5 GB/day\nCompressed per day = 105.5 / 5 = 21.1 GB/day\nTotal compressed (90 days) = 21.1 x 90 = 1,899 GB\nIndex overhead = 1,899 x 0.15 = 285 GB\nBefore replicas = 2,184 GB\nWith 2 replicas = 4,368 GB = 4.27 TB
Result:5,000 lines/sec | 105.5 GB/day raw | 4.27 TB total with replicas | ~$436/month storage
Example 2: High-Volume Microservices Architecture
Problem:50 microservice instances generating 2,000 log lines per second each, 400 bytes average, 30-day retention, 6x compression, 20% index overhead, 3 replicas.
Solution:Total lines/sec = 2,000 x 50 = 100,000\nRaw bytes/sec = 100,000 x 400 = 40 MB/s\nRaw per day = 40 x 86,400 = 3,375 GB/day = 3.3 TB/day\nCompressed per day = 3,375 / 6 = 562.5 GB/day\nTotal compressed (30 days) = 562.5 x 30 = 16,875 GB\nIndex overhead = 16,875 x 0.20 = 3,375 GB\nBefore replicas = 20,250 GB\nWith 3 replicas = 60,750 GB = 59.3 TB
Result:100K lines/sec | 3.3 TB/day raw | 59.3 TB total storage | Enterprise infrastructure required
Frequently Asked Questions
How do I estimate log volume for my application?
Log volume estimation requires understanding your application logging patterns across different components and severity levels. A typical web server generates 100-1000 log lines per second under moderate traffic, with each line averaging 200-500 bytes for structured JSON logs or 100-300 bytes for plain text logs. Application logs vary widely based on logging verbosity configuration. Debug-level logging can generate 10-50x more volume than info-level logging. To estimate accurately, enable logging at your planned level for a representative period and measure the actual output. Common sources include HTTP access logs (one line per request), application logs (variable), database query logs (one per query if enabled), and system metrics logs. Remember that log volume scales with traffic, so plan for peak traffic periods, not just average load.
What compression ratios can I expect for different log formats?
Log compression ratios vary significantly based on log format and content redundancy. Plain text logs with repetitive patterns (like Apache access logs) typically achieve 5-10x compression with gzip. Structured JSON logs compress slightly less at 4-8x because of the repeated field name overhead, though this overhead itself compresses well. Binary formats like protobuf logs are already compact and may only achieve 2-3x additional compression. Log-specific compression algorithms and columnar storage formats used by systems like ClickHouse can achieve 10-20x compression for highly structured data. The compression level setting also matters. Gzip level 6 (default) provides a good balance, while level 9 achieves marginally better ratios at significantly higher CPU cost. Zstandard (zstd) generally outperforms gzip with better ratios and faster compression speeds, making it the preferred choice for modern log aggregation systems.
How should I set log retention periods for different compliance requirements?
Log retention requirements vary by regulatory framework and business needs. PCI DSS requires audit trail retention for at least one year with the most recent three months immediately accessible for analysis. HIPAA requires audit logs to be retained for six years. SOC 2 typically requires 90-day retention as a minimum. GDPR does not specify exact retention periods but requires logs containing personal data to follow the data minimization principle. Beyond compliance, operational best practices suggest keeping high-resolution logs for 30-90 days for active debugging and incident investigation, then archiving compressed summaries or sampled logs for longer periods. A tiered retention strategy is cost-effective: hot storage (fast SSD) for 7-14 days, warm storage (standard disk) for 30-90 days, and cold archive (object storage like S3 Glacier) for long-term compliance retention at dramatically reduced cost.
What are the costs of popular log management platforms?
Log management costs vary dramatically across platforms and scale. Datadog charges $0.10 per GB ingested per month with 15-day retention included and additional charges for longer retention. Splunk Enterprise Cloud costs $150-200 per GB ingested per day for their standard tier. Elastic Cloud pricing starts around $95 per month for basic clusters with storage-based pricing. New Relic offers free tier up to 100 GB per month then charges $0.30 per GB. Self-hosted ELK (Elasticsearch, Logstash, Kibana) eliminates licensing costs but requires significant infrastructure investment, typically $0.05-0.15 per GB stored in cloud infrastructure. ClickHouse-based solutions like Signoz offer open-source alternatives at lower operational costs. At scale (over 1 TB per day), self-hosted or open-source solutions often cost 3-10x less than managed SaaS platforms, but require dedicated engineering resources for maintenance and reliability.
References
Reviewed by Daniel Agrici, Founder & Lead Developer · Editorial policy