Kubernetes Resource Rightsizer

Q: How do cloud costs relate to Kubernetes resource settings?

Cloud providers charge for provisioned node capacity, not pod requests. However, requests determine how many pods fit per node. Over-provisioned requests mean fewer pods per node, requiring more nodes. Right-sizing requests directly reduces node count and costs.

Q: What tools help with right-sizing?

VPA (Vertical Pod Autoscaler) provides recommendations. Goldilocks visualizes VPA recommendations. Kubecost and CAST AI provide cost-aware recommendations. Prometheus + Grafana for custom usage dashboards. Cloud provider tools: GKE recommender, AWS Compute Optimizer.

Formula

Recommended Request = Average Usage × Buffer Multiplier; Recommended Limit = Peak Usage × 1.2

Requests should accommodate typical usage plus headroom for variance (1.2-2x depending on workload type). Limits should exceed peak observed usage to prevent throttling (CPU) or OOMKill (memory). The buffer multiplier varies by workload: web services (1.3x), batch jobs (1.5x), ML workloads (2.0x), databases (1.4x). This formula balances efficiency with reliability.

Worked Examples

Example 1: Web Application - Over-Provisioned

Problem:3-replica web app. Current: 1000m CPU request, 2Gi memory request. Actual usage: 200m avg / 600m peak CPU, 400Mi avg / 800Mi peak memory.

Solution:Analysis: - CPU utilization: 200m / 1000m = 20% (wasteful) - Memory utilization: 400Mi / 2048Mi = 20% (wasteful) Recommended (1.3x buffer for web): - CPU request: 200 × 1.3 = 260m - CPU limit: 600 × 1.2 = 720m - Memory request: 400 × 1.3 = 520Mi - Memory limit: 800 × 1.2 = 960Mi Resource reduction: - CPU: 1000m → 260m (74% reduction) - Memory: 2Gi → 520Mi (75% reduction) Cost impact (assuming $30/core/mo, $5/GB/mo): - Current: 3 × ($30 + $10) = $120/mo - Optimized: 3 × ($7.80 + $2.54) = $31/mo - Savings: $89/mo ($1,068/year)

Result:CPU: 260m/720m | Memory: 520Mi/960Mi | Savings: $89/month

Example 2: ML Inference - Bursty Workload

Problem:ML model serving with high variance. 2 replicas. Current: 2000m CPU, 4Gi memory. Usage: 500m avg / 1800m peak CPU, 2Gi avg / 3.5Gi peak memory.

Solution:Analysis (2.0x buffer for ML): - CPU is bursty: 500m base, 1800m peak (3.6x variance) - Memory spikes during batch inference Recommended: - CPU request: 500 × 2.0 = 1000m (handle variance) - CPU limit: 1800 × 1.2 = 2160m (allow bursts) - Memory request: 2048 × 1.3 = 2662Mi - Memory limit: 3584 × 1.2 = 4301Mi For ML workloads, slightly over-provision to avoid latency spikes. Cost comparison: - Current: 2 × ($60 + $20) = $160/mo - Optimized: 2 × ($30 + $13) = $86/mo - Savings: $74/mo while maintaining performance headroom

Result:CPU: 1000m/2160m | Memory: 2662Mi/4301Mi | Savings: $74/month

Example 3: Database - Memory-Critical

Problem:PostgreSQL pod. 1 replica. Current: 500m CPU, 8Gi memory. Usage: 100m avg / 300m peak CPU, 6Gi avg / 7.2Gi peak memory.

Solution:Analysis (1.4x buffer for database): - CPU is over-provisioned (100m vs 500m) - Memory is appropriately sized (high utilization is expected) Recommended: - CPU request: 100 × 1.4 = 140m - CPU limit: 300 × 1.2 = 360m - Memory request: 6144 × 1.1 = 6758Mi (databases benefit from stable memory) - Memory limit: 7372 × 1.2 = 8847Mi CAUTION: For databases, memory OOM is catastrophic. Keep memory limit generous. Memory savings are less important than stability. Cost: - Current: $15 + $40 = $55/mo - Optimized: $4.20 + $33 = $37.20/mo - Savings: $17.80/mo (primarily from CPU)

Result:CPU: 140m/360m | Memory: 6758Mi/8847Mi | Savings: $17.80/month (conservative)

Frequently Asked Questions

What's the difference between requests and limits in Kubernetes?

Requests are guaranteed resources the container needs to run—used by the scheduler to place pods on nodes. Limits are maximum resources a container can use—exceeding CPU causes throttling; exceeding memory causes OOMKill. Set requests based on typical usage; set limits based on peak usage plus buffer.

How do I determine the right CPU request?

Analyze actual CPU usage over 1-2 weeks using metrics (Prometheus, Datadog). Set request to: (average usage × 1.2-1.5 buffer). The buffer accounts for variance and prevents throttling during normal operation. For bursty workloads, use higher buffers (1.5-2x).

How do I determine the right memory request and limit?

Memory is less elastic than CPU—exceeding limit causes OOMKill. Set request to (average usage × 1.3-1.5). Set limit to (peak usage × 1.2) minimum. For memory, it's better to over-provision slightly than risk OOMKill. Monitor OOMKilled events in your cluster.

What happens if requests are set too high?

Over-provisioned requests waste cluster resources. The scheduler reserves requested resources even if unused. You'll need more nodes than necessary, increasing costs. Other pods may be unable to schedule due to 'insufficient resources' even when actual utilization is low.

Should limits equal requests?

For memory: limits should be ≥ requests, often 1.5-2x requests to handle spikes without OOMKill. For CPU: limits can be higher than requests (burstable) or equal (guaranteed QoS). Equal requests/limits provide predictable performance but reduce flexibility.

How do I right-size for variable workloads?

For variable workloads: (1) Use VPA (Vertical Pod Autoscaler) for automatic recommendations, (2) Set requests to handle baseline load, (3) Set limits to handle peak, (4) Combine with HPA for horizontal scaling during sustained high load. Analyze weekly patterns, not just daily.

What is QoS class and how does it affect right-sizing?

Kubernetes assigns QoS classes: Guaranteed (requests=limits), Burstable (requests<limits), BestEffort (no requests/limits). Guaranteed pods are last to be evicted under pressure. For critical workloads, use Guaranteed QoS. For batch jobs, BestEffort may be acceptable.

How do cloud costs relate to Kubernetes resource settings?

Cloud providers charge for provisioned node capacity, not pod requests. However, requests determine how many pods fit per node. Over-provisioned requests mean fewer pods per node, requiring more nodes. Right-sizing requests directly reduces node count and costs.

What tools help with right-sizing?

VPA (Vertical Pod Autoscaler) provides recommendations. Goldilocks visualizes VPA recommendations. Kubecost and CAST AI provide cost-aware recommendations. Prometheus + Grafana for custom usage dashboards. Cloud provider tools: GKE recommender, AWS Compute Optimizer.

Background & Theory

Kubernetes resource right-sizing optimizes CPU and memory requests/limits to balance cost efficiency, performance, and reliability for containerized workloads.

Concept Overview

In Kubernetes, every container specifies resource requests (guaranteed allocation) and limits (maximum allowed). Requests affect scheduling—the scheduler only places pods on nodes with sufficient unrequested resources. Limits affect runtime—exceeding CPU limits causes throttling; exceeding memory limits causes OOMKill.

The right-sizing challenge is finding values that are high enough to prevent performance degradation but low enough to maximize cluster utilization. Too high wastes money; too low degrades reliability. Most organizations find their initial guesses are 2-5x higher than needed.

Right-sizing isn't one-time: workloads evolve, traffic patterns change, and code optimizations affect resource needs. Continuous monitoring and periodic adjustment are essential for maintaining efficiency.

Key Variables & Intuition

CPU Request — Guaranteed CPU; set to average usage × buffer (1.2-1.5x)
CPU Limit — Maximum CPU before throttling; set to peak usage × 1.2
Memory Request — Guaranteed memory; set to average × buffer
Memory Limit — Maximum memory before OOMKill; set to peak × 1.2 minimum
Buffer Multiplier — Accounts for variance; higher for bursty workloads
Workload Type — Web (consistent), batch (variable), ML (bursty), database (memory-critical)

Assumptions

Historical usage patterns predict future needs
Monitoring data accurately reflects actual usage
Workload characteristics remain relatively stable
Node capacity is the primary cost driver

Limitations & Edge Cases

Cold start spikes — Applications may spike resources during startup; don't over-optimize for steady-state
Garbage collection — JVM and similar runtimes have periodic memory spikes
Batch processing — Job-based workloads have highly variable usage
Caching — Applications may benefit from excess memory even if not "required"
Multi-tenant clusters — Noisy neighbor effects complicate right-sizing

Scenario: A Java application shows 500Mi average memory but occasional 2Gi spikes during GC. Setting limit to 1Gi causes OOMKill during GC. Solution: set limit to 2.5Gi to accommodate GC, even though average is much lower. Memory right-sizing must account for runtime behavior, not just application requirements.

Interpretation Guide

Utilization Efficiency:

70-85%: Well-optimized; good balance of efficiency and headroom
50-70%: Moderately over-provisioned; optimization opportunity
Below 50%: Significantly over-provisioned; prioritize right-sizing
Above 85%: Risk of throttling/OOMKill; increase requests

Request/Limit Ratio:

1:1 (Guaranteed QoS): Predictable but inflexible
1:1.5-2 (Burstable): Good for most workloads
1:3+ (Very Burstable): Risk of noisy neighbor issues

Practical Tips

Monitor for 1-2 weeks before right-sizing; include weekend/month-end patterns
Use VPA in recommendation mode before applying changes
Right-size development environments aggressively; production more conservatively
Set memory limits 20%+ above peak to prevent OOMKill
For critical workloads, use Guaranteed QoS (requests=limits)
Review right-sizing quarterly; workloads drift over time
Combine with HPA for horizontal scaling during sustained load

Common Mistakes

Using default values without measuring actual usage
Right-sizing based on short observation periods (hours, not weeks)
Setting memory limit equal to request (OOMKill risk)
Ignoring startup resource spikes
Over-optimizing non-production environments (minimal savings)
Not accounting for workload growth after optimization

When NOT to Aggressively Right-Size

Production-critical workloads where downtime cost exceeds infrastructure savings
Workloads with high variance that metrics may not capture
Immediately before major traffic events
When observability is insufficient to validate changes

History

Kubernetes resource management evolved from static provisioning to dynamic right-sizing as containerized workloads grew in scale and complexity.

Origins & Why It Emerged

Early container orchestration (pre-2015) often used simple resource limits or none at all. Containers competed for host resources, causing unpredictable performance. Google's Borg system, Kubernetes' predecessor, pioneered resource quotas and bin-packing algorithms to efficiently place workloads.

Kubernetes (2014) formalized requests and limits: requests for scheduling guarantees, limits for enforcement. This dual model balanced predictability (guaranteed resources) with efficiency (allowing bursting). However, setting appropriate values remained manual and error-prone.

How It Evolved in Practice

Early Kubernetes users often copied example configurations with arbitrary resource values. Operations teams discovered issues empirically: OOMKilled pods, throttled applications, or clusters at "capacity" with 20% actual utilization. Right-sizing became a recognized discipline.

Monitoring tools (Prometheus, 2012-2016) enabled observability. Teams could finally see actual CPU and memory usage versus configured requests/limits. The gap was often 3-10x, revealing massive waste.

The Vertical Pod Autoscaler (VPA, 2018) automated recommendations based on historical usage. Tools like Goldilocks (Fairwinds) and commercial solutions (Kubecost, CAST AI) emerged to visualize and act on right-sizing opportunities.

Modern Usage Today

Today, right-sizing is a core FinOps practice for Kubernetes. Mature organizations: run VPA in recommendation mode on all workloads, review recommendations in CI/CD pipelines, set budget alerts based on resource requests, and combine VPA with HPA for complete autoscaling.

Cloud providers offer integrated recommendations: GKE's pod recommender, AWS Compute Optimizer, Azure Advisor. The frontier is predictive scaling based on traffic patterns and ML-driven optimization.

Common Misconceptions Historically

"Higher limits are always better" — Oversized limits don't improve performance; they waste resources and prevent efficient bin-packing
"Requests don't matter for performance" — Requests determine scheduling; under-provisioned requests cause noisy neighbor issues
"Set once and forget" — Workloads change; quarterly right-sizing reviews should be standard practice
"VPA can replace HPA" — They serve different purposes: VPA for sizing, HPA for scaling; use both