Test Flakiness Retry Budget
Optimize your test retry strategy by calculating the impact of flaky tests on CI costs, build confidence, and developer productivity.
Formula
Suite Confidence = (1 - Flake Rate^(Retries+1))^Flaky Tests × 100%; Retry Overhead = (Expected Retries × Avg Duration) / Base Duration × 100%
Suite confidence calculates the probability that no flaky test causes a false failure. For each flaky test, the probability of passing with N retries is (1 - FlakeRate^(N+1)), since the test must fail N+1 consecutive times to ultimately fail. This probability is raised to the power of the number of flaky tests, since all must pass for the suite to pass. Retry overhead measures the additional time spent on retries as a percentage of base test time. This formula works because flaky failures are typically independent events, making the compound probability calculation valid. The exponential relationship explains why 2-3 retries capture most of the confidence benefit.
Worked Examples
Example 1: Small Test Suite Retry Planning
Problem:A team has 100 tests, 5% are flaky with 10% failure rate. They run CI 30 times/day at $0.01/min with 20-second average test time.
Solution:Configuration: Total tests: 100 Flaky tests: 5 (5%) Flake rate: 10% Max retries: 2 Probability analysis: Pass on first try: 90% Pass with 1 retry: 99% Pass with 2 retries: 99.9% Expected retries per flaky test: 0.10 + 0.01 = 0.11 retries Total expected retries: 5 × 0.11 = 0.55 Time impact: Base run: 100 × 20s = 2000s = 33.3 min Retry time: 0.55 × 20s = 11s = 0.18 min Overhead: 0.5% Cost: Base: 33.3 × $0.01 × 30 = $10/day Retries: ~$0.05/day Monthly: ~$222
Result:2 retries optimal | 0.5% overhead | $222/month | 99.9% confidence
Example 2: High-Flakiness CI Pipeline
Problem:An e-commerce platform has 2000 tests, 12% flaky with 25% avg failure rate. 100 CI runs/day, $0.005/min, 45s avg test time.
Solution:Configuration: Total tests: 2000 Flaky tests: 240 (12%) Flake rate: 25% Max retries: 3 Probability: Pass on first: 75% Pass with 3 retries: 1 - 0.25⁴ = 99.6% Expected retries per flaky test: 0.25 + 0.0625 + 0.0156 = 0.33 Total: 240 × 0.33 = 79 retries Time impact: Base: 2000 × 45s = 90,000s = 1500 min Retries: 79 × 45s = 3555s = 59 min Overhead: 4% Cost: Base: 1500 × $0.005 × 100 = $750/day Retries: 59 × $0.005 × 100 = $30/day Monthly: $17,160 ⚠️ 12% flakiness is high! Recommendation: Fix top 50 flakiest tests
Result:⚠️ High flakiness | 4% overhead | $17K/month | Prioritize fixing tests
Example 3: Optimizing Retry Strategy
Problem:Compare retry strategies: 0, 1, 2, 3 retries for 500 tests, 8% flaky, 15% fail rate.
Solution:Scenario analysis at 50 runs/day: 0 Retries: - Suite pass rate: 52% (many false failures) - Cost: $550/month - Developer time wasted: High 1 Retry: - Suite pass rate: 93% - Overhead: 1.2% - Cost: $556/month - Improvement: 41% fewer false failures 2 Retries: - Suite pass rate: 98.9% - Overhead: 1.4% - Cost: $558/month - Improvement: 47% from baseline 3 Retries: - Suite pass rate: 99.8% - Overhead: 1.5% - Cost: $559/month - Marginal improvement: 0.9% Optimal: 2 retries Reason: 98.9% confidence at minimal extra cost 3rd retry adds little value for cost
Result:2 retries optimal | 98.9% confidence | $558/month | Diminishing returns at 3+
Frequently Asked Questions
What is test flakiness?
Test flakiness refers to tests that sometimes pass and sometimes fail without any code changes. Causes include timing issues, race conditions, test order dependencies, shared state, network variability, and environmental differences between runs.
How do retries help with flaky tests?
Retries give flaky tests additional chances to pass, reducing false failures. If a test has a 10% flake rate, one retry reduces the false failure rate to 1%, and two retries reduce it to 0.1%. However, retries add time and cost.
What's a good retry budget?
A good starting point is 2-3 retries for tests known to be flaky. The optimal budget balances confidence (reducing false failures) against cost (CI time and compute). Most teams find 2 retries sufficient for <15% flake rates.
Should I retry all tests or only flaky ones?
Best practice is to only retry known flaky tests. Retrying all tests wastes resources on stable tests and can mask real failures. Use historical data to identify and tag flaky tests for selective retries.
How do I measure test flakiness?
Track pass/fail results across multiple runs of the same code. A test that fails even once on unchanged code is flaky. Calculate flake rate as (inconsistent runs / total runs). Many CI systems provide built-in flakiness detection.
What causes test flakiness?
Common causes include: timing/async issues (race conditions, timeouts), test pollution (shared state, order dependency), external dependencies (network, databases, APIs), environment differences (timezone, locale), and resource constraints (memory, CPU).
Is it better to fix flaky tests or add retries?
Fixing root causes is always preferred long-term. Retries are a tactical solution that adds cost and hides underlying issues. Use retries as a temporary measure while prioritizing permanent fixes for high-impact flaky tests.
How much do retries cost?
Retry costs include CI compute time, delayed feedback, and engineering time investigating false failures. A test suite with 10% flaky tests and 2 retries might add 15-20% to CI costs. High-frequency CI pipelines amplify these costs.
What's quarantining and when should I use it?
Quarantining removes persistently flaky tests from blocking builds while tracking them separately. Use it for tests with >50% flake rates that can't be quickly fixed. Quarantined tests should have owners and deadlines for resolution.
How do retries affect build confidence?
Retries increase confidence that build failures represent real issues rather than flakiness. With properly configured retries, you can achieve 99%+ confidence that a failing build indicates actual problems, reducing wasted investigation time.
Background & Theory
Managing test flakiness effectively requires understanding both the mathematical impact of retries and the engineering economics of different mitigation strategies. This calculator helps teams make data-driven decisions about retry budgets.
Concept Overview
Test flakiness occurs when tests produce inconsistent results on unchanged code. A test might pass 90% of the time but fail 10% due to timing issues, race conditions, or environmental factors. In a suite of hundreds of tests, even small individual flake rates compound into frequent false failures.
The retry strategy addresses flakiness by giving tests additional chances to pass. If a test has a 10% failure rate (90% pass rate), one retry increases the effective pass rate to 99% (must fail twice: 10% × 10% = 1%). Two retries achieve 99.9%. However, retries consume CI resources and can delay feedback.
The optimal retry budget balances several factors: the cost of false failures (developer time investigating, blocked deployments), the cost of retries (CI compute, slower feedback), and the feasibility of fixing underlying flakiness.
Key Variables & Intuition
- Total tests - More tests mean more opportunities for flaky failures
- Flaky test percentage - The proportion of tests that exhibit non-determinism
- Average flake rate - How often flaky tests fail (typically 5-30%)
- Max retries - The ceiling on retry attempts per test
- Test duration - Longer tests make retries more expensive
- CI cost per minute - Varies by provider ($0.004-0.02/min typical)
- Runs per day - High-frequency CI amplifies retry costs
Assumptions
- Flaky failures are independent (no cascading failures)
- Retry success probability equals initial success probability
- CI costs scale linearly with time
- All flaky tests have similar failure rates (simplification)
- No parallelization changes between retries
Limitations & Edge Cases
- Correlated failures - Some flakiness affects multiple tests simultaneously (e.g., database connection issues)
- Time-of-day patterns - Some flakiness correlates with system load or time
- Resource exhaustion - Retries may worsen underlying resource issues
- False confidence - Excessive retries can mask degrading test quality
- Bimodal distributions - Some tests are either very flaky or not at all
Interpretation Guide
Suite confidence above 95% is generally acceptable for development branches; 99%+ is preferred for production deployments. Retry overhead below 10% is typically acceptable; above 25% indicates systemic flakiness problems requiring root cause fixes rather than more retries.
Practical Tips
- Tag flaky tests - Only retry known flaky tests, not the entire suite
- Track flakiness metrics - Monitor trends over time to catch regressions
- Set retry limits - Cap at 2-3 retries; more rarely helps
- Quarantine severe cases - Tests failing >50% should be quarantined, not retried
- Fix high-impact flakes - Prioritize fixes for frequently-run or slow flaky tests
- Use test analytics - Most CI systems offer flakiness detection features
- Review retry costs monthly - Ensure retry spend remains reasonable
Common Mistakes
- Retrying all tests - Wastes resources on stable tests
- Too many retries - Masks problems and delays feedback
- Ignoring flakiness trends - Small increases compound over time
- Not tracking retry costs - Hidden costs accumulate
- Retrying instead of fixing - Retries should be temporary, not permanent solutions
- Inconsistent retry policies - Different rules for different environments cause confusion
When NOT to Use
- For tests with >50% failure rate (quarantine instead)
- When flakiness indicates real system instability
- For tests that affect shared state (retries may corrupt state)
- When fast feedback is critical and retries add unacceptable delay
History
Test flakiness emerged as a recognized engineering problem in the early 2000s as automated testing became standard practice. What was initially dismissed as random noise evolved into a major productivity concern for software teams worldwide.
Origins & Why It Emerged
The concept of test flakiness crystallized as organizations scaled their test automation. Google engineers first documented the phenomenon extensively around 2010-2012, recognizing that what seemed like occasional test quirks was actually a systemic problem consuming significant engineering resources. Early automated tests were often written without considering non-determinism, leading to timing dependencies, shared state issues, and environment assumptions that caused intermittent failures.
How It Evolved in Practice
Initially, teams responded to flaky tests by simply re-running failed builds manually. As CI/CD became more prevalent in the 2010s, this manual approach became unsustainable. Automated retry mechanisms emerged in CI systems like Jenkins, Travis CI, and CircleCI. Google developed sophisticated internal tooling to track, quarantine, and prioritize flaky tests, eventually sharing their approaches through engineering blog posts and conference talks.
The 2015-2020 period saw the development of dedicated flaky test detection features in major CI platforms. Statistical approaches emerged to identify tests whose pass/fail patterns deviated from expected deterministic behavior. Teams began treating flakiness as a measurable metric rather than an unavoidable nuisance.
Modern Usage Today
Modern approaches combine prevention, detection, and mitigation. Linters and code review practices catch common flakiness patterns. CI systems automatically detect and flag flaky tests. Retry budgets provide controlled tolerance for remaining flakiness. Quarantine systems isolate persistently problematic tests.
Leading organizations like Google, Meta, and Spotify report investing significant engineering resources in flakiness reduction, with dedicated teams and OKRs around test reliability. The industry consensus has shifted from accepting flakiness to actively managing and reducing it.
Common Misconceptions Historically
- Flaky tests are just bad tests - Many flaky tests catch real issues; the non-determinism often reflects actual system behavior
- More retries always help - Diminishing returns kick in quickly; 2-3 retries capture most value
- Flakiness is inevitable at scale - While harder to eliminate at scale, well-architected systems can achieve <1% flakiness
- Retries hide real bugs - Properly configured retries distinguish flaky failures from consistent failures
- Only UI tests are flaky - Integration tests, async code, and database tests commonly exhibit flakiness