Experiment Design Assistant: Sample Size & Power

A power calculation ties four quantities together so that fixing three determines the fourth: the significance level alpha, the power (1 minus beta), the effect size, and the sample size. Alpha is the tolerated rate of false positives, conventionally 0.05. Power is the probability of detecting a real effect of the stated size, conventionally 0.80 in most applied fields and 0.90 in confirmatory clinical trials. Effect size here is Cohen's d, the difference between group means divided by the pooled standard deviation, with 0.2, 0.5, and 0.8 as the conventional small, medium, and large anchors. Sample size scales with the inverse square of d, which is why halving the effect you want to detect quadruples the participants required. Minimum detectable effect inverts the same relation and is the more useful framing when the sample is capped by budget or recruitment reach. Two cautions apply to the numbers shown. First, the standard formula for a two-independent-group comparison is n = 2 * ((z_alpha + z_beta) / d)^2 per group; the form implemented here omits that factor of 2, so treat the output as a lower bound and roughly double it for a conventional two-arm design. Second, inflate the result for expected attrition, typically 10 to 20 percent, and adjust alpha if you plan multiple primary comparisons. Power analysis is only meaningful before data collection: post hoc power computed from the observed effect is a deterministic function of the p-value and carries no additional information.

The framework comes from two lines of work in the 1920s and 1930s. Ronald Fisher set out randomization, replication, and blocking in Statistical Methods for Research Workers in 1925 and The Design of Experiments in 1935, arguing that random assignment is what licenses a causal reading of a group difference. Jerzy Neyman and Egon Pearson, in papers published between 1928 and 1933, added the decision-theoretic structure of two competing hypotheses, errors of the first and second kind, and the power of a test, concepts Fisher himself disputed for the rest of his career. The hybrid taught today combines Fisher's p-value with the Neyman and Pearson error rates. Power analysis reached applied researchers largely through Jacob Cohen. His 1962 survey of articles in the Journal of Abnormal and Social Psychology found that typical studies had little chance of detecting the effects they were designed to find, and his 1969 handbook, revised in 1988 as Statistical Power Analysis for the Behavioral Sciences, supplied both the tables and the small, medium, and large conventions still quoted. G*Power, developed by Franz Faul, Edgar Erdfelder and colleagues and widely cited after its 2007 release, made these calculations routine on a desktop. Concern resurfaced in the 2010s: a 2013 analysis led by Katherine Button in Nature Reviews Neuroscience estimated median statistical power in neuroscience at around 21 percent, and preregistering the target sample size became a standard remedy.