The logic of power analysis
Four quantities are locked together: sample size, effect size, significance level (α, usually .05), and statistical power (usually .80). Fix any three and the fourth follows. Sample-size planning means choosing the effect size you care about, the error rates you accept, and solving for n.
The hard input is the effect size. Three defensible sources: previous studies in your area (halve their published effect — publication bias inflates them), a pilot, or the smallest effect that would matter in practice (SESOI). Cohen’s benchmarks (d = 0.2 small, 0.5 medium, 0.8 large) are a last resort, not a plan.
Surveys: margin of error, not power
Descriptive surveys use different arithmetic: you pick a margin of error (±5% is conventional) and confidence level (95%), and n follows from the population size. The counterintuitive part: population size barely matters once it is large — a city of 100,000 and a country of 10 million both need ~385 respondents for ±5%.
Then correct for reality: divide by the expected response rate (often 10–30% for cold surveys) to get the number you must invite, and account for the incomplete responses you will exclude.
The three classic mistakes
Committees and reviewers see the same failures on repeat:
- Post-hoc power — computing power after the study from the observed effect. It is a mathematical restatement of the p-value and adds no information; justify n before data collection.
- Ignoring attrition — longitudinal designs lose 10–30% per wave. Recruit for the final wave, not the first.
- Powering for the main effect but claiming interactions — detecting an interaction typically needs 4× the sample of the corresponding main effect.
Where Kahubi fits
Kahubi’s experiment flow runs power analysis as a step — pick the design, set the smallest effect of interest, get n per arm with the reasoning written into your preregistration draft. The survey flow does the margin-of-error version and tracks live response counts against the target.
Last updated 2026-07-09.