Hypothesis Testing

Hypothesis testing is a framework for using sample data to decide between two competing claims about a population:

• Null hypothesis $H_0$: the default / "nothing interesting" claim (e.g. coin is fair, drug has no effect).
• Alternative $H_a$: what we suspect / want to demonstrate.

Procedure:

1. State $H_0$ and $H_a$.
2. Choose a significance level $\alpha$ (commonly 0.05) — the probability of false rejection (Type I error).
3. Compute a test statistic from the data (z-score, t-statistic, chi-square, F-ratio).
4. Compute the p-value — probability under $H_0$ of seeing data at least as extreme.
5. Decide: if $p < \alpha$, reject $H_0$; otherwise fail to reject.

Two error types:

• Type I: rejecting a true $H_0$ (probability $\alpha$).
• Type II: failing to reject a false $H_0$ (probability $\beta$); $1 - \beta$ is the power.

Common confusion: "fail to reject" ≠ "accept $H_0$". Absence of evidence is not evidence of absence — small sample sizes can hide real effects.

This framework underlies clinical trials, A/B tests, quality control, and most published "statistical significance" claims.