P-value

A p-value is the probability — under the assumption that the null hypothesis $H_0$ is true — of observing data at least as extreme as the actual sample. Small p-values mean the observed data would be unlikely if $H_0$ were true, providing evidence against $H_0$.

Convention: reject $H_0$ if $p < \alpha$ (commonly $\alpha = 0.05$). The threshold $\alpha$ is the Type I error rate you accept.

Common misconceptions (drilled by every stats professor):

• $p$ is not "the probability that $H_0$ is true."
• $p$ is not "the probability the result is due to chance."
• A small $p$ doesn't mean a large effect — only an unlikely-under-$H_0$ effect. With huge samples, trivially small effects become "statistically significant."
• $p > 0.05$ is not proof that $H_0$ is true — only insufficient evidence to reject it.

The American Statistical Association (2016) explicitly warned against treating p-values as binary "significant / not" decisions; report effect sizes and confidence intervals alongside.