AGES 9-14 - GUIDED STATISTICS
The P-value Detector
Start with two samples, compute one p-value, then see how false alarms, missed signals, and power appear over many repeats.
P-value, Type I, Type II
one-sided two-sample t test
Null H0
Group B and group A have the same true average.
Alternative H1
Group B has a higher true average than group A.
Decision rule
Reject H0 when p-value is smaller than alpha.
Classic overlap picture
Repeated tests from simulation
Type I false alarm
Beta / Type II missed signal
detected signal
correct no alarm
Controls
normal groups, equal spread
Jump ahead
Live rates
rates from repeated tests!
P-value. In this one-sided test, the p-value asks:
if H0 were true, how often would random sampling produce a standardized B minus A difference this large or larger?
A small p-value is evidence against H0, so we reject H0 when
p-value smaller than alpha.
Type I error is a false alarm when H0 is true, and alpha is its target probability.
Type II error is a missed signal when H1 is true, and beta is its probability.
Power is the probability of detecting the signal, so power = 1 - beta.
The overlap picture is an approximate picture for the test statistic; the repeated tests use the two-sample t-test p-value.