The American Statistical Association (ASA) have released a statement warning about the misuse of the P value, following concern amongst its members that the P value was being misapplied. The statement published in The American Statistician contains six principles to guide in proper use of the P value:
- P values can indicate how incompatible the data are with a specified statistical model.
- P values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
- Scientific conclusions and business or policy decisions should not be based only on whether a P value passes a specific threshold.
- Proper inference requires full reporting and transparency.
- A P value, or statistical significance, does not measure the size of an effect or the importance of a result.
- By itself, a P value does not provide a good measure of evidence regarding a model or hypothesis.
The recommendations should encourage authors to disclose all statistical analyses and prevent them from drawing important conclusions based on P value alone.
The ASA’s executive director, Ron Wasserstein has warned that the P value was never intended to be a substitute for scientific reasoning, claiming “Well-reasoned statistical arguments contain much more than the value of a single number and whether that number exceeds an arbitrary threshold. The ASA statement is intended to steer research into a ‘post p<0.05 era.’”
P values are commonly used to test a null hypothesis by giving the probability of obtaining results at least as extreme as the ones observed, assuming that the null hypothesis is true. In such a test, the smaller the P value, the less likely an observed set of values would occur by chance. This can lead to rejection of the null hypothesis if the P value is less than or equal to a chosen significance level (typically 0.05). However, rejection of a null hypothesis based on the P value does not prove that the tested hypothesis is true.
The limitations of the P value have long been debated with one journal even banning its use. However, this is the first time the ASA board of directors have issued a statement to address the such a foundational statistical matter, which is being published alongside several articles that are intended to give more perspective on the topic.