Suffering from low p?

Suffering from low p?

Statistics makes my head hurt. I took, and dropped, statistics once a year for 4 years in college. It wasn't the math. My undergraduate degree was physics. It was the concepts. Once they got past flipping a coin and a bell shaped curve, my brain froze.

While statistics are important in medicine, they can be misleading. One of the clear concepts that I have learned is a p of 0.05 might be "statistically significant" it is neither clinically important or, more importantly, proof that the effect is real.

We want to know if an effect is true, that it is real, and many think think that if a p value is 0.05 then the effect is the real deal.

Not so.

It would appear that for an effect to be the real deal, taking into consideration Bayes, the p value should be 0.005 or better yet, 0.001. Yes, I know that is a  misrepresentation, but as a clinician trying to apply a heterogeneous literature to patients, I need some cut off that gives me a general idea that an effect is real.

The problems with the p value are legion and well described on the internet. I like DC's Improbable Science's lucid discussions of the topic.

Unfortunately, possibly misleading p values are rampant in the medical literature, as noted in Evolution of Reporting P Values in the Biomedical Literature, 1990–2015

They used

Automated text-mining analysis was performed to extract data on P values reported in 12 821 790 MEDLINE abstracts and in 843 884 abstracts and full-text articles in PubMed Central (PMC) from 1990 to 2015.

What has happened over 25 years is interesting

The distribution of reported P values in abstracts and in full text showed strong clustering at P values of .05 and of .001 or smaller. Over time, the "best" (most statistically significant) reported P values were modestly smaller and the "worst" (least statistically significant) reported P values became modestly less significant.

It is curious that the p values tend to cluster around 0.05, the arbitrary cut off for statistical significance.

I would bet that most of these studies are in fact not clinically relevant. A little data mining here, a little bias there, a touch of multiple comparisons and, with a bit of p hacking, you can get 0.05 and a publication.

It would be very interesting to apply their data mining techniques to just pseudo-medical studies. I suspect that virtually the entire literature of pseudo-medicine that is used to support it's practice hovers around the 0.05.

And a foundation of p = 0.05 would be an edifice built on nothing, as

P values do not provide a direct estimate of how likely a result is true or of how likely the null hypothesis ("there is no effect") is true. Moreover, they do not convey whether a result is clinically or biologically significant. P values depend not only on the data but also on the statistical method used, the assumptions made, and the appropriateness of these assumptions.

Of course pseudo-medicine uses science and statistics like the proverbial drunk and street light: for support, not illumination.

Points of Interest 04/09/2016
Points of Interest 04/07/2016

Related Posts