ALL IS NOT LOST

Nova Scotia, 2000

That's the Canadian Journal of Serious Literature she's reading

or maybe People Magazine

This was too long for a "Comment", so I'm making it a stand-alone "Post".  It refers to the article about the Stanford Prof I commented on earlier.  

As threatened, I am going to comment on the article about John P.A. Ioannidis which is the cover story of the latest Stanford Magazine.  That story was primarily based on an article that Dr. Ioannidis published in the journal PLoS Medicine, which I presume is an authentic and well respected  outlet for medical research.    Thus, we should  take Dr. Ioannidis’s ideas seriously.  However, after three laborious passes through his “Essay”, I think he blew it.  Here’s why.

In bold type, in the middle of the first page of the PDF containing this article is the phrase “It can be proven that most claimed research findings are false”. That’s scary.   However, to reach most of us the statement should have read  “It can be shown that most claimed research finding are based on faulty statistics”.  To the biostatistical crowd those two statements may be equivalent, but to the rest of us they aren’t.  Maybe readers of PLoS Medicine will get his drift but  the average reader of Stanford Magazine may get the wrong message entirely.  To most of us outside the medical world Ioannidis seems to be saying “So-and-so says that A is a cure for B, but I can prove that it isn’t.”  What he means to say  is “The proposition that A is a cure for B has not been established with proper statistical rigor.”  What we outsiders should realize is that A may still cure B but that formal statistical verification is lacking.  More experimentation may be in order, but the concept is not ready for the trash heap.

Here is a simple example from my field of research.  In paleomagnetism we frequently try to determine whether two mean magnetic directions are or are not “ different”.  To do this we first calculate the two means, as well as things we call “circles of confidence”.  The radius of a circle of confidence (which has as its center the mean direction) is determined by how many samples have been studied, and how much these individual sample directions vary (are scattered).  We also get to choose a “confidence level”; usually 95%, but the choice of confidence level fundamentally  is arbitrary.  If the confidence circles around the two mean directions overlap we state that “at the (95%) confidence level the two directions are not different.”  Note that we do NOT say  that “the two directions are different” because one time in twenty (for 95% confidence, on average) such a statement would be wrong.   We can change the radii of the circles by studying more samples, or by choosing  a different confidence level.  Unless the statistics are improbably, wildly indicative, any  geological conclusions need other types of evidence to be convincing.

I think the same is the case with medical statistics.  They use far more sophisticated  methods than we dumb geophysicists are used to, but they still must face up to the same limitations.  Statistics alone are not capable of establishing truth; they merely give the odds, so to speak.  Thus, don’t lose heart; the bulk of conclusions derived from modern  medical research are not false – in any practical sense of the word.

Ioannidis does give some useful “Corollaries” toward the end of his article that are, in some cases, obvious, but that are true and important nevertheless.  There are six such.  I will comment only on the ones I think I understand:

               The  smaller the study, the less likely it is to be true.  Natch.

2     The smaller the effect size, the less likely the study is to be true.  I take this to mean that if the process            studied has only a minute effect, it is hard to evaluate  correctly.  Natch, again

      The greater the financial and other interests and prejudices on a scientific field, the less likely that the research findings will be true.  Obvious, I think.  Did you know that many critical studies are funded by drug companies that stand to profit from a positive result?  That some of the funds involved are huge?  We need more support from "disinterested" institutions. Never thought I'd advocate for more governmental spending.

      The “hotter” the field, the less likely the findings are to be true.  His reasoning here is a little obscure (to me), but insofar as I understand, I agree.  From my own career, there was a feeding frenzy when Davy Jones introduced the “terrane” concept; lots of the studies done during that frenzy turned out to be crap

.

Okay, that’s plenty.  Thanks to those of you patient enough to read all the way to here.