How Science Works. And Doesn’t Work.
This is a transcript of a real conversation between Doc Nolan and Johnny Science. This meeting of the minds occurred on September 26th, 2013
Doc: OK, I believe that Jim Nolan Therapy (let’s call it JNT for short) works. I want you to provide PROOF that it works.
Johnny Science: That is not really how science works…
Doc: Sure it does—I see 20 clients, you interview them afterwards and see how they are doing, and if all 20 say “I feel a lot better after JNT”, that would be proof that JNT works.
Johnny Science: Actually, Jim, that is not really “proof” and it’s not really science. That is a series of interviews, and the data are what we would call “self-report”, which is notoriously unreliable. Go ask 20 drunks at the bar if they have a problem. You’ll get 18 “no” answers. Those “self-reports” of “no problem” are just not reliable. And just to be clear, if 20 out of 20 clients said they really liked their work with you, I think you should feel really good about that. It’s not “scientific evidence”, but you should be happy about it anyway.
Doc: OK, I get that. How about this: You do a questionnaire or a survey BEFORE they see me, and then the same questionnaire or survey AFTER they see me, and if they show improvement on the “post therapy” questionnaire, that would be proof that JNT works.
Johnny Science: Well, better, but not exactly. Again, if the questionnaires show 20 out of 20 “improved”, that is a very nice thing. You should feel good. But that is again self-report, and it is simple pre-post stuff. Not very powerful. We’re not sure how you measured “improvement.” You don’t know if the JNT is responsible for their feeling better, you don’t know whether they were being completely honest on the questionnaires, and so on. You have a kind of correlational relationship, but there is no strong “evidence” that JNT is causally related to their self-reported improvement. Don’t get me wrong, those results sound great, but that is not great scientific method, and it is not “proof.”
Doc: OK, so what the heck IS the scientific method? How would Johnny Science approach this one?
Johnny Science: OK, well first, science, or the scientific method, does not start out with the goal of “proving” something.
Doc: That’s news to me. And half the planet, I would guess.
Johnny Science: Yeah, I know. Science kind of works backwards. Your hypothesis is “JNT works.” (We can operationalize “works” later.) So instead of trying to prove that hypothesis is true, here is what we do: We establish the “NULL HYPOTHESIS”, which is the negation of your hypothesis. The Null Hypothesis is “JNT Does Not Work.”
Doc: OK, yeah, that’s weird.
Johnny Science: I know. I am just the messenger, dude. But this is how it works. So, what we do, instead of attempting to prove your hypothesis, we test the null hypothesis, which is “JNT Does Not Work.” We do the questionnaires, the surveys, the measurements, the blood levels, whatever we decide to measure that would be indicative of positive change. Our hypothesis is that after 20 sessions with Jim Nolan, NOTHING we are measuring will be statistically significantly better. (Ideally, we would have another 20 clients on your waiting list and run them through the whole lineup of measurements, and then compare the two groups, but let’s keep this real…)
Doc: But wait–we think things WILL be better with JNT!??
Johnny Science: Of course we do. But that is not how science tests it. So we run your clients through the measurements, you see them for 20 sessions, and along 6 of the 7 measurements, 16 of the 20 participants show improvement.
Doc: So the Null Hypothesis that JNT does not work was proved false?!
Johnny Science: No, not exactly. The evidence we collected fails to support the Null Hypothesis.
Doc: Isn’t that the same thing as “proved it was false?”
Johnny Science: Nope. Sorry. I know it sounds similar. But no. The sample size was small, their improvement could be related to factors other than JNT—there are all kinds of other possible explanations for the findings. And again, I would LOVE to have those findings if I were a therapist, but they do not prove the Null Hypothesis false. They simply fail to support it. And as far as your REAL HYPOTHESIS, which is that JNT works, well, we can say it this way: “The data failed to disprove that hypothesis.”
Doc: Oh, yeah, Wow, that’s real exciting.
Johnny Science: Yeah, I hear you. Sounds pretty underwhelming, but in science, that is still a really good outcome. Any Science Guy would be pleased with those results, even if they do not constitute “proof.”
Doc: So I can’t say that “Scientific evidence proves the efficacy of Jim Nolan Therapy”?
Johnny Science: Nope. You can say “The evidence from one scientific study supports the use of JNT in the treatment of blah blah blah.” Or if you did a couple more related studies, you might say “The findings in the body of research on JNT are consistent with the claims in Nolan’s book that JNT is useful in treating blah blah blah.”
Doc: And then I could say JNT is evidence-based?
Johnny Science: Yep. Of course, the more studies you have, and the better the studies are, the more robust that statement is, but yeah, you could say that, and it would not be inaccurate.
Doc: So what therapies ARE proven to be effective??
Johnny Science: None. That is not what the scientific method does. People do not get that. Science does not set out to “PROVE” something. Science does not PROVE ANYTHING. It provides evidence that supports a perspective, or hypothesis. It fails to disprove hypotheses. But remember this: “WHENEVER you hear anybody say “Well, they have proved that…” or “Studies prove that cigarette smoking damages the lungs”, or “It is a well-known scientific fact that…” —they probably do not know what they are talking about, and certainly do not understand the scientific method. They may even have a good point to make, but in over-stating it, in attributing explanatory value to science that science simply cannot provide, they lose points and credibility. As soon as somebody hauls out the words “proof”, or “”prove”, I kind of check out, because I know they don’t know how to report this stuff, which means they probably do not know how to READ research, so I just don’t trust anything else they have to say. May sound dismissive, but that’s just how it is.
Doc: Well, thanks, Johnny. I am enlightened. Sort of.
Johnny Science: My pleasure, Doc.