Skepticism of scientific research

I recently wrote a series about health science myths, with the finale about the widespread biases in scientific research: http://gracefool.posterous.com/why-90-of-the-published-medical-informati...

It's essential and shocking reading, and highlights how we need to be skeptical about everything we're told, no matter how supposedly authoritative the source is.

Trackback URL for this post:

http://skepticsinthepub.net.nz/trackback/309

First thing I have to say is

First thing I have to say is that I would generally support the premise, we do have to be careful of bias and be sceptical even if it comes from an authoritative source.

I've got a few "buts" though. For a starter, I always read the stuff at the bottom. Patrick Holford is not a good source to recommend as further reading. If you are going to complain about bias, it pays not to use a source someone whom is not qualified and makes their money from supplements, that notably don't get tested *at all* before launching them on the market. He's got a vested interest in diminishing the credibility of pharmaceuticals, if you are complaining about bias and vested interests it should be equal opportunity for all, not just medical science. See: http://holfordwatch.info/holford-myths/

"Why 90% of the published medical information that doctors rely on is critically flawed"

One big, big, thing I think you miss here is that medical information is not resting entirely on scientific studies saying this and that. Knowledge of anatomy and physiology and biological plausibility count for a lot too and are fundamental in decision making. There are even things in medicine that have never been subjected to testing. It's stuff like suturing wounds and plaster casting broken limbs, but I think it's obvious that those things are necessary and that they work on the basis of biological plausibility and what we know about wound and fracture healing. This does mean that a lot more is required that just a study on X to determine what to do about a clinical case. In addition, many results are in fact rigorous and have been replicated so there is plenty there that is solid - like the lung cancer/smoking link. That's my way of saying "it's more complicated than that."

"I first encountered the problem at university when we were told what the result of our lab work should be and taught how to manipulate our results."

Do you mean being taught how to set up an experiment there? In which case I know what you mean, and it is that at the start our tutors set up experiments with entirely predictable results (one I remember was testing blood types) and told us exactly how to set it up to get that result. That was because if another result was found or we couldn't get a result, we'd have to have mucked up somehow. It was about learning how to do an experiment, nothing wrong with that and a great way to work on the techniques required as well as how to structure experiments so you can get a result full stop. Otherwise, we did a lot of other cool stuff like microbiological experiments and were required to set up an hypothesis to test and then let loose to do that. As for the bit about fudging results, I get that to. It's true a "positive" study is more likely to be published than a "negative" study, but I bet if you asked they'd also say that acknowledging that doesn't translate to outright misrepresentation of results as seems to be suggested here nor that we can't often learn from negative results and they can be important in their own right. The work towards a HIV vaccine is a good example, it's failed many times but each time the scientists learn more about how HIV works and this can actually help in developing an effective vaccine. It's not just about being right.

Getting on to Ioannidis, one thing I'd say is never rely entirely on what a journalist says, even if it appears to be a direct quote. Best thing to do is check multiple sources and/or try and get the original source material. What is being presented may not be the complete story or fully represent what the researcher is intending. Then try and understand the context and the thing or scientific principles you are looking at. Often, people don't actually know as much as they think they do about medical science and it can show unless you work on getting some basic background knowledge of what you are talking about.

One point here that is missed is that medical science changes with the evidence and that is one of it's strengths, and I can attest that if I think back over the last twenty odd years things have changed vastly as we've gained more knowledge and we are getting better outcomes for patients in that. One of the real messages here is that poorer quality, un-randomised trials are unreliable and for randomised trials, replication is required if at all possible. Getting that necessary replication can be a problem when resources are scarce. That's a good one to know. Getting back to what I said about biological plausibility, the likeliness of a trial having a solid result reduces the less biologically plausible the thing studied. You might like to think about that in relation to studies of so-called alternative medicine. As an example Reiki or acupuncture invokes unmeasured and unmeasurable "energy" and testing this as if it's real is going to cause a problem with false positives arising. I'd say that it could be a problem if he's lumping those in with medical studies that test a biologically plausible hypotheses, as that muddies the water when we want to look at only medical studies that are testing something real and see if they have any problems.

In other cases, the bias isn't always what it seems - it can be towards publishing new and startling results (in other words, work that might be controversial and against accepted science) rather than bog-standard replications or just generic positive findings. The subtlety though (assuming the work is rigorous enough to pass muster) is new findings are built upon the old findings and often it's modification that occurs rather than an need to outright discard all the other studies because they are false. It's a bit like how we didn't need to discard Newtonian physics when Einstein came along.

"...however the establishment can't admit there is a problem."

*cough, cough* "To say that Ioannidis's work has been embraced would be an understatement. His PLoS Medicine paper is the most downloaded in the journal's history, and it's not even Ioannidis's most-cited work--that would be a paper he published in Nature Genetics on the problems with gene-link studies. Other researchers are eager to work with him: he has published papers with 1,328 different co-authors at 538 institutions in 43 countries, he says. Last year he received, by his estimate, invitations to speak at 1,000 conferences and institutions around the world, and he was accepting an average of about five invitations a month until a case last year of excessive-travel-induced vertigo led him to cut back."

I also don't find it that horrifying that scientists are human beings, with the normal set of flaws. That's why we use the scientific method, to try and reduce that bias and personally I find the truth outs itself even if it's not clear at the outset. We've got methods for dealing with potential bias, such as scientists being required to declare actual or potential conflicts of interest. Medical science isn't scared of these issues, but want to know about it from the look of it and will take it into account when assessing a study. I think you have to give medicine a bit more credit than that, and realise that health professionals can assess the evidence and put it into context. I'm a health professional myself and slightly mystified as to who this "establishment" is though. That's not it, medicine is not some undefined monolithic entity that is inflexible and never changes. The various specialities within medicine tend to counter any effect like that, they all have their own interests.

Lot's more I could say, but I think you could go back to the drawing board on this and the other stuff on your blog and relook at this. Probably the real problem there is communication, it's a problem when people assume that science is all about being right or that scientists are unbiased robot-types or that they can't see the problems and flaws inherent in science and work around them or that journalists are telling the truth and don't have biases themselves. Journalists can be a particular problem, as with little scientific knowledge often make a real hash of things when doing a story. In fact, with journalism it's a real problem that they'll report stuff that seems to be alarming to sell newspapers(study shows MMR causes autism!!!) but often don't get around to show the multiple studies refuting exactly that claim - it's just not as sexy to say MMR doesn't cause autism. Education in science and how it works can solve that.

P.S. There's a lot of junk science claims around cholesterol too - http://www.sciencebasedmedicine.org/index.php/the-international-network-... and http://www.sciencebasedmedicine.org/index.php/cholesterol-skeptics-strik... Not new or surprising stuff.

Thanks heaps for your reply

Thanks heaps for your reply Michelle!

Cheers for the heads-up on Holford.

Good point on my headline, it is misleading. I'm guilty of using shock tactics to get attention :p I've changed it to "Why 90% of published medical studies are critically flawed".

As for medicine "not resting entirely on scientific studies" - I would say there are still scientific studies, just less formal. Saying things have never been scientifically tested seems a bit silly to me - plaster casting, suturing, have been subjected to (informal) testing millions of times. We have plenty of evidence to show what happens when those techniques aren't used. That less formal experience is nonetheless still scientific - the wide-ranging repeatability of the results more than makes up for the lack of rigour. And when you say some results *are* rigorous, you're saying there *is* support from scientific studies.

I was saying that we got taught how to fudge our results even in first year university. Of course I understand the desire to get the right results for predictable things, but often bad science happens when a scientist fudges results because they think something is predictable (and then a latter, better study shows they were wrong). Fudging is more common than we'd like to admit.

Too right, my statement about "the establishment" was stupid. I've removed it.

And thanks for the cholesterol links - I disagree with my February post now, but it was interesting to learn more about it, to solidify in my mind why the consensus view is best.

I think my posts on the shoe industry (https://www.facebook.com/gracefool/posts/150231125040416) and anti-depressants (https://www.facebook.com/note.php?note_id=10150127929393926) are more interesting, as they are more controversial, and I have personal experience with both.

We're in agreement. My problem is more with the way laypeople regard science as an absolute thing, presuming it to be more accurate than it is. And it's not limited to laypeople, I've had seen terribly biased lectures delivered by university professors. Science is the best method we have, but we need to acknowledge the limitations and the slow progress so we're not taken in by breathless news reports. The fact that so many studies are overturned is only shocking because of the common misunderstanding of what science is and how it works.

As for medicine "not resting

As for medicine "not resting entirely on scientific studies" - I would say there are still scientific studies, just less formal.

I would disagree with this point.

If the studies are less formal they're unable to follow the scientific method. Thus, not "scientific studies".

Scientific

Fair enough. However it's still scientific information. Almost all facts have never been tested with scientific studies, and it would be a waste of time to do so. E.g. "a human finger will burn when pressed against a stovetop at 200C" is a fact inferred from other science, although there is no scientific study for it.

I would call them

I would call them observations. I believe we have tested how human tissue responds to burning and also have that practical experience as burns do happen out here in real life, it's a bit more than just inference. I think you missed the point a bit that in medicine, it deals with real people with real health care needs. It's more than what any particular study says, and everyone, including doctors know that all studies to some degree have flaws.

try chiropractic on Bellevue

try chiropractic on Bellevue chiropractor

Bellevue Chiropractic Therapy works

If you work with a spinal neurologist that can help you without drugs or surgery, that is what a Doctor or Chiropractic is.

Look us up on our Bellevue Chiropractor website and ask a question!

Wondering if this is a spam

Wondering if this is a spam bot...

probably

definitely spam to be deleted anyway