A few weeks ago, the British medical journal the BMJ published an article that offered a critical appraisal of Tamiflu. While there have been other bits of research that challenge the efficacy of Tamiflu and the decisions of governments to stockpile the medication in anticipation of a flu pandemic, this most recent article was remarkable because it was a systematic review and was prepared as part of the Cochrane reviews.
For the uninitiated, a systematic review is quite different from an individual study. Rather, as the Cochrane Library puts it: “A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a given research question. Researchers conducting systematic reviews use explicit methods aimed at minimizing bias, in order to produce more reliable findings that can be used to inform decision making.” And Cochrane reviews have been described as “the highest level of evidence on which to base clinical treatment decisions.”
Serious stuff. In other words, when a Cochrane review is published, people stand and take notice.
The political result of the publication of the review critical of Tamiflu was a chorus of criticism of governments that had stockpiled the medication at a cost of millions of dollars, pounds, Euros and yen. Some even have begun to refer to the vaccine as “Scamiflu”.
This does not mean, however, that systematic reviews, be they from Cochrane or elsewhere are faultless. Therefore, almost as soon as it was published, the article that was so critical of Tamiflu attracted its own chorus of criticism (see also here and here). Critics pointed to what they saw as flaws in the analysis and argued that the study as such did not justify the conclusion that governments were wrong to stockpile Tamiflu. Some pointed out that in the real world of policy making governments have to manage risk and that the Cochrane systematic review did little to consider.
I am not qualified to evaluate the claims and counterclaims. And I am cynical enough to ask about the motives of the various parties in the debate. Having said that, governments are routinely faced with having to make decisions when the evidence is conflicting and where they must balance clinical effectiveness against public confidence that the health system will be able to protect them from harm. In this case governments opted to stockpile Tamiflu, knowing it might only be somewhat useful. However, in the face of the prospect of large number of people dying in the event of a serious pandemic, being able to do something, however limited, is preferable to being able to do nothing. This is the real world of politics and difficult political decision making.
More generally, I think that a systematic review critical of Tamiflu appeals to those of us who are, justifiably I think, quite critical of large pharmaceutical companies and their motives and ultimate goals. In other words, combine a nascent mistrust of Big Pharma, add a systematic review done by the Cochrane collaboration no less, and you inevitably get a torrent of criticism not only of the drug but of the governments who spent large sums of money acquiring it.
Yet it is in the nature of scientific inquiry to subject any given study to critical appraisal and this case was no different. Systematic reviews, even the best ones, are not the final word on an issue. And when it comes to translating scientific research into policy, systematic reviews, be they good, bad or indifferent, are but one factor shaping policy decisions. And as I and many, many others have argued, this is as it should be. Scientific evidence shapes and informs public policy but does not and indeed cannot determine policy.