Scientific evidence alone is not sufficient basis for health policy


This photo shows a view down the middle of a boron nitride nanotube. Credit: Vin Crespi, Pennsylvania State Physics. Distributed under the Creative Commons license

Earlier this year two senior officials from the US and UK governments got together to write a short commentary in the BMJ provocatively (but appropriately) entitled “Scientific evidence alone is not sufficient basis for health policy“.  Some highlights:

Based on experience as researchers and as policy makers at the White House and United Nations, we argue that although science should inform health policy, it cannot be the only consideration.

Although it may frustrate scientists when politicians are swayed by the possible electoral consequences of various policy options, few scientists (including us) would want to live in a society in which politicians completely ignored the views of those who have elected them as their representatives. Voting, free speech, debate, and the push and pull of politics must have an important role in what free societies choose to do if the concept of democracy is to be meaningful.

To say that an advocate or policy maker is guided by more than technocratic considerations is a compliment and not an insult.

In the rapid response to their commentary someone effectively asked if it is possible to find anyone who would defend the position that science alone should determine (health) policy.  Admittedly, this may be a challenge.  However, the implicit assumption in so much of the writing in health policy is that policy decisions should be based on evidence.  The very large literature on knowledge translation, at least as it applies to public policy, seems to assume that the problem to be solved is finding the best ways to bring to the attention of “policy makers” (who are rarely if ever defined very clearly) the best available scientific evidence.


Source: Safe Kids Kansas Coalition


Debating the merits of Tamiflu

Source: Social Science Space April 15, 2014

Source: Social Science Space April 15, 2014

A few weeks ago, the British medical journal the BMJ published an article that offered a critical appraisal of Tamiflu.  While there have been other bits of research that challenge the efficacy of Tamiflu and the decisions of governments to stockpile the medication in anticipation of a flu pandemic, this most recent article was remarkable because it was a systematic review and was prepared as part of the Cochrane reviews.

For the uninitiated, a systematic review is quite different from an individual study.  Rather, as the Cochrane Library puts it: “A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a given research question. Researchers conducting systematic reviews use explicit methods aimed at minimizing bias, in order to produce more reliable findings that can be used to inform decision making.”  And Cochrane reviews have been described as “the highest level of evidence on which to base clinical treatment decisions.”

Serious stuff.  In other words, when a Cochrane review is published, people stand and take notice.

The political result of the publication of the review critical of Tamiflu was a chorus of criticism of governments that had stockpiled the medication at a cost of millions of dollars, pounds, Euros and yen.  Some even have begun to refer to the vaccine as “Scamiflu”.

Source: Twitpic

This does not mean, however, that systematic reviews, be they from Cochrane or elsewhere are faultless.  Therefore, almost as soon as it was published, the article that was so critical of Tamiflu attracted its own chorus of criticism (see also here and here).  Critics pointed to what they saw as flaws in the analysis and argued that the study as such did not justify the conclusion that governments were wrong to stockpile Tamiflu.  Some pointed out that in the real world of policy making governments have to manage risk and that the Cochrane systematic review did little to consider.

I am not qualified to evaluate the claims and counterclaims.  And I am cynical enough to ask about the motives of the various parties in the debate.  Having said that, governments are routinely faced with having to make decisions when the evidence is conflicting and where they must balance clinical effectiveness against public confidence that the health system will be able to protect them from harm.  In this case governments opted to stockpile Tamiflu, knowing it might only be somewhat useful.  However, in the face of the prospect of large number of people dying in the event of a serious pandemic, being able to do something, however limited, is preferable to being able to do nothing. This is the real world of politics and difficult political decision making.

More generally, I think that a systematic review critical of Tamiflu appeals to those of us who are, justifiably I think, quite critical of large pharmaceutical companies and their motives and ultimate goals.  In other words, combine a nascent mistrust of Big Pharma, add a systematic review done by the Cochrane collaboration no less, and you inevitably get a torrent of criticism not only of the drug but of the governments who spent large sums of money acquiring it.

Yet it is in the nature of scientific inquiry to subject any given study to critical appraisal and this case was no different.  Systematic reviews, even the best ones, are not the final word on an issue.  And when it comes to translating scientific research into policy, systematic reviews, be they good, bad or indifferent, are but one factor shaping policy decisions.  And as I and many, many others have argued, this is as it should be. Scientific evidence shapes and informs public policy but does not and indeed cannot determine policy.

Policy as Values

In July 2012 I published a short piece entitled “Policy as Values” in the newsletter of the Institute for Science, Society and Policy. While this was a response to an earlier item by my University of Ottawa colleague, Scott Findlay, I think my contribution stands on its own and may have continuing interest to a wider audience.

I am reproducing the piece here since it is no longer avaialble in its entirety from the ISSP.

ISSP logo

Policy as Values

Patrick Fafard

June 2012

Long ago and far away I took a course in philosophy of the social sciences. Surprisingly, this philosophy course involved a field trip and a very peculiar one indeed. On campus there was a small office in a bit of commercial space that housed, if memory serves, Technocracy Inc. The organisation was committed to advancing the cause of a rational and scientific approach to life in general and government in particular.

I have repeatedly encountered variations on this desire for a scientific approach to government, most recently in the blog post by my University of Ottawa colleague Scott Findlay provocatively entitled (at least for me) “Policy as Science”. To summarize and oversimplify, he advocates a rational policy process that selects among candidate policies in a rational manner, using the scientific method. My first inclination was to discount if not dismiss the argument by simply asserting that in real life things are not so simple and grumble about the hubris of scientists.

But the very fact that the desire for a rational approach to policy making is always present and, at first glance, quite seductive, demands a more fulsome response. There are any number of possible objections: much has been written on why policy making is not rational and indeed cannot be so. In fact, I have argued, as have many others, that policy making should not be reduced to rational problem solving.

But for the moment I want to focus on one aspect of the argument for a more rational or scientific approach to policy making. To reduce policy making to problem solving, as Scott suggests, assumes that we can agree on the nature of the problem and on the desired outcome. Let us consider each of these claims in turn.

Before there can be a policy choice there has to be agreement that there is a problem to be solved or at least one that government can do something about. In effect then, the art of governing is to choose the problems that will be addressed and which are to be more or less ignored. However, among the many challenges of governing is the simple fact that we do not always agree on the nature of the problem. I say Canadians eat too much salt, which causes widespread high blood pressure. Others say that there is no scientific consensus on the matter and scoff at the idea that there is a problem to be solved. In effect, fighting over how to define the problem and the science underlying problems is often a big part of the policy process – witness the debates about climate change or drug addiction.

Assuming we can agree on the nature of the problem (or at least most of us can, at least for a time) according to Scott a rational policy process would see us choose the option that is most likely to achieve desired outcomes. However, we are confronted by the reality that we are unlikely to agree on what is desirable. Policymaking is never only about solving a problem. It is addressing a problem in a way that is acceptable to at least some citizens some or most of the time. It is making decisions that advance a broader overall agenda if not a broader philosophy. It is addressing public concerns in a politically prudential way.

To return to the case of dietary sodium, we have no way of demonstrating unequivocally whether what is required is social marketing, industry self-regulation or government regulation of the food industry. And even if it could be shown that government regulation of the food industry is the optimal way to reduce the amount of salt in our diet, small-c conservative governments are unlikely to want to do so on broadly philosophical grounds.   Government regulation of food raises concerns about undue government influence in the lives of citizens. The latter objection cannot be resolved with reference to science alone. It is a normative claim and requires a different kind of reasoning altogether.

In effect, most of the truly interesting and non-trivial policy issues do not lend themselves to rational decision-making. Why? Because they involve disagreements over values and such disagreements, as Hume reminded long ago, cannot be resolved with reference to science alone.

Why INSITE is a not a good case study of evidence-based decision making.

Calls for evidence-based policy are routine.  And in Canada, the saga of the Vancouver safe injection facility INSITE is cited as an example of where politicians tried to ignore the evidence.  Consider two recent examples.

Source: iPolitics

Source: iPolitics

On Tuesday my university hosted a panel discussion by leading politicians in federal politics with an overriding theme of why young people should engage in the electoral process and, if nothing else, vote.  A laudable effort to be sure.  As is common with this type of event there were the usual questions of the audience.  Interestingly enough, the first such question asked the assembled leaders to comment on the place of evidence-based policy making.  Needless to say, all were quick to endorse the idea, some with more nuance than others.  Of particular interest were the comments of the New Democratic Party leader Thomas Mulcair who went out of his way to point to the efforts by the current Conservative government to shut down INSITE as an example of ignoring evidence in policy making.

Consider as well a recent blog post by my University of Ottawa colleague Scott Findlay where he argues that the Supreme Court of Canada’s decision to overrule the Government and insist that INSITE must remain open was about the proper use of evidence.  In his words, “evidence was critical to the Court’s decision”.  (Note that Findlay’s larger argument is not so much for evidence-based policy making as it is for transparency and openness).

Does this mean that the story of INSITE is the story of how evidence can and should influence public policy?  I think not.

As I argued in a paper in the Journal of Urban Health a few years ago, that INSITE continues to operate is not the result of a straightforward application of evidence to a public health intervention. In that article I argued that, on the contrary, “INSITE is the result of coalition building, the mobilization of public opinion, lobbying, and political and ideational struggle.” Without a doubt the use of evidence by the Supreme Court of Canada was a critical part of the story.  But to focus on that is to miss the fact that INSITE exists as a result of a complex combination of factors of which scientific evidence is but one.

hb_coverWhat is more, for many policy problems, this is the predominant pattern.  Evidence does play a role in policy making but it is most influential, as Roger Pielke puts it in his book The Honest Broker: Making Sense of Science in Policy and Politics, “in circumstances where the scope of choice is fixed and the decision-maker has a clearly defined technical question”. So, it is not that scientific evidence is not important, it is that its role is variable.  In other words, while we may want evidence-based decision making, only some decisions can be, or for that matter should be based on evidence.


The Supreme Court as a window on federalism (again)

So, the Supreme Court of Canada has ruled on the constitutional validity of the appointment of Mr. Justice Nadon.  In essence, the majority took the position that to be appointed to the Court from Quebec a person must be a current member of the Quebec Bar or a sitting judge in Quebec.  Marc Nadon was neither at the time of his appointment so he was deemed to be ineligible.


Source: Canadian Press Archive/ La Presse

In their decision the majority wrote that the legislation creating the Supreme Court requires that three judges be from Quebec (Section 6).  The go on to argue that: “The purpose of s. 6 is to ensure not only civil law training and experience on the Court, but also to ensure that Quebec’s distinct legal traditions and social values are represented on the Court, thereby enhancing the confidence of the people of Quebec in the Supreme Court as the final arbiter of their rights.”

In an analysis of the Nadon decision University of Montreal law professor Jean Leclair goes a bit further and argues that: “Bref, une compétence technique en droit civil ne suffit pas pour bien représenter le Québec. La légitimité de la Cour suprême, déjà l’objet d’une démonisation au Québec, requiert du candidat qu’il puisse se réclamer d’une appartenance étroite et contemporaine à l’univers québécois.”

In a highly critical review of the decision, Andrew Coyne disagrees with the reasoning of the majority, but concludes that “If ever there were an argument for a more robust process of legislative review of such appointments, this is it”.

I agree.

But legislative review of appointments need not, and indeed should not, be limited to the House of Commons.  As I suggested a few weeks ago on this blog, I wrote “Had the Harper government been required to publicly and formally consult with the Quebec government, it is almost certain that the person nominated would not have been so controversial with the risk of undermining the legitimacy of future decisions by the Supreme Court of Canada.”

A fulsome discussion of the merits of a provincial role in nominations to the Supreme Court of Canada is unlikely in the near term.  But it is plausible to imagine a Liberal government in Ottawa and a Liberal government in Quebec City.  It is also plausible to imagine that a new federal government will want to improve on the current flawed process of naming judges to the highest court in the land.  In anticipation, best perhaps to read up on both the Meech Lake Accord and on the general matter of how we name judges to the Supreme Court of Canada.


Political perspectives in public health

For several years now I have been pondering the intersection between public health and political science and between public health and politics.  How, where and with what effect the two intersect is manifest in a number of dimensions, some scholarly, many decidedly practical and real (my earliest effort in this area was published in 2008).

One of the perennial debates in public health generally and health promotion in particular is the role of “individual” vs. “societal” changes in improving the health of populations.

The “individual” perspective on public health (let’s call it IPH for short), emphasizes the importance of each of us making small and not so small changes in how we live our lives as the key to improving our own health and, collectively, the health of the population overall.  As my University of Ottawa colleague Yoni Freedhoff recently put it “what we choose to do with our forks and our feet” has significant implications for our health.

The “societal” or “structural” perspective on public health (let’s call it

John Locke

John Locke
Source: Wikimedia Commons

SPH for short) argues first, that societal or structural changes are required to improve population health, and second, that absent these structural changes, it is difficult if not impossible for many people to make the individual changes required to improve their own health (e.g., the poor cannot eat well if they live and “food deserts” and do not have access to health food choices where they live).  There are various versions of SPH from those that emphasize changes to our built environment (e.g., how we design and build cities) all the way to those that argue that the structural changes required go much deeper and point to the design of the welfare state.

My working hypothesis is that there is a set of relationships between these different perspectives on what matters for public health and different ideological if not philosophical positions.  IPH clearly has it roots in traditional liberalism with an emphasis on the central importance of individual choices about the good life.  Conversely, an SPH perspective is linked to a more social democratic interpretation of what makes the world go ’round.

In contemporary politics the IPH perspective is by far the stronger of the two (or at least something that most governments can agree on) and gives rise to the bureaus, agencies and departments devoted to health promotion.  The SPH perspective is not completely absent but pops up more selectively (e.g., the largely ineffectual interest of the government of Tony Blair in health inequality; the WHO Commission the Social Determinants of Health – which itself has been accused of being excessively liberal in its orientations).


Source: Oxford University Press USA

This is by no means a novel observation, at least in public health circles. Nor is it anything close to a complete account of the different perspectives in public health (for this you would be well served to read Nancy Krieger’s magisterial survey of public heath theory).

I draw attention to it because it would seem that there is a growing awareness in the broader population that health is more than health care (a truism among students of public health).  Yet, when a well-intentioned citizen or politician wishes to take the claim seriously, she or he is immediately confronted by quite different accounts of what it means to tackle public health using tools beyond the health system.  In other words, perennial debates inside public health are being moved into a broader arena.  This means that the usually polite disagreements between proponents of different flavours of both IPH and SPH accounts of what keeps up healthy risk being transformed into a more partisan, and therefore more heated and perhaps partisan discussion if not debate.

And this could be a good thing.

“To write or even speak English is not a science but an art. There are no reliable words. Whoever writes English is involved in a struggle that never lets up even for a sentence. He is struggling against vagueness, against obscurity, against the lure of the decorative adjective, against the encroachment of Latin and Greek, and, above all, against the worn-out phrases and dead metaphors with which the language is cluttered up.”

George Orwell, The English People