As any meteorologist will tell you, making long-term forecasts is a tricky business. To be relevant, forecasts should focus farther into the future than most people can predict, but only as far as can be reliably inferred through analysis of current data. Policymaking is a type of forecasting in that it involves the prediction of future benefits. Evidence-based policy aims to back up forecasting with data and findings derived from rigorous research. However, when the evidence is flawed, the predicted benefits may not materialize and the actual effects may be undesirable. Policymakers can avoid the pitfalls of faulty forecasts by considering the collective body of evidence before instituting policy or program changes.
An early example of forecasting risk is Paul Ehrlich’s "Population Bomb" argument. In 1968, his research suggested that huge, unsustainable increases in global population would occur within a few decades. As a result, he recommended policies such as increasing taxes on individuals because they have larger families. But the extreme population increases did not materialize, even amid society’s rejection of Ehrlich’s policy proposals.
Research has spurred a vast number of important changes in policy and practice throughout modern history, often bringing about important results. As far back as the mid-1800s, studies revealed that it was the hands of health care workers that often transmitted hospital-acquired diseases; thus began efforts to promote handwashing as a means of infection prevention. Research findings from the 1970s on the role of chlorofluorocarbons (CFC) in damaging the ozone layer eventually led to the 1987 "Montreal Protocol" international agreement to phase out CFC industrial production (substitute, less toxic compounds were readily available). And research published in the 1990s showed the negative health effects of transfats (good substitutes were available here, too). However it was only in 2006 that the U.S. started to require that it be listed on food labels. This change has led to many companies removing transfats from their products.
Today, lawmakers’ embrace of policies supported by data-driven evidence is growing ever tighter. This year, both political parties are using the evidence-based policy theme to justify their positions on the federal budget. For example, Office of Management and Budget Director Mick Mulvaney has argued that public programs such as Meals on Wheels or after-school care should be cut given the lack of “demonstrable evidence” that they are fulfilling their purpose. Democrats have enlisted experts to show, for example, that reducing funding to the National Institutes of Health will slow advances in cancer research and therefore treatment. In addition, a growing number of public programs, like the Investing in Innovation Fund, are working with grantees to expand the use of practices demonstrated to have benefits.
Meanwhile, we’re finding that some evidence used to validate policy changes is less conclusive than formerly assumed. For example, recent research in the BMJ shows that a reduction in the intake of saturated fat in order to lower blood cholesterol does not necessarily translate into lower risk of death from coronary heart disease. This means that the benefit of replacing saturated fat with vegetable oils may have been overestimated, even though myriad dietary guidelines call for such substitution.
Even second-hand cigarette smoke may be less harmful than previously thought. For example, a study published in 2004 found that, during the first six months after a “smoke-free ordinance” took effect in Helena, Montana, the number of heart attack victims admitted to the hospital dropped by nearly 40 percent. However, other researchers seeking to replicate the findings in similar populations reported much lower numbers. Furthermore, a 2016 study in Medical Care Research and Review found that health benefits thought to have resulted from bans on indoor smoking actually resulted from the imposition of cigarette taxes (reducing demand for cigarettes).
Basing policy on reviews of large bodies of literature is a way to safeguard it from problems with research quality. The high quality of the saturated fat study published in the BMJ is clear: the authors used a systematic review to identify five relevant clinical trials that could be combined in a “meta-analysis,” supporting consideration of the trials in combination and producing results that surpass the trials’ individual predictive power.
Systematic reviews involve an exhaustive five-step process and result in comprehensive syntheses of all evidence relevant to a given research question. Like the research they review, systematic review findings may be communicated via publication in academic journals as well as through platforms that are more accessible to the policymakers who need the findings to make informed decisions, such as through briefings, fact sheets, data visualizations, videos, and other innovative channels. I’ve published two reviews that relate to health policy: one focused on the cost-effectiveness of different melanoma therapies and another on the impacts of physician payment methods on health care costs, outcomes, and quality.
Indeed, evidence-based policy is becoming standard in medicine and health policy as well as in other fields like education, labor, home visiting, and pregnancy prevention. For example, a publication I co-authored last year with Stuart Peacock and others regarding the cost-effectiveness of genetic testing in cancer treatment has led to policy change in British Columbia, Canada. However, the benefits should be more fully extended to all areas of public policy. Several federal agencies have established clearinghouses to make high-quality evidence more accessible, but the frequency and ways that policymakers use the information lacks consistency. Better alignment of research questions with the most pressing policy and program topics will help increase this consistency. And consideration of the comprehensive body of evidence through systematic reviews will help ensure that policies are grounded in the highest quality research available, chasing away storms caused by poor data and analysis.
