Policy Forum: A Comparison of Frameworks Evaluating Evidence for Global Health Interventions

PLoS Medicine
(Accessed 13 July 2013)

Policy Forum
A Comparison of Frameworks Evaluating Evidence for Global Health Interventions
Jill Luoto, Margaret A. Maglione, Breanne Johnsen, Christine Chang, Elizabeth S. Higgs, Tanja Perry, Paul G. Shekelle
PLoS Med 10(7): e1001469. doi:10.1371/journal.pmed.1001469
Published: July 9, 2013

Summary Points
:: Evidence-based decision-making is critical to informing policy in global health interventions and programs.

:: Existing frameworks for evaluating evidence that were developed or recommended for community or public health decision-making vary in their criteria and application.

:: We compared how different community or public health evidence frameworks assessed the same body of evidence for three advocated global health interventions and find there can be substantial differences in the rating of evidence, which could contribute to differences in policy recommendations.

:: All current frameworks emphasize effectiveness, and have shortcomings on other important factors into policy decision-making such as costs, implementation issues, context, and sustainability.

:: As global health policymakers move towards evidence-based approaches, we find a gap between what is currently available and the needs for an evidence framework appropriate for application to a global health setting in a low- and middle-income country context. More work is needed to either adapt one or more existing frameworks, or to develop an entirely new framework to meet the needs of policymakers and others responsible for implementing global health interventions.

A major movement in global health and development in the past 10 years has been the enthusiastic adoption by many of randomized controlled trials (RCTs) from the field of medicine to represent the most rigorous method to evaluate a program’s causal impact [1][4]. More recently, this movement has brought about a conceptual debate in global health and development about the proper role for RCTs in informing policy, with increasing efforts to “mind the gap” [5] between the evidence generated by RCTs (which focus on internal consistency) and the larger policy questions at the level of communities or populations (which require, among other things, generalizability) [4],[6][10]. The field of medicine that developed the RCT also developed the concept of “evidence-based” medicine that aims to improve health policy decision making by encouraging policymakers to base their policies on the best available evidence. Large international policy-making bodies appear set on applying a similar concept to global health and health systems research [4],[11]. In order to be evidence-based, decisions about global health interventions must consider the available evidence in terms of its quantity, quality, and relevance. Rather than use implicit judgment or other ad hoc methods, in evidence-based medicine it is now advocated and common practice to use a formal framework for considering the evidence as part of a systematic review, the advantages of which include increased transparency and better decision-making. Formal frameworks for evaluating evidence about community-level public health interventions have been proposed and advocated for similar reasons [12][17]. These frameworks differ in the degree to which they weight the importance of data from RCTs as compared to data from other study designs, the magnitude of potential benefits and harms, the role of context and implementation, and other factors. At present, there are no commonly accepted guidelines within global public health for how to evaluate evidence, and there is scant evidence to guide policymakers when selecting a framework to use for assessing a body of evidence about a global health intervention. We sought to assess how summary conclusions about the evidence for interventions or programs currently in use or proposed for wide adoption could be influenced by the choice of framework. Consistent results across frameworks would increase policymakers’ confidence in using and applying evidence frameworks, and may thereby help to narrow the gap between the questions asked by global health researchers and policymakers. Inconsistent results would call for a re-examination of current frameworks in terms of the domains they assess and the ways in which they are applied.