Bulletin of the World Health Organization

Fair tests of health-care policies and treatments: a request for help from readers

Andrew D Oxman a & Iain Chalmers b

a. Norwegian Knowledge Centre for Health Services, Oslo, Norway.
b. James Lind Library, Oxford, England.

Correspondence to Andrew D Oxman (e-mail: oxman@online.no).

Bulletin of the World Health Organization 2009;87:407-407. doi: 10.2471/BLT.09.066787

Health-care policies impact on peoples’ lives. For example, a policy decision not to have publicly funded health insurance with universal coverage limits peoples’ choices to what they can afford. Those who make policy decisions are ethically and politically bound to make decisions that are in the interests of the people whom they serve. Evaluating the effects of policies is important because this is the only way of knowing the extent to which policies are doing more good than harm.

We would appreciate assistance from Bulletin readers to address the question: what is a fair test of a health-care policy? There are three ways in which you can help us:

1. Provide examples of randomized evaluations of health-care policies

There are many ways in which health-care policies can be evaluated. The advantage of randomized evaluations is that they help to ensure that like is compared with like. Recent examples of randomized trials of policies relevant to reducing neonatal mortality include evaluations of community-based behaviour change management,1 home care versus community care,2 and community participation with women’s groups.3 Examples of randomized trials of pharmaceutical policies include evaluations of the impacts of supervision on stock management and adherence to treatment guidelines, trials of strategies for improving adherence to prescribing guidelines and a trial of reimbursement restriction.46 An inventory of randomized evaluations such as these would make it easier for policymakers to find and use this evidence. Other benefits would include illustrating: (i) the principles of fair tests of health-care policies, (ii) the feasibility of randomized evaluations of health-care policies, (iii) practical strategies for addressing the challenges of randomized evaluations of health-care policies, and (iv) limitations of randomized evaluations of health-care policies.

Because these evaluations may be difficult to identify, we would be grateful if Bulletin readers could send us examples of randomized evaluations of health-care policies, particularly if they have not been published in journals, or have been reported in languages other than English. Specifically, we welcome examples of cluster randomized trials of arrangements for delivering, financing or governing health-care services, and of public health interventions.

2. Provide examples of compelling evidence from non-randomized evaluations of health-care policies

There may be practical reasons and other arguments for using non-randomized comparisons and sometimes these provide compelling evidence, particularly when there are dramatic effects of policies. Non-randomized comparisons include interrupted time-series analyses, controlled before–after studies, uncontrolled before–after studies and cross-sectional studies. We would be pleased if Bulletin readers could help us identify good examples of individual non-randomized evaluations of the impacts of health-care policies, or systematic reviews of them, including a brief comment on what makes the results of these evaluations compelling.

3. Provide early examples of treatment evaluations

Evaluations of preventive, therapeutic and rehabilitative treatments inform health-care policy decisions, as well as clinicians’ and individuals’ decisions about health care. An article in the news section of this issue of the Bulletin introduces readers to the James Lind Library (available at: www.jameslindlibrary.org) – a multilingual web site explaining and illustrating the features of fair tests of treatments,7,8 with explanatory essays in Arabic, Chinese, English, French, Portuguese, Russian and Spanish. These essays explain the need to: (i) make unbiased treatment comparisons, by minimizing differences between the people compared and in assessing treatment outcomes, including unanticipated outcomes; (ii) take account of the effects of the play of chance, when interpreting unbiased comparisons; and (iii) review systematically all the relevant evidence from fair tests, minimizing the effects of biased reporting and selection of evidence, and using meta-analysis to reduce the effects of the play of chance, when appropriate and possible.

To illustrate these principles and their application, the James Lind Library draws on an archive of hundreds of records, from ancient to modern times. The authors of these records come from many countries, and although the majority of their contributions are written in English, many of them are in other languages.

We would be grateful if readers would draw our attention to additional records eligible for inclusion in the James Lind Library if you know of examples outlining principles of fair tests of treatments and the need for comparisons published before the end of the 19th century, or examples of studies using alternation, random allocation or blinded outcome assessment published before 1950.

If you are aware of examples relevant to any of the three categories described above, please send us copies of them, identifying the key passages and providing a translation if the text is not in English, by post, facsimile or e-mail to: Bulletin of the World Health Organization Project, c/o James Lind Initiative, Summertown Pavilion, Middle Way, Oxford OX2 7LG, England. Fax: +44 1865 516 311; e-mail: feedback@jameslindlibrary.org. Your help will be acknowledged explicitly unless you instruct us otherwise. ■