MCE is based on an impact model describing how the introduction of the IMCI strategy was expected to reduce child mortality and improve child health and development. Computer simulations based on this model were carried out early in the development of IMCI, and demonstrated the need to improve key family-behaviour, especially care-seeking, for IMCI to achieve its full impact. On the basis of this impact model, advisers and technical staff developed indicators and data-collection tools. The impact model is now widely used as a basis for presenting IMCI in programmatic contexts as well as to guide the presentation and interpretation of MCE results. The main results from the MCE, interpreted in the light of the impact model, can be found in the paper entitled Programmatic pathways to child survival: results of a multi-country evaluation of integrated management of childhood illness, published in 2005 in Health Policy and Planning.

The MCE is not a standard multi-centre study using identical design in all sites. Instead, it employs a set of compatible designs, based on the stage of IMCI implementation in each country and on local characteristics. The generic steps in the evaluation are: 1) Collect baseline data on impact indicators and costs; 2) Begin IMCI implementation; 3) Ensure that implementation is adequate; 4) Wait two or more years for impact to become measurable, while providing feedback at all levels; 5) Collect and analyse outcome, cost and impact data; and 6) use the evaluation results to improve child health programmes. In Bangladesh, the MCE is fully prospective – that is, the evaluation was undertaken before IMCI was introduced. In the other MCE sites, IMCI was already being implemented prior to the evaluation, and the MCE therefore relied on a combination of prospective and retrospective data-collection tools. A full description of the MCE design is available at Evaluation of the impact of IMCI: design issues.

For each country, country teams developed designs for the MCE studies, with support from international collaborators and the central MCE coordinating team. The designs vary from a probability design (randomized trial) in Bangladesh to plausibility designs (pre/post comparisons between IMCI and control arms) in Brazil, Uganda and Tanzania, and an ecological design in Peru. For the studies in Bangladesh, Brazil, Tanzania and Uganda, before and after comparisons were made of intervention and control areas; in Peru 25 departments with different levels of IMCI were compared.

Work on designing the MCE led to questions about the common assumptions about the types of evidence needed to demonstrate the efficacy and effectiveness of public health interventions. In Evidence-based public health: moving beyond randomized trials, published in the American Journal of Public Health in March 2004, the authors argued that the probability approach, and specifically randomized controlled trials (RCTs), was often inappropriate for the scientific assessment of the performance and impact of large-scale interventions. The paper described the limitations of using RCTs alone as a source of data on the performance of public health interventions, and suggested complementary and alternative approaches that will yield valid and generalizable evidence.

The MCE was designed to overcome the limitations of RCTs for evaluating large-scale interventions being implemented under actual conditions, while at the same time preserving a high level of internal study validity. A paper entitled The Multi-Country Evaluation of the Integrated Management of Childhood Illness Strategy: Lessons for the Evaluation of Public Health Interventions, published in the American Journal of Public Health in March 2004, described the main characteristics of the MCE design. The paper Ten methodological lessons from the multi-country evaluation of integrated Management of Childhood Illness, which appeared in Health Policy and Planning in 2005, summarizes the methodological experience gained with the MCE.

The changes in process and health-impact indicators in large geographical areas in actual conditions may be due to the intervention or to contextual factors. To attribute the changes to the interventions, confounding factors, including contextual factors, should be known. The MCE team therefore developed tools for collecting information on changes in socioeconomic, demographic, environmental and other relevant factors over the course of the evaluation. Also, the delivery of other child health interventions in the study area and their population coverage were documented, as these may obviously also affect child health.

In all sites, data on levels and trends in the following areas were collected:

• Socioeconomic factors, including family income, parental education and occupation, unemployment, land tenure, and the existence of economic crises (inflation rates, structural adjustment, etc);

• Environmental factors, including water supply, sanitation, housing, and environmental pollution;

• Demographic factors, including fertility patterns and family size;

• Health-services related factors, including structure of health services, health manpower, health worker pay, drug supply, availability of referral services; and

• Presence of other projects and programs that may affect child health, including other child survival and nutrition projects being implemented by governmental and non-governmental institutions (for example, micronutrient supplementation, promotion of insecticide-treated materials, etc).

Analysis of IMCI impact takes account of these characteristics, as well as other locally relevant factors. The statistical techniques used to adjust for external factors included both simulation and multivariate analyses.

Appropriate analyses of contextual factors are a key element of the MCE and will contribute to increasing the plausibility of a possible impact of IMCI on child health and nutrition. The lessons learned in the MCE about how to measure and interpret contextual factors are summarized in the paper entitled Context matters: interpreting impact findings in child survival research, published in Health Policy and Planning in 2005.