At Last Mile Health, we believe that programs designed to reach vulnerable communities must be held to a high standard of evidence. We invest in measurement and evaluation because we want to know––not simply assume––that our work is improving people’s lives. We approached our evaluation of Liberia’s National Community Health Program in Grand Bassa County with this commitment to impact in mind. Our findings, recently published in PLOS Global Public Health, have generated valuable lessons about the design and evaluation of complex health system interventions.
The study found substantial improvements in access to appropriate care for children living in rural and remote communities
The evaluation provides strong evidence that the launch of the National Community Health Program improved access to appropriate care for sick children. Between 2018 and 2022, more children sought care from a qualified provider, while reliance on informal providers declined substantially. The study also found improvements in the quality of treatment for common childhood illnesses. These findings are consistent with the core purpose of the National Community Health Program: expanding access to quality primary health services to families who live far from health facilities via professional community health workers.
The study did not detect a reduction in under-five mortality during the evaluation period
The evaluation found that under-five mortality was similar before and during program implementation. At the same time, this does not mean we should conclude that the National Community Health Program has no effect on child survival. Rather, we can point to shortcomings in our study design that limit our ability to draw broader conclusions based on this result.
The study had important limitations in both design and implementation
This study was designed to detect a relatively large effect on mortality of 25% or more––meaning that if we observed a smaller or delayed effect, our study was not statistically powered to detect it. We were also assessing an outcome that, while devastating, is statistically rare. Because deaths among young children are uncommon, measuring a statistically significant effect on mortality would have required a larger sample size or a longer evaluation period than we planned. Moreover, unforeseen circumstances, including implementation delays and COVID-19 disruptions, compounded difficulties in isolating the program’s impact on mortality.
More fundamentally, while Liberia’s original community health service delivery package addressed important causes of childhood illness, under-five mortality is influenced by a wide range of factors that the original package was not designed to address, including those that require facility-based services. In hindsight, these complexities should have challenged our assumption that the introduction of the National Community Health Program would lead to a significant drop in child mortality in just five years.
Findings and lessons learned from this study are driving improvements to Last Mile Health’s approach to program and evaluation design
Programmatically, this evaluation showed us that even though introducing community-based primary healthcare successfully increased access to treatment for common childhood illnesses, gains in child survival may plateau without a broader package of services and stronger connections across the continuum of care. These insights are already informing the next generation of the National Community Health Program. As of 2023, routine childhood immunizations have been integrated into Liberia’s national community health policy, directly addressing a gap identified by the evaluation. The policy has also been expanded to strengthen competency-based training, supervision, nutrition screening, and referral between community- and facility-based care.
This evaluation changed the way we think about evaluating complex health programs. Our takeaway is not to avoid difficult evaluations or to move away from measuring mortality simply because it is challenging. Instead, in our ongoing and future evaluations, we will ensure from the outset that the program is sufficiently mature, implementation conditions are relatively stable, and the evaluation design has enough statistical power to credibly answer our research questions.
We also continue to interrogate how we can better map the causal pathway from intervention to impact. This includes engaging in a broader conversation, led by expert researchers in global health, to ensure we are measuring what matters. Specifically, in addition to measuring reductions in mortality, we are exploring intermediate outcomes that can contribute to a more complete understanding of implementation quality and improvements in health outcomes, while generating actionable evidence to guide policies, investments, and advocacy.
Last Mile Health will continue producing high-quality evidence on the impact of integrated, professionalized community health worker programs
In the years since we conducted this evaluation, the National Community Health Program has reached national scale, deploying more than 5,400 community health workers to serve 1.6 million people––representing 70% of the country’s rural population. Community health workers are now the top provider of care for childhood illness; with consistent financing, this professionalized workforce reliably conducts three million routine household visits and delivers 250,000 treatments for child diarrhea, malaria, and pneumonia per year. For malaria, community health workers deliver 28% of all treatments nationally, serving as a key pillar of Liberia’s fight against the leading cause of death among children under-five. As the program continues to mature, we are eager to generate evidence on how access translates to improved health outcomes.
In Liberia and beyond, with lessons from the Grand Bassa Impact Evaluation in-hand, Last Mile Health is more committed than ever to contributing to an evidence base that deepens and expands our collective knowledge on how to build high-quality community health programs that ultimately improve the lives of people living in the world’s most remote communities.
