Using electronic technology to improve clinical care – results from a before-after cluster trial to evaluate assessment and classification of sick children according to Integrated Management of Childhood Illness (IMCI) protocol in Tanzania
© Mitchell et al.; licensee BioMed Central Ltd. 2013
Received: 18 December 2012
Accepted: 13 August 2013
Published: 27 August 2013
Poor adherence to the Integrated Management of Childhood Illness (IMCI) protocol reduces the potential impact on under-five morbidity and mortality. Electronic technology could improve adherence; however there are few studies demonstrating the benefits of such technology in a resource-poor settings. This study estimates the impact of electronic technology on adherence to the IMCI protocols as compared to the current paper-based protocols in Tanzania.
In four districts in Tanzania, 18 clinics were randomly selected for inclusion. At each site, observers documented critical parts of the clinical assessment of children aged 2 months to 5 years. The first set of observations occurred during examination of children using paper-based IMCI (pIMCI) and the next set of observations occurred during examination using the electronic IMCI (eIMCI). Children were re-examined by an IMCI expert and the diagnoses were compared. A total of 1221 children (671 paper, 550 electronic) were observed.
For all ten critical IMCI items included in both systems, adherence to the protocol was greater for eIMCI than for pIMCI. The proportion assessed under pIMCI ranged from 61% to 98% compared to 92% to 100% under eIMCI (p < 0.05 for each of the ten assessment items).
Use of electronic systems improved the completeness of assessment of children with acute illness in Tanzania. With the before-after nature of the design, potential for temporal confounding is the primary limitation. However, the data collection for both phases occurred over a short period (one month) and so temporal confounding was expected to be minimal. The results suggest that the use of electronic IMCI protocols can improve the completeness and consistency of clinical assessments and future studies will examine the long-term health and health systems impact of eIMCI.
There is considerable interest in the use of mobile technology to improve health care in low and middle income countries. However, there has been a relative shortage of evidence demonstrating its impact on health outcomes [1–3]. This study explored whether mobile technology can improve the quality of care in the area of child health and in particular in the delivery of the widely used Integrated Management of Childhood Illness (IMCI) protocol for classifying and treating common causes of death including pneumonia, diarrhoea, malaria, measles, and malnutrition. A multi-country evaluation coordinated by the WHO concluded that IMCI has the potential to improve quality of care, reduce the cost of treatment, and reduce under-5 mortality when used correctly [4–7]. However, despite a worldwide effort, the use and impact of IMCI protocols remains limited due to the expense of training, the lack of sufficient supportive supervision, the tendency for health workers to follow protocols less rigorously over time, and insufficient resource and policy support [8, 9].
When facing sub-par performance, most countries and most health systems still rely on training and supervision as the basis for quality management. Unfortunately, due to the uneven quality of training and general lack of supervision, this approach is often not sufficient . The emerging use of mobile technology in health offers new strategies for improving quality of health care [3, 11, 12]. The availability of mobile phones has grown significantly in low-income countries. In 2010 over 45% of households owned at least one mobile phone in Tanzania  and a recent unpublished study by the telecom companies in Tanzania now puts that estimate at 70%. This suggests that the use of cell phones for improving care in low-income countries is both feasible and affordable.
A preliminary study suggested that some of the barriers to correct use of IMCI can indeed be alleviated with the use of mobile technology . This paper evaluates the impact of mobile technologies on the quality of IMCI implementation at multiple health facilities in Tanzania, including completeness of assessment, accuracy of classification, and consistency of care across facilities.
Tanzania adopted IMCI protocols as national policy in 1998 . The study was conducted between May 2008 and December 2009 in four districts in Pwani Region, two semi-urban (Bagamoyo and Morogoro Urban) and two rural (Mkuranga and Morogoro Rural). The study was restricted to primary care facilities where children are seen primarily by clinical officers. Throughout Tanzania, these facilities typically provide the front line of care to children under-five and provide referral for more serious cases for care at hospitals. Clinical officers have three years of post-secondary school training and are registered as medical practitioners by the Ministry of Health.
All primary care facilities with at least one health worker trained in IMCI were eligible for the study. Eighteen facilities were randomly selected from the 227 facilities that had sufficient volume of patients (minimum expected five per day) across the four districts and the one health worker trained in IMCI at the facility was invited to participate in the study. If there was more than one health worker trained in IMCI, then a single health worker was randomly selected and invited to participate.
The eIMCI protocol followed the same protocol in the current paper system guiding healthcare workers step-by-step through the child’s assessment, classification, treatment, and communication of instructions to the caregiver. During the assessment phase, results were recorded by the healthcare worker directly into the PDA. For symptoms that required additional follow-up to assess severity, the algorithm prompted the necessary questions as per the IMCI protocol. Based on the assessment, the eIMCI tool provided classification, treatment and communication recommendations in accordance to the IMCI protocol. For steps that did not require data gathering, for example the communication of an instruction to a mother, the healthcare worker selected the “next” button in the PDA to indicate that the step was completed. All eIMCI instructions were available in English and Swahili.
Implementation and data collection
At each site, data collection occurred over two phases – the first phase under the current paper based system (pIMCI) and the second phase under eIMCI. Between the two phases, the health worker was introduced to eIMCI and given instructions on how to operate the tool. No additional training on IMCI was given. Both the paper phase and the eIMCI phase of observations occurred over three to six clinic days at each site.
Any child age 2–59 months visiting the facility for their first sick child visit was eligible for inclusion. If the child’s caregiver consented to participate, an external data collector silently observed the clinic visit. Occurrence of critical pieces of the assessment and classification, treatment and recommended follow-up instructions were documented in a study form. The data collectors had university degrees, were experienced data collectors at a research organization in Tanzania and received extensive training to minimize observer and desirability bias. Following the standard clinic visit, the child then proceeded to an IMCI expert, individuals who lead national-level IMCI trainings and were available at the clinic as part of the study. The expert completed a second full assessment of the child and provided their classification, suggested treatment and follow-up instructions for the child. During the study, the classification, treatment and advice of the expert was the basis of final treatment of the child.
Primary and secondary outcomes
Completeness of assessment areas under paper-based and electronic systems for ten assessment areas programmed into eIMCI
Assessment area from the IMCI protocol
(Np = 671)
(Ne = 550)
Ability to drink
Ability to breastfeed‡‡
Feeding change during illness*
Completeness of assessment areas under paper-based and electronic systems for five assessment areas not programmed into eIMCI
Assessment area from the IMCI protocol
For secondary outcomes, we documented completed follow-up assessment questions (as appropriate) and agreement between the classification of the health worker and the IMCI expert on the four primary disease areas – pneumonia, dehydration (with diarrhea for less than 14 days), diarrhea (for more than 14 days) and malaria. For the comparison of diagnoses, the IMCI expert was used as the gold standard.
Sample size and power
Based on the preliminary study, we anticipated 65% adherence to the assessment sections under pIMCI and 80% adherence under eIMCI. Assuming no clustering, 185 individuals were needed to achieve 90% power to detect a difference at a 0.05 significance level. During the design phase, we anticipated observing 30 patients per arm per clinic at 20 clinics, corresponding to an intraclass correlation (ICC) of 0.077. Because of logistical constraints, we only included 18 facilities in this study. However, power to detect this difference remained high at 87%.
Basic demographics and disease profiles of the children seen under each system were compared using frequencies. The impact of the system on the completeness of assessment was assessed using generalized estimating equations with an exchangeable correlation matrix to account for the clustering by clinics. Because of the small number of clinics, all results were validated using the Wilcoxon signed-rank test. The Wilcoxon signed-rank test was also used to compare changes in the median number of follow-up questions (by symptom area) for each clinic. For each assessment area, the ICC was calculated by system. The percent agreement of classification between the clinical health worker and the IMCI expert was estimated with binomial confidence intervals for each disease area and tested for significant difference in percent agreement by system using generalized estimating equations. All analyses were completed in Stata/MP 11.0.
Consent for clinic participation was obtained from both the district health officer and the healthcare worker at the clinic. Consent was also obtained from the caregiver of each child observed. The study was approved by the institutional review boards for Harvard School of Public Health, Ifakara Health Institute and the National Institute of Medical Research in Tanzania.
Characteristics of children observed under the two systems
N = 671
N = 550
less than 6 months
6 months - 1 year
1 year - 2 years
2 years - 5 years
Median temperature, Celcius (IQR)
Presentation of illnesses according to expert classification
Any severe disease
Competing illnesses, pneumonia plus
Competing illnesses, malaria plus
Impact of electronic system on adherence to IMCI protocols
For the 15 critical assessment areas, ten were included on both eIMCI and pIMCI protocols (Table 1). For all ten areas, questions were more likely to be asked by the healthcare worker when using eIMCI compared to pIMCI (p < 0.05). Only 20.7% of children had all ten items assessed under the paper system compared to 70.9% under the electronic system (p < 0.001). For the five areas not included in the electronic protocols, the completeness of assessment was nearly identical under the paper and electronic systems (Table 2). For the full fifteen items, the proportion of complete assessments were low for both systems – 4.6% for pIMCI and 8.4% for eIMCI (p = 0.212).
Three symptom areas, namely cough/difficulty breathing, diarrhea, and fever, required additional follow-up questions to determine the severity of the final classification. For all sites and all symptoms, at least as many and often more questions were asked under the electronic system as compared to the paper system (p < 0.01 for all three assessment areas).
Impact of electronic system on classification of diseases
Accuracy of classification compared to IMCI expert
Percent of correct classification for all levels of disease (none, some, severe)
% (95% CI)
% (95% CI)
Percent of correct classification for severe disease
Systems impact of electronic system
Intraclass correlation, listed by specific assessment areas
Control (Paper-based system)
Included in the electronic protocol
Ability to drink
Ability to breastfeed
Ability to eat foods
Not included in the electronic protocol
The average time per visit under the current paper-based system was 8.98 minutes (standard error, 0.20). For eIMCI, the average visit length was 9.06 minutes (standard error, 0.20). The length of visit was not significantly different between the two systems (p = 0.996). Trainers noted the time required to train health workers in use of the electronic protocols, and consistent with previous studies, the time required for training was 1–2 hours (not including basic IMCI training that all health workers had undergone previously through the Ministry of Health) .
The results of this study provide strong evidence supporting the potential of mobile health to improve quality of health care in low-income countries, including thoroughness of assessment and accuracy of classification. For diagnoses, the agreement between the health worker and IMCI expert was higher under eIMCI compared to pIMCI, but was only significantly different for one of the four classification areas. An interesting comparison between pIMCI and eIMCI is the impact on classification of severe disease, which is distinguished from non-severe disease through the completion of the follow-up assessment questions and is critical for identification because these are the leading causes of preventable death in children . Showing significance with this data is impossible because of the few cases of severe disease. Therefore the true impact of mobile technology on the classification of severe disease and impact on health outcomes of children must be evaluated in future studies.
Another improvement observed under eIMCI was better consistency in care. The ICC values under pIMCI were very high, suggesting that the completeness of assessment very much depends on the individual conducting the assessment. In comparison, the assessment areas included in eIMCI had much lower values of ICC. These results, combined with the proportion of visits with complete assessments, imply that assessments are not only better but are also more consistent across clinics with the support of mobile technology.
The mechanisms for the improvements observed in this study are supported by results from a qualitative study conducted at these sites . Health workers indicated a stronger preference for the PDA compared to the paper charts used to support pIMCI implementation, both because the tool was easier to use and because of the perceived improvement in confidence from the children and caregivers. Indeed, the caregivers also expressed a preference because they believed the health workers were better equipped with the PDA and because they believed they were more engaged when eIMCI was used.
There are several limitations to this study. First, the number of clinics included in the study were fewer than originally planned and the ICC values for pIMCI were larger than that used in the design phase. Fortunately, the effect of eIMCI was greater than anticipated and so we were able to achieve statistical significance. These estimated ICC values will be used to improve study designs in the future. Another limitation in the design was the before-after nature of the assessments as such designs have the potential for temporal confounding. Given the logistical challenges and early stages of evaluation for eIMCI, a cluster randomized trial was not feasible for this study. However, in this case, we believe the introduction of temporal confounding is minimal since both phases of data collection at a site occurred within three weeks. A third limitation is the possible Hawthorne effect, since there was an observer present in the room during the IMCI assessment. While such effect is inevitable when using observers, we feel this effect would only dilute the estimated impact of eIMCI. A further limitation is that the study focused exclusively on assessment and classification elements. Correct treatment and counseling are critical components of the IMCI encounter, and to fully understand the usefulness of eIMCI, these elements must be included in future studies. Finally, an additional limitation is that five of the 15 critical assessment areas were not included in the eIMCI protocol. The elements omitted were not critical for the classification in the four main disease areas, but have been noted as important and will be included in future iterations. Further, the omission of these items provided an important insight. For items programmed into the system, the observers documented improvement, for the others no improvement was observed. This suggests that any desirability bias was minimal.
Although this study looked narrowly at adherence to IMCI protocols and subsequent classification, there are many implications of its results. Perhaps most significant is the role that computerized protocols can have on effective task shifting in low-income countries. Given the severe shortages of trained health personnel in low-income countries , there is an urgent need to find ways to engage lesser-trained health personnel to accurately diagnose and treat patients while maintaining high quality of care. While electronic technology cannot replace the role of training and supervision, such tools can complement these existing mechanisms to ensure consistency and quality. A second important implication of this study is the ability to build other components onto an electronic decision support system including a medical record system. The eIMCI system that is currently being developed facilitates the transfer of data about each patient and each encounter to a centrally stored patient record. This record would enable the health worker to track a child’s weight, immunization status or other health problems over time. This improves the ability of the IMCI protocols to accurately assess a child’s condition, and there is evidence from other applications that an effective electronic medical record has the ability to improve care . Further, this data can also be made available for disease surveillance, remote supervision and routine reporting improving information available to health planners and managers on a timely basis. A third benefit of the approach used in this study is the ability to quickly change protocols as policies change . This has been a problem faced by IMCI since a change in protocols for the treatment of fever has required a complete retraining and reprinting of all booklets to include the newer drugs and incorporate the use of rapid diagnostic tests. We are currently engaged in studies to better understand the impact of mobile technology, and specifically eIMCI, on these areas of task-shifting, contributing to a functional electronic medical record, and improving the ability to update protocols in a timely manner. Further, we are presently studying the sustained impact of eIMCI on adherence and classification and the impact of the protocol on health outcomes after 12 months of implementation.
The emerging field of mobile and electronic health is showing great promise in improving both the delivery and monitoring of health care in low-income countries. This study demonstrates its benefit to deliver a proven standard of care: IMCI in a rural and semi-urban area of Tanzania. While this study showed improved adherence to a specific set of clinical guidelines (IMCI) it seems likely that this same result could be achieved with the use of electronic guidelines in other areas such as maternal health, chronic illness and infectious disease where national guidelines are often available but used sporadically. Based on this finding, the authors anticipate a much broader use of mobile technology for the delivery of clinical standards aimed at improving clinical care in low-income countries, including studies investigating the benefits with other clinical protocols and with other cadres of healthcare workers.
This study was funded by a grant from The Rockefeller Foundation. Additional funding was provided by D-tree International. We thank Joanna Schellenberg, Karen Emmons, Hamish Fraser, and Brian DeRenzi who offered valuable comments on this manuscript and Chris Toretsky who supported data cleaning.
- Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, et al: The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med. 2011, 8 (1): e1000387-10.1371/journal.pmed.1000387. PMID: 21267058View ArticlePubMedPubMed CentralGoogle Scholar
- World Health Organization (WHO): mHealth: new horizons for health through mobile technologies: second global survey on eHealth. 2011,http://www.who.int/goe/publications/goe_mhealth_web.pdf,Google Scholar
- Zurovac D, Talisuna AO, Snow RW: Mobile Phone Text Messaging: Tool for Malaria Control in Africa. PLoS Med. 2012, 9 (2): e1001176-10.1371/journal.pmed.1001176.View ArticlePubMedPubMed CentralGoogle Scholar
- Gouws E, Bryce J, Habicht JP, Amaral J, Pariyo G, Schellenberg JA, et al: Improving antimicrobial use among health workers in first-level facilities: results from the multi-country Evaluation of the Integrated Management of Childhood Illness strategy. Bull World Health Organ. 2004, 82 (7): 509-515. PMID: 15508195PubMedPubMed CentralGoogle Scholar
- Bryce J, Victora CG, Habicht JP, Vaughan JP, Black RE: The multi-country evaluation of the integrated management of childhood illness strategy: lessons for the evaluation of public health interventions. Am J Public Health. 2004, 94 (3): 406-415. 10.2105/AJPH.94.3.406. PMID: 14998804View ArticlePubMedPubMed CentralGoogle Scholar
- Bryce J, Gouws E, Adam T, Black RE, Schellenberg JA, Manzi F, et al: Improving quality and efficiency of facility-based child health care through Integrated Management of Childhood Illness in Tanzania. Health Policy Plan. 2005, 20 (Suppl 1): i69-i76. 10.1093/heapol/czi053. PMID: 16306072View ArticlePubMedGoogle Scholar
- Tanzania IMCI Multi-Country Evaluation Health Facility Survey Study Group: The effect of Integrated Management of Childhood Illness on observed quality of care of under-fives in rural Tanzania. Health Policy Plan. 2004, 19 (1): 1-10.View ArticleGoogle Scholar
- Rowe AK, Onikpo F, Lama M, Osterholt DM, Rowe SY, Deming MS: A multifaceted intervention to improve health worker adherence to integrated management of childhood illness guidelines in Benin. Am J Public Health. 2009, 99 (5): 837-846. 10.2105/AJPH.2008.134411. PMID: 19299681View ArticlePubMedPubMed CentralGoogle Scholar
- Ahmed HM, Mitchell M, Hedt B: National implementation of Integrated Management of Childhood Illness (IMCI): policy constraints and strategies. Health Policy. 2010, 6 (2): 128-133. 10.1016/j.healthpol.2010.01.013. PMID: 17223476View ArticleGoogle Scholar
- Walter ND, Lyimo T, Skarbinski J, Metta E, Kahigwa E, Flannery B, et al: Why first-level health workers fail to follow guidelines for managing severe disease in children in the Coast Region, the United Republic of Tanzania. Bull World Health Organ. 2009, 87 (2): 99-107. 10.2471/BLT.08.050740. PMID: 19274361View ArticlePubMedGoogle Scholar
- Kuepper BM, Stekelenburg J, Milen A: Improvement of maternal health services through the use of mobile phones. Trop Med Int Health. 2011, 16: 622-626. 10.1111/j.1365-3156.2011.02747.x.View ArticlePubMedGoogle Scholar
- Lester RT, Ritvo P, Mills EJ, Kariri A, Karanja S, Chung MH, et al: Effects of a mobile phone short message service on antiretroviral treatment adherence in Kenya (WelTel Kenya1): a randomised trial. Lancet. 2010, 376 (9755): 1838-1845. 10.1016/S0140-6736(10)61997-6.View ArticlePubMedGoogle Scholar
- National Bureau of Statistics (NBS) [Tanzania] and ICF Macro: Tanzania Demographic and Health Survey 2010. 2011, Dar es Salaam: NBS and ICF MacroGoogle Scholar
- DeRenzi B, Lesh N, Parikh TS, Sims C, Mitchell M, Maokola W, et al: e- IMCI: Improving Pediatric Health Care in Low-income Countries. 2008, Florence, Italy: Presented at the ACM Conference on Computer-Human Interaction (CHI), 5-10.http://homes.cs.washington.edu/~bderenzi/Papers/chi1104-bderenzi.pdf,Google Scholar
- Prosper H, Macha J, Borghi J: Implementation of Integrated Management of Childhood Illness in Tanzania: successes and challenges. 2009, Dar es Salaam: Consortium for Research on Equitable Health Systems and Ifakara Health Institute, Online at: http://www.crehs.lshtm.ac.uk/downloads/publications/Implementation_of_IMCI_in_Tanzania.pdf Google Scholar
- WHO and UNICEF Integrated Management of Childhood Illness Booklet. (Accessed 25 March 2013) http://whqlibdoc.who.int/publications/2008/9789241597289_eng.pdf
- Black RE, Morris SS, Bryce J: Where and why are 10 million children dying every year?. Lancet. 2003, 361 (9376): 2226-2234. 10.1016/S0140-6736(03)13779-8. PMID: 12842379View ArticlePubMedGoogle Scholar
- Mitchell M, Getchell M, Nkaka M, Msellemu D, van Esch J, Hedt B: Perceived improvement in Integrated Management of Childhood Illness (IMCI) implementation through use of mobile technology: qualitative evidence from a pilot study in Tanzania. J Health Commun. 2012, 17 (Supplement 1): 118-127.View ArticlePubMedGoogle Scholar
- World Health Organization (WHO): Working together for health: the world health report 2006. 2006, Geneva: World Health Organization, Online at: http://www.who.int/whr/2006/whr06_en.pdf Google Scholar
- Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma’Luf N, et al: The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999, 6 (4): 313-321. 10.1136/jamia.1999.00660313. PMID: 10428004View ArticlePubMedPubMed CentralGoogle Scholar
- Mitchell M, Lesh N, Cranmer H, Fraser H, Haivas I, et al: Improving care – improving access: the use of electronic decision support with AIDS patients in South Africa. Int J Healthc Technol Manage. 2009, 10 (3): 156-168. 10.1504/IJHTM.2009.025819.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/13/95/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.