Smartphone-based mobile applications for adverse drug reactions reporting: global status and country experience

Abstract

Background

Smartphone technology can support paperless reporting of adverse drug reactions (ADRs). The aims of this study were to systematically assess smartphone ADR-reporting applications, understand their qualitative and quantitative impact on ADR reporting, and garner key lessons from owners and developers.

Methods

This study had three components: (1) An assessment of ADR-reporting apps, (2) an online survey on the impact of app implementation on ADR reporting and the experiences of app developers and owners, and (3) a search of VigiBase, the World Health Organization global database of individual case safety reports (ICSRs), to observe trends in the number of ADR reports targeting countries where the apps were implemented.

Results

Twenty-two apps were included. Eight out of the 22 apps were for countries in the WHO African region. Features observed included E2B data elements (E stands for efficacy) and functions supporting reporting and user engagement. Seventeen app developers and owners answered to the survey and reported overall positive experiences with app features, and post-launch increases in the total number of ICSRs. User type and user environment were cited as factors influencing app use: Respondents said younger people and/or those with an inclination to use technology were more likely to use apps compared to older or more technology-averse people, while respondents in countries with limited internet connectivity reported persistent difficulties in app use.

Conclusions

Smartphone apps for reporting ADRs offer added value compared to conventional reporting tools. Reporting tools should be selected based on interface features and factors that may influence app usage.

Background

Spontaneous reporting of adverse drug reactions (ADRs), defined here as unsolicited communication by a reporter to a competent authority, such as a national regulatory authority (NRA), has been the conventional approach to pharmacovigilance notification since the early 1960s [1, 2]. Many studies have been conducted to identify potential factors affecting reporting behavior among different types of reporters [3,4,5], and various approaches have been undertaken to promote pharmacovigilance and to encourage reporting [6,7,8].

An average of 2 million individual case safety reports (ICSRs) are recently added each year to the World Health Organization’s global database of ICSRs, VigiBase, from more than 100 countries. By the end of 2020, the total number of ICSRs had exceeded 24 million [9] since VigiBase’s establishment in 1968. The E2B format (E stands for efficacy) defined by the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) has helped standardize the data elements, structure and formats of ICSRs regardless of source or destination [10], ultimately contributing to the overall improvement in the quality of reports in the global database.

However, despite this fundamental improvement, spontaneous reporting has been significantly limited by under-reporting, poorly documented reporting and reporting delays. These challenges were due to a lack of knowledge and awareness of pharmacovigilance and a lack of motivation, as well as indifference, insecurity, complacency, workload, and lack of training [11, 12]. The lack of prompt, well-documented reports hampers NRAs from making timely public health decisions for the safer use of medicinal products [13].

Paper-based reporting has long been the primary method for collecting ADRs [14]. However, to improve ADR reporting, paperless approaches have been explored, including via text message [15], telephone [16], websites (e-reporting) [17] and social media [18]. Furthermore, we are seeing the use of smartphone-based tools [19, 20]. Such tools also exist in a variety of other healthcare areas, such as disease management, drug dictionaries and drug dosage calculators [21, 22].

Mobile phone applications (apps) designed for ADR reporting have been attracting the attention of NRAs, health care professionals (HCPs) and patients, not only from high-income countries (HICs) but also from low- and middle-income countries (LMICs) [23,24,25,26,27]. Since 2017, The World Health Organization (WHO) has been supporting LMICs to roll out the ADR reporting app, Med Safety, in collaboration with technical partners, including the United Kingdom Medicines and Healthcare products Regulatory Agency (MHRA) and the Uppsala Monitoring Centre (UMC). The focus has been on improving the quality and quantity of ADR reports within countries in real time, reducing the burden of reporting and managing those reports, and improving user engagement by providing post-reporting feedback and drug safety information.

While there is a strong interest among NRAs, HCPs and patients in transitioning to app-based ADR reporting, little was known about existing ADR reporting apps, their characteristics and performance. This study aimed at systematically assessing and characterizing accessible smartphone ADR reporting apps, to understand the impact of their use on the quality and quantity of ADR reporting, and to collate lessons learned from the experiences of developers and owners.

Methods

This study consisted of three components: (1) A systematic assessment of ADR reporting apps identified through an extensive search of smartphone app stores and the internet, (2) an online survey targeting developers and owners on the impact of app implementation on ADR reporting and their experiences of the identified apps and (3) a quantitative search of VigiBase to observe trends in the number of ADR reports in countries where the identified apps were implemented. We defined “developers” as entities that technically develop mobile apps, “owners” as entities that oversee the apps and for whom the apps are developed (e.g., NRAs), and “users” as individuals who use the apps to report ADRs or access drug safety information.

Systematic assessment of ADR reporting apps available worldwide

The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) checklist was consulted to draw up an evidence-based minimum set of items required for reporting in systematic reviews [28]. This checklist provides for transparent reporting of a systematic review of scholarly references. Since this study targeted smartphone apps instead of scholarly references, we conducted it using deviations from the original checklist of items to include when reporting a systematic review when it came to bias risk-related items in individual studies and across studies (items 12, 15, 19 and 22) [28].

Eligibility

Inclusion criteria: An “app” was defined as a software application designed to run on a smartphone, and was included in the study if it permitted users to directly create an ADR report within the app and submit it to a given addressee. Apps in any language were eligible.

Exclusion criteria: Apps without any automated ADR report submission or provision of appropriate addressees were excluded. Apps that only focused on reporting events on items unrelated to medicines, such as cosmetic products, medical devices and vaccines, were also excluded. App price was not included in the selection criteria.

Information sources and search strategy

A search of the App Store and the Google Play Store was conducted using 17 search terms to identify potentially eligible ADR reporting apps. The search terms were reviewed and validated to ensure their relevance and exhaustivity. The search terms are listed in “Appendix 1”.

The App Store is a digital distribution service developed by Apple (Apple Inc., United States of America) and serves as the official app store of devices using the iOS mobile operating system. The Google Play Store is a service developed by Google (Google, LLC, United States of America) and serves devices using the Android system. Other operating systems were not considered in this study given that, as on July 2020, more than 99% of the global market share of mobile operating systems was covered by either Android or iOS [29] and the search on both the App Store and the Google Play was assumed to be comprehensive enough to generalize to the global setting.

In general, search results retrieved by an internet search engine are personalized based on search history and user location. Logging out of an account or searching through an incognito window were ensured to get neutral search results that were not influenced by personalized information [30].

The search using the defined search terms in both smartphone app stores was performed on August 19, 2020. Search results were exported and saved in two Excel spreadsheets. The following information was collected in the spreadsheets: name of app, name of developer or owner, specific URL in each mobile app store and content type. Deduplication of apps was performed using the URL information.

In addition, an internet search for apps satisfying the selection criteria was conducted using an incognito window in Chrome and Firefox browsers. The same search terms as used in the app stores were applied. This desk research was performed in November 2020.

Each search was performed in Geneva, Switzerland.

Selection process

A two-step process was adopted to select the apps for inclusion in the study. The first consisted of reviewing the name of the app and its descriptions available in the app stores to determine whether it met the selection criteria. The second consisted of downloading and installing the apps on either an Android phone (Xiaomi Mi9T) or Apple iPhone (iPhone 8) depending on the source. A full review was then performed of the entire app. The PRISMA flow diagram was created using open-source software draw.io to illustrate the selection process [31].

Data extraction process

The following information was collected and extracted using Excel software:

• General information on each app, such its name, developer or owner name, available platform, app version tested, size, cost per installation and use, and the latest update.

• Geographic scope of where each app could submit reports. To categorize the countries, we used the six WHO regions, namely African Region, Region of the Americas, South-East Asia Region, European Region, Eastern Mediterranean Region, and Western Pacific Region, and the definition of the World Bank income groups, namely low, lower-middle, upper-middle, and high [32].

• The features of each app (e.g., offline report development, supplementary materials attachment and field for laboratory test results, as well as two-way communication-related features).

• The E2B data elements available in each app. The most frequently available data elements were identified during a pilot assessment of multiple apps. The identified data elements were explored during the data extraction process and stored under a category of the “minimum information for valid safety report” as defined by the ICH: Identifiable patient, identifiable reporter, adverse event/reaction, and suspect or interacting drug [10]).

Apps available only in languages other than English were translated into English during data extraction.

Online survey

The online survey was designed to interview developers and owners of the selected apps to obtain information on the impact of app implementation on the quality and quantity of ADR reports, and to understand their experience of the apps. The online survey was created using Google Forms and an invitation was emailed to the developers and owners of the selected apps. The online survey targeted pharmacovigilance officers in the National Regulatory Agencies, academia and pharmaceutical industry to ensure reliable information and subject knowledge when responding to the survey queries. Up to two reminders were sent to those who did not respond. The survey consisted of 9 open-ended and 10 closed questions (“Appendix 2”).

The apps’ quantitative impact was assessed using responses to questions on total number of downloads, trends in the overall number of ADR reports after the app launch, total number of ADR reports received from the app, and proportion of ADR reports received from the app among all reports since the launch. The total numbers of ADR reports received from the app were also averaged by year based on the number of years from app launch to the survey. The collected responses on the number of downloads of different versions of the Med Safety app were confirmed by MHRA, which had access to such information.

The apps’ qualitative impact was assessed using responses to questions on the availability of the minimum four information items for a valid safety report as defined by the ICH: Identifiable patient, identifiable reporter, adverse event/reaction, and suspect or interacting drug [10]. The respondents were invited to select those of the four that were judged to have been appropriately filled in by the reporters of the ADR reports received through paper forms and apps.

The lessons learned from the respondents’ experiences of the app were sought through open-ended questions. The feedback was reviewed and synthesized into the following themes: simplicity of use, report quality, accessibility, innovativeness, data transferability and data sharing, two-way communication, cost, and data security.

Additional information—such as the app’s launch date, the purpose of its implementation, and its development and maintenance costs—was also obtained.

Quantitative search of number of reports submitted to VigiBase before and after app launch

VigiBase was searched to observe trends in the number of ADR reports submitted before and after an app’s launch. VigiLyze, the analytical platform of VigiBase, was used for this analysis. VigiLyze did not allow searches by type of reporting tool. Thus, only the overall number of ADR reports, regardless of type of reporting tool, was extracted according to the following search conditions:

• Geographical scope: Countries where the selected apps were implemented.

• Timeframe:

  • The pre-launch period was defined as the 12 months preceding the app’s launch.

  • Post-launch Period A was defined as the period from the launch to Month 12 after the launch.

  • Post-launch Period B was defined as the period from Month 13 to Month 24 after the launch.

Although the numbers of ADR reports were not analyzed by type of reporting tool, this analysis was considered to illustrate the quantitative impact of app implementation in the countries.

The VigiBase search was performed on February 15, 2021. Only apps that had already been launched for at least one year (12 months) by that date were included in the analysis, as well as those that had been launched for at least two years (24 months) so that, at a minimum, data for Period A was available, and in some cases also data for Period B. Since the launch date was required to set the timeframes to be analyzed, an app was only included if the online survey response supplied its launch date. Relative percentage changes were calculated to compare the numbers of ADR reports received in post-launch Periods A and B with the number in the pre-launch period for each app.

Results

Search results and app inclusion

A flowchart of the app selection is shown in Fig. 1. The searches in the Google Play Store and App Store resulted in the retrieval of 4126 and 1359 apps, respectively. The number of hits varied depending on the search terms, from 181 hits for ‘ADR reporting’ to 250 hits for each of nine other terms in the Google Play Store, and from four hits for ‘ADR reporting’ to 209 hits for ‘drug safety report’ in the App Store. After removing 2681 duplicates, 2144 and 660 apps, from each store respectively, were included in the first screening stage during which the names and descriptions of apps were reviewed. Following the first screening stage, 2110 apps retrieved from the Google Play Store and 649 apps from the App Store were excluded as they did not meet the selection criteria. This resulted in the retrieval of 34 apps from the Google Play Store and 11 apps from the App Store for the second screening stage (full screening). Thirty apps were excluded in the full screening for the following reasons: “Duplicate” (n = 3), “Dysfunctional ADR reporting system” (n = 3), “No ADR reporting system” (n = 6), “Report submission not automated” (n = 13), “Unable to log into app” (n = 4) and “Unidentifiable app”¹ (n = 1). No additional apps were identified during the desk research using the Chrome and Firefox browsers.

Fig. 1

PRISMA flow chart of the app selection process

One of the identified reporting apps, Med Safety, had several versions, each adapted to the ADR reporting system of each of eight countries. To assess the versions separately and to facilitate the data extraction process, each of the eight was considered an individual app. Consequently, the final number of apps that met the inclusion criteria was 22.

Basic and geographic specifications

The basic specifications of the 22 selected apps are listed in Table 1. No app required payment by users for download and use. The selected apps were developed by 12 developers, and the MHRA was cited as a developer for 11 apps, such as HALMED, Med Safety, UAE RADR and Yellow Card Scheme. Six of the 22 apps were available only on Android, one only on iOS, and 15 on both platforms. The last update occurred in the study year (i.e. 2020) for 14 apps, one year before the study year (2019) for three apps, two years before (2018) for two apps, three years before (2017) for two apps, and four years before (2016) for one app.

Table 1 Technological specifications of the selected apps

The geographic specifications of the 22 selected apps are listed in Table 2. The highest number of apps were mapped for countries in the WHO African region (8/22), seven of which were Med Safety. These eight apps enabled users to report ADRs to the NRAs in Botswana, Burkina Faso, Côte d’Ivoire, Ethiopia, Ghana, Tanzania, Uganda and Zambia. Six apps were mapped for countries in the WHO European region, four in the WHO South-East Asia region, two in the WHO Eastern Mediterranean Region and one each in the WHO Region of the Americas and the WHO Western Pacific Region. In terms of the languages, English was used in most of the apps (20/22), while languages other than English, such as French, were also included in nine apps. Two apps from Argentina and the Russian Federation used only their own national language, respectively. When we classified the apps by country income level, as defined by the World Bank [33], more than 70% of the selected apps were in LMICs.

Table 2 Geographic specifications of the selected apps

E2B data elements

The E2B data elements that were available in the selected 22 apps are listed in Table 3. With the exception of one app (Salamtok, developed for reporting in Yemen), where only limited free-text fields were available, all apps consisted of E2B data elements, ensuring that at least the minimum required information could be collected. Patient name or initials were the most commonly adopted fields (21/22) for patient identification, followed by fields related to age and gender (20/22). To identify the reporter, a field for the reporter’s name was included in most of the apps (20/22). A field to describe the adverse reaction or select the reaction term from a defined list was available in 21 of the apps, of which three did not have further fields, such as date of reaction onset, outcome of reaction and seriousness criteria of reaction. A field to describe or select drug names from a defined list was available in 21 of the apps, of which two did not have further fields to add further details about the suspect drug, such as dosage, indication and route of administration.

Table 3 E2B data elements available in the selected apps

Additional apps features

The available features that each selected app offered are shown in Fig. 2. The features that contributed to ADR reporting included providing support to create a report, report management, and two-way communication, such as not only enabling users to report but also providing drug safety information. Most of the apps enhanced reporting and two-way communication through a feature for offline report development (17/22), provision of contact addresses for inquiries (18/22) and provision of drug safety information (15/22). Apps rarely had an automated feature to allow a reporter to share the report with their physician or other addressees of their choice (1/22).

Fig. 2

Number of apps offering each feature. Feature item No. 1 was present in the following apps: 1, 3, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 18, 19, 20, 21, 22. Feature item No. 2 was present in the following apps: 1, 2, 17, 18, 19, 20. Feature item No. 3 was present in the following apps: 2, 4, 19, 20 (While not having a laboratory field by default, there were apps which automatically displayed supplementary fields and questions in response to an answer to certain conditions such as pregnancy). Feature item No. 4 was present in the following apps: 2, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 21, 22. Feature item No. 5 was present in the following apps: 1, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 21, 22. Feature item No. 6 was present in the following apps: 17. Feature item No. 7 was present in the following apps: 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 21, 22. Feature item No. 8 was present in the following apps: 1, 3, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22 (Apps were also counted if contact was included in an acknowledgment message or in an introduced website)

Online Survey

The developers and owners of the selected 22 apps were invited to share their experiences of the apps through the online survey. Answers for 17 of the 22 apps (77.2%) from 19 interviewees were collected in November and December 2020. The answers were provided from No responses were received from the following five apps: ADR PvPI, Easypharm, Medicinal Vigilance, Salamtok and TMDA Adverse Reactions Reporting Tool.

All of the apps were launched after 2014 except ADR Online (developed for reporting in New Zealand), which was launched in 2010. The respondents confirmed that all the apps were developed to offer an additional tool for ADR reporting besides conventional methods. It was stated that the apps’ implementation was also intended to provide users with drug safety information and to raise awareness of ADR reporting through the engagement of a larger reporting population. The most frequent user types who reported ADRs through the apps were HCPs, such as pharmacists (5/17), medical doctors (4/17) and other HCPs (4/17). Following HCPs, patients or family members were reported as the next most frequent reporter type (3/17). Public health programs were not cited as frequent users in any of the responses.

Quantitative and qualitative impact of app implementation on ADR reporting

The quantitative data of app downloads and ADR reports are summarized in Table 4. The number of downloads tended to be higher for Android users compared to iOS, where both Android and iOS platforms were available. The apps which enabled users to report ADRs to public entities such as NRAs tended to have the higher number of downloads compared to the apps addressing ADRs only to private entities. The average percentage of ADR reports received from the apps compared to reports using all reporting methods varied from 0 to 60%. This range was broader for LMICs (0% to 60%) than for HICs (0% to 5%). A percentage greater than or equal to 5% was reported for eight apps: seven from LMICs and one from HICs.

Table 4 Quantitative impact of app implementation for the apps cited in the online survey

More than half of the respondents said that they had observed a post-launch upward trend in the overall number of ADR reports, not only for those received through the apps but also from other reporting tools, such as paper-based reporting (9/17).

Figure 3 shows qualitative differences in “essential elements” appropriately filled in for ADR reports submitted from apps and in paper form. 65% of respondents (11/17) said that the number of elements appropriately filled-in on reports submitted via app was the same as that for paper reports. Six respondents said that all of the four minimum elements were appropriately filled-in on reports submitted using the apps and paper forms.

Fig. 3

Comparison of essential elements* appropriately filled in for ADR reports on apps and in paper form. *Elements making up the “minimum information for valid safety report” as defined by the ICH

Real-world experiences with ADR reporting apps

The respondents of the online survey shared the pros and cons of their ADR reporting apps (Table 5). The inputs from the respondents were synthesized into the following themes:

Table 5 Respondents’ comments on use of ADR reporting apps

Simplicity of use and report quality

Respondents said that the apps simplified ADR reporting, making it quick and easy, and reducing delays in reporting to a data addressee such as a competent authority. The simplicity was enhanced by digital features, such as drop-down menus, a defined drug list, and data file attachment capability. Defining mandatory reporting fields reduced issues of missing data and increased the overall quality of reports. However, the limited information collected by such simple reporting apps needed to be complemented by more comprehensive reporting afterward.

Innovativeness and accessibility of reporting

Feedback indicated that apps make ADR reporting accessible to a broader population since anyone with a smartphone can download the app and report ADRs. Although a feature enabled the use of some functions without an internet connection, some internet connectivity was needed later to upload the offline activity so it could be transmitted. Respondents in countries with limited internet connectivity reported persistent difficulties in using the apps. Respondents said the younger generation and/or users with an inclination to use technology were more likely to use apps compared to more technology-averse or older people.

Data transferability and data sharing

NRAs valued the direct receipt of ADR reports from apps to their databases since it did not require the manual entry of data, thus saving time and potentially avoiding transcribing errors. Moreover, apps structured in the E2B format were convenient for data processing. A limitation was the inability of the app to save a copy of a report for the reporter to refer back to once it had been sent and to thus be able to share the report with other relevant stakeholders.

Two-way communication

Two-way communication enabled owners and end users to communicate drug safety information and other related news in a timely manner, and strengthened the engagement of end users with an interest in drug safety. When end users received an immediate acknowledgment of their report’s submission, it bolstered their commitment by making them feel they were making a valuable contribution to pharmacovigilance activities.

Costs

Respondents expressed only positive comments on cost since the apps were free of charge for reporters and reduced the cost of distributing paper reporting forms.

Quantitative search on number of reports submitted to VigiBase before and after app launch

Of the 17 apps considered in this analysis, the full observational period of 12 months in post-launch Period A was available in 16 apps, and post-launch Period B in 10 apps. The apps that did not have a post-launch Period B were DGDA Drug Verification, Med Safety (Armenia), Med Safety (Botswana), Med Safety (Côte d’Ivoire), Med Safety (Ethiopia), Med Safety (Ghana) and Med Safety (Uganda), while Med Safety (Uganda) was not included in either analysis.

An upward trend in the number of ADR reports was observed in 81.2% of the apps (13/16) in post-launch Period A compared to the pre-launch period. The remaining three apps—ADR Reporter, Med Safety (Burkina Faso) and SiddAR—showed a negative trend. The relative change in post-launch Period A varied from 11.9% to 596.7% in the apps showing an upward trend. Only four apps showed a continuous upward trend over post-launch Periods A and B compared to the pre-launch period. Med Safety (Burkina Faso) was the only app to show an upward trend in post-launch Period B (+ 71.4%) following a downward trend in post-launch Period A (− 42.4%). Further details are shown in Table 6.

Table 6 Quantitative trend of ICSRs received in VigiBase over different pre- and post-launch periods for the apps cited in the online survey

Discussion

To our knowledge, this is the first study mapping ADR reporting apps. Of the 22 apps selected for the study, more than 70% were based in LMICs, with the majority in the WHO Africa region.

Although the number of LMICs with pharmacovigilance centers reporting to VigiBase has increased substantially in the last 10 to 15 years, actual reporting per 100,000 population per year is relatively low in LMICs compared to many HICs, which have well-established and more mature pharmacovigilance systems [34,35,36]. Paper-based reporting requires reports to be manually transported by postal services to a pharmacovigilance center once filled out by a reporter, and to be manually entered into a data management system, which is one of the most time-consuming activities in the reporting process [37]. It is likely to result in quantitative and temporal challenges, i.e. delay from the time of the onset of an ADR to the time the report reaches VigiBase [38]. Moreover, LMICs face qualitative challenges due to poorly documented reports [39]. Needless to say, the comprehensiveness of a report plays a critical role in pharmacovigilance in order to aid in the detection of signals [40].

This study showed that ADR reporting apps offer unique added value compared to conventional reporting tools, such as paper-based formats, through various features, and they were likely to support countries to more efficiently collect data. One of the identified features was a standardized format meeting E2B standards [41]. Hence data collected by the apps were easily transferable to the E2B-compatible national data management system and, moreover, the flow of data to VigiBase would be facilitated, thereby addressing the challenges that LMICs face. In fact, according to the online survey results, LMICs were more likely than HICs to take advantage of smartphone-based apps features for reporting. The survey respondents also appreciated that the apps supported data transmission and improved report quality in LMICs. However, not all respondents were able to provide information to thoroughly describe the direct impact of the apps. Further investigation would be warranted on outcomes, such as the time from event onset to data submission on VigiBase, and the report quality appraised through various measurements [39, 40, 42] between those received from ADR reporting apps and those via conventional routes.

Notably, in this study, more than 50% of the survey respondents commented that apps contributed to an increase in the number of all ADR reports regardless of the type of reporting tool. The upward quantitative trend of all ICSRs in the post-launch periods was also confirmed by the VigiBase search. As mentioned in the introduction, several countries have been supported by the WHO to roll out the ADR reporting app, Med Safety. The first country to pilot the app was Burkina Faso in 2017 [43, 44], initially as part of the seasonal malaria chemoprevention campaign. Promotional campaigning and trainings were integrated into app roll-out as part of a national malaria disease program. Along with the launch, an extensive campaign was also conducted in Uganda, including a month-long intensive mass media campaign through television, radio, local newspapers and a press conference. The launch attracted hundreds of stakeholders in pharmacovigilance [45]. ADR reporting tools based on user-friendly technology could be appealing to the public and strengthening the culture of reporting. Moreover, integration of pharmacovigilance into the work of other relevant partners, such as public health programs, should have been an indispensable contributor to successful app implementation, and might have indirectly resulted in a rise in the overall number of reports.

Although offline app function was cited as a positive driver in influencing patients and HCPs to use the ADR reporting app in the Innovative Medicines Initiative (IMI) WEB-Recognizing Adverse Drug Reactions (RADR) project [46, 47], this study reported persistent difficulties due to internet connectivity issues in some geographical settings. In this survey, this challenge was reported particularly by LMICs and given as one of the reasons for the low use of the app despite the offline function. Although it was reported that there was a high coverage of fourth generation broadband cellular network technology (4G) in LMICs (82% of the population), it is also known that such services are not affordable in more than half of LMICs [48, 49]. In addition, internet stability is a challenge due to unexpected technical issues and sporadic political restrictions [50, 51]. App use could also be influenced by individual socio-demographic status and level of interest [46], and is preferred by the younger generation and those with an interest in the technology. Additional barriers exist in LMICs due to rural–urban differences and gender gaps in mobile internet use, as well as a lack of literacy and digital skills [48]. It may be important to consider the level of digital literacy required to master the settings when introducing an app, to identify additional follow-on support, such as training, that might be required to ensure optimal app implementation and use.

The overall benefit of the apps should be examined alongside the financial affordability of the technology. Conventional paper reporting incurs various production costs, such as for paper, printing, distribution and postage. In addition to these direct costs, the indirect costs related to the human resources, time and effort required to manually process each paper form received would not be negligible [37]. The cost saving on these items that the apps offer was positively appraised in our online survey. Nevertheless, apps incur their own costs throughout implementation, e.g. pre-development research, design, technical development, testing, deployment, and continual support and maintenance [52]. To ensure that NRAs make cost-effective decisions, we highlight the importance of considering the cost as well as the benefits of an app when deciding whether to develop and implement it.

Although this study presents globally generalizable evidence based on a robust systematic methodology, the results should be cautiously interpreted due to their limitations. Firstly, although the number of hits from the searches of the Google Play Store and App Store varied depending on the search terms, their seemed to be a ceiling of about 250 items in the displayable hit numbers, especially in the Google Play Store. As a result, it may be possible that some apps did not appear in our searches. Secondly, we ran the searches using English terms, possibly excluding some of the non-English apps, although apps available only in languages other than English were also picked up by our search strategy and included in the study. Also, searches were performed from a single location (Geneva, Switzerland) and the results may differ from another search location. A supplemental study using search terms translated into various languages, such as the official United Nations languages, and performed in different locations worldwide would be worthwhile. Lastly, information technology is advancing rapidly. The selected apps will no doubt be updated and the latest versions currently in use might not be identical to the ones reviewed in this study. Also, additional apps may have been developed in the interim. It would be useful to repeat this survey periodically, to capture innovations and new pharmacovigilance apps.

Conclusions

There are various kinds of ADR reporting tools. App-based ADR reporting tools are becoming more popular in different regions of the world and they contribute to ADR reporting through technological features. They also strengthen the overall culture of ADR reporting in their appeal to a wider group of reporters and due to the ease of reporting. ADR reporting apps have the potential to support pharmacovigilance activity; however, in launching an app, it is important to consider features and functions that can contribute to a qualitative and quantitative improvement in reporting, and considering the proposed user group, any training needs, and the costs involved to develop, launch and maintain the app. Moreover, further post-implementation studies would help assess the long-term impact of app-based tools and how these can be sustained.

Appendices

Appendix 1: Search terms used in App Store and Google Play Store

1. ‘adverse event’ 2. ‘ADR drug’ 3. ‘ADR reporting’ 4. ‘adverse drug event’ 5. ‘adverse drug reaction’ 6. ‘adverse effect’ 7. ‘adverse outcome’ 8. ‘adverse reaction’ 9. ‘drug interaction’ 10. ‘drug safety report’ 11. ‘drug surveillance’ 12. ‘drug toxicity’ 13. ‘medication error’ 14. ‘medicine safety’ 15. ‘patient safety’ 16. ‘pharmacovigilance’ 17. ‘side effect’

Appendix 2: List of questions asked in the online survey

1. On which date was the app launched? 2. On which date was the app last updated? 3. What was the purpose of the app’s development/implementation? 4. To which countries does the app allow users to submit ADR reports? 5. What was the total number of downloads of the app from the iOS platform since the app’s launch?     a. 1–100     b. 101–1000     c. 1001–10,000     d. 10,001–50,000     e. 50,001 or more     f. App is not available on the iOS platform     g. Unknown 6. What is the total number of downloads of the app from the Android platform since the app’s launch?     a. 1–100     b. 101–1000     c. 1001–10,000     d. 10,001–50,000     e. 50,001 or more     f. App is not available on the Android platform     g. Unknown 7. Did you notice an upward or downward trend in the number of downloads of the app per month in 2020 compared to 2019?     a. Upward trend     b. Downward trend     c. No change     d. Unknown 8. How has the app’s implementation impacted the overall number of ADR reports that you receive?     a. The app’s implementation has increased the number of ADR reports     b. The app’s implementation has decreased the number of ADR reports     c. The app’s implementation has not impacted the number of ADR reports     d. Unknown     e. Other 9. How many ADR reports have been received by the app since its launch? 10. What is the proportion of ADR reports received from the app among all reports received since the app’s launch? (%) 11. What was the most frequent reporter type using the app for ADR reporting?     a. Medical doctor     b. Pharmacist     c. Other healthcare professional     d. Patient or family member     e. Public health program     f. Unknown     g. Other 12. In general, among the following essential elements*, which were appropriately filled-in on the ADR reports received by the app? (Check all relevant answers.)     a. Patient     b. Reporter     c. Product exposure     d. Event     e. The information is unknown 13. In general, among the following essential elements*, which were appropriately filled-in on the ADR reports received on a paper reporting form? (Check all relevant answers.)     a. Patient     b. Reporter     c. Product exposure     d. Event     e. The information is unknown 14. If you have any further comments on the quality and quantity of ADR reports received by the app compared to reports received through other tools, please add these below 15. What was the total cost of app development? Please enter an amount in the appropriate currency. If unknown, please say ‘Unknown’ 16. What is the annual cost of app maintenance? Please enter an amount in the appropriate currency. If unknown, please say ‘Unknown’ 17. What are the funding sources for these costs? If unknown, please say ‘Unknown’ 18. Please share your experiences of the development of the app 19. What are the pros and cons of the app?

1. *Elements making up the “minimum information for valid safety report” as defined by the ICH.