Identifying assessment frameworks for digital public health interventions: First results of a scoping review
Henrik Hülsmann 1,2Walter Swoboda 1
Felix Holl 1,2
1 DigiHealth Institute, Neu-Ulm University of Applied Sciences, Neu-Ulm, Germany
2 Leibniz Science Campus Digital Public Health, Bremen, Germany
Abstract
The global pandemic accelerated the rise of digital public health (DPH), yet standardized assessment procedures for digital public health interventions (DPHIs) remain limited. This scoping review aims to identify existing assessment frameworks applicable to DPHIs. A comprehensive search was conducted across PubMed, the Healthcare Administration Database, and Google Scholar, following Joanna Briggs Institute (JBI) guidelines. English-language full-text studies published since 2020 were included. Of 1,196 studies screened, 12 met the inclusion criteria. One DPHI framework was identified, alongside three generic digital health frameworks and an adapted RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) model. Findings reveal a significant gap in validated assessment frameworks for DPHIs. Broader frameworks that consider social, cultural, and country-specific factors are needed to fully realize the potential of DPHIs and address issues like the digital divide.
Keywords
digital public health, health technology assessment, framework
Introduction
The COVID-19 pandemic has forced national healthcare systems around the world to adapt, accelerating the digital transformation of healthcare in particular [1]. At the same time, the field of digital public health (DPH) advanced and changed the area of population-based disease prevention, control, and health promotion with digital transformation [2], [3]. Digital public health interventions (DPHIs) could enable personalized and precise health solutions, improve prediction and data analytics, contribute to a shift from cure to prevention, bring care closer to people, and provide more efficient services [4].
Assessment methods for DPHIs have lagged behind rapid technological advancements, as evidenced by the limited number of evaluation studies [2]. This gap has led to insufficient integration of public health perspectives in digital health intervention planning and assessment [5]. Critical assessment frameworks are needed to ensure the effectiveness, sustainability, and safety of DPHIs [6]. This scoping review aims to identify existing or potential frameworks applicable to DPH contexts that could support the practice of systematic and evidence-based decision-making.
Method
A scoping review was conducted in August 2024 according to the Joanna Briggs Institute (JBI) scoping review methodology guidelines [7], using predefined search strings (e.g.("Digital health" OR "eHealth") AND "public health" AND ("assessment" OR "evaluation") AND ("framework" OR "tool")) in PubMed, Healthcare Administration Database (HAD), and Google Scholar. Inclusion criteria were studies published in English since January 1st, 2020, which describe generic assessment frameworks for digital health interventions, the generic assessment of a DPHI, and functional frameworks specific to DPH. A total of 1,196 studies were screened by title and abstract. After full-text review, twelve studies were included. An adapted EUnetHTA model [8] was used to compare frameworks across key domains.
Results
Five relevant frameworks for evaluating DPHIs were identified: DigiPHrame [9], NICE’s Evidence Standards Framework (ESF) for digital health technologies [10], Digi-HTA [11], Digital Health Technology Evaluation for Health Organizations (DHTEfHO) [12], and an adapted RE-AIM framework [13]. Of the initiatives reviewed, DigiPHrame was uniquely designed to address the specific needs of DPHIs, while the others have a broader focus on either digital health technologies or public health. The development of the frameworks typically involved literature reviews, expert panels, and stakeholder engagement.
Assessment methods varied, including scoring systems and structured guidance through key questions. DigiPHrame demonstrated the most comprehensive coverage, particularly in addressing ethical, legal, and equity considerations. Most frameworks covered to some extent effectiveness and safety, while technical [12], [13] and economic [13] aspects were inconsistently evaluated. Social, ethical, and organizational impacts were underrepresented [10], [11], [13].
Discussion
DigiPHrame includes evaluation criteria tailored to DPHIs, but its methodological foundation remains underdeveloped. It lacks standardized procedures for evidence generation, mechanisms for weighting criteria, and a quantifiable scoring system to ensure comparability, remaining untested and unvalidated. Notably, other frameworks either neglect technical aspects such as interoperability or define country-specific technical guidelines [14]. Overall, these assessment frameworks do not adequately address the most important, mutually reinforcing pillars of digital public health [6], which include ethical and social dimensions. The strengths of this review and its further results will be the focus on specific digital public health frameworks. At this stage, the review is limited to its restricted search terms and databases.
Conclusion
The assessment of DPHIs faces significant challenges due to their complexity, diversity of stakeholders, and dynamic development, making evidence generation and transferability difficult. There is a critical need for adaptable, context-sensitive assessment frameworks that address the evolving nature of DPHIs and support evidence-based public health strategies.
Notes
Authors’ ORCIDs
- Henrik Hülsmann: 0009-0003-5283-4488
- Walter Swoboda: 0000-0002-3836-311X
- Felix Holl: 0000-0002-4020-9509
Competing interests
The authors declare that they have no competing interests. Felix Holl and Henrik Hülsmann are affiliated with the Leibniz Science Campus Digital Public Health. DigiPHrame was developed by members of the Leibniz Science Campus Digital Public Health.
References
[1] Wong BLH, Maaß L, Vodden A, van Kessel R, Sorbello S, Buttigieg S, Odone A; European Public Health Association (EUPHA) Digital Health Section. The dawn of digital public health in Europe: Implications for public health policy and practice. Lancet Reg Health Eur. 2022 Mar;14:100316. DOI: 10.1016/j.lanepe.2022.100316[2] Zeeb H, Pigeot I, Schüz B. Digital Public Health – Rasanter technischer Fortschritt, aber viele offene Public-Health-Fragen [Digital public health-rapid technological progress, but many open public health questions]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2020 Feb;63(2):135-6. DOI: 10.1007/s00103-020-03092-0
[3] Zeeb H, Schüz B, Schultz T, Pigeot I. Entwicklungen in der Digitalisierung von Public Health seit 2020: Beispiele aus dem Leibniz-WissenschaftsCampus Digital Public Health Bremen [Developments in the digitalization of public health since 2020: Examples from the Leibniz ScienceCampus Digital Public Health Bremen]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2024 Mar;67(3):260-7. DOI: 10.1007/s00103-023-03827-9
[4] Odone A, Buttigieg S, Ricciardi W, Azzopardi-Muscat N, Staines A. Public health digitalization in Europe. Eur J Public Health. 2019 Oct;29(Supplement_3):28-35. DOI: 10.1093/eurpub/ckz161
[5] Dockweiler C, Razum O. Digitalisierte Gesundheit: neue Herausforderungen für Public Health [Digital Health: new Challenges for Public Health]. Gesundheitswesen. 2016 Jan;78(1):5-7. DOI: 10.1055/s-0041-110679
[6] Iyamu I, Gómez-Ramírez O, Xu AX, Chang HJ, Watt S, Mckee G, Gilbert M. Challenges in the development of digital public health interventions and mapped solutions: Findings from a scoping review. Digit Health. 2022 Dec;8:20552076221102255. DOI: 10.1177/20552076221102255
[7] von Elm E, Schreiber G, Haupt CC. Methodische Anleitung für Scoping Reviews (JBI-Methodologie). Z Evid Fortbild Qual Gesundhwes. 2019 Jun;143:1-7. DOI: 10.1016/j.zefq.2019.05.004
[8] Kolasa K, Kozinski G. How to Value Digital Health Interventions? A Systematic Literature Review. Int J Environ Res Public Health. 2020 Mar;17(6):2119. DOI: 10.3390/ijerph17062119
[9] Pan CC, Pedros Barnils N, Jürgens D, Muellmann S, Janetzki S, Kolschen J, Freye M, Dassow HH, Lange O, Rogowski W, Reinschluessel A, Forberger S, Jahnel T, Schüz B, Gerhardus A. Developing and Assessing Digital Public Health Interventions: A Digital Public Health Framework (DigiPHrame). Version 1.1. Bremen: Open Science Framework; 2023. DOI: 10.17605/OSF.IO/UB3W4
[10] National Institute for Health and Care Excellence. Evidence standards framework for digital health technologies. 2022 [cited 2024 Apr 21]. Available from: https://www.nice.org.uk/corporate/ecd7/chapter/section-a-technologies-suitable-for-evaluation-using-the-evidence-standards-framework
[11] Haverinen J, Keränen N, Falkenbach P, Maijala A, Kolehmainen T, Reponen J. Digi-HTA: Health technology assessment framework for digital healthcare services. Finn J EHealth EWelfare. 2019 Jan 1;11(4):326-41. DOI: 10.23996/fjhw.82538
[12] Digital Health Division, Israel Ministry of Health. Digital health technology evaluation for health organizations: an evaluation framework for early-stage technologies. 2021 [cited 2024 Apr 24]. Available from: https://www.gov.il/BlobFolder/generalpage/digital-health-guide-062021/he/files_publications_digital_health_digital-health-hta-062021.pdf
[13] de Moel-Mandel C, Lynch C, Issaka A, Braver J, Zisis G, Carrington MJ, Oldenburg B. Optimising the implementation of digital-supported interventions for the secondary prevention of heart disease: a systematic review using the RE-AIM planning and evaluation framework. BMC Health Serv Res. 2023 Dec;23(1):1347. DOI: 10.1186/s12913-023-10361-6
[14] NHS Transformation Directorate. Digital Technology Assessment Criteria (DTAC). [cited 2024 Apr 2]. Available from: https://transform.england.nhs.uk/key-tools-and-info/digital-technology-assessment-criteria-dtac/
