Identifying assessment frameworks for digital public health interventions: First results of a scoping review

mibe000289 10.3205/mibe000289 urn:nbn:de:0183-mibe0002896 Research Article Identifying assessment frameworks for digital public health interventions: First results of a scoping review Identifizierung von Bewertungsrahmen für digitale Public-Health-Interventionen: Initiale Ergebnisse eines Scoping Reviews Hülsmann Hülsmann Henrik H

Hochschule Neu-Ulm, Wileystr. 1, 89231 Neu-Ulm, GermanyDigiHealth Institute, Neu-Ulm University of Applied Sciences, Neu-Ulm, GermanyLeibniz Science Campus Digital Public Health, Bremen, Germany

Henrik.huelsmann@hnu.de author Swoboda Swoboda Walter W

DigiHealth Institute, Neu-Ulm University of Applied Sciences, Neu-Ulm, Germany

author Holl Holl Felix F

DigiHealth Institute, Neu-Ulm University of Applied Sciences, Neu-Ulm, Germany Leibniz Science Campus Digital Public Health, Bremen, Germany

author German Medical Science GMS Publishing House

Düsseldorf

610 digital public health health technology assessment framework Digital Public Health Health Technology Assessment Framework EFMI STC 2025 20251017 engl This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License. Dieser Artikel ist ein Open-Access-Artikel und steht unter den Lizenzbedingungen der Creative Commons Attribution 4.0 License (Namensnennung). 1860-9171 21 GMS Medizinische Informatik, Biometrie und Epidemiologie GMS Med Inform Biom Epidemiol 17 Die globale Pandemie hat den Aufstieg von Digital Public Health (DPH) beschleunigt, jedoch sind standardisierte Bewertungsverfahren für digitale Public-Health-Interventionen (DPHI) nach wie vor selten. Diese Übersichtsarbeit zielt darauf ab, bestehende Bewertungsrahmen zu identifizieren, die auf DPHI anwendbar sind. Es wurde eine umfassende Suche in PubMed, der Healthcare Administration Database und Google Scholar durchgeführt, die den Richtlinien des Joanna Briggs Institute (JBI) folgte. Eingeschlossen wurden englischsprachige Volltextstudien, die seit 2020 veröffentlicht wurden. Von 1.196 gescreenten Studien erfüllten 12 die Einschlusskriterien. Es wurden ein Bewertungsrahm</PlainText></TextGroup>en für DPHI, drei generische Bewertungsrahmen für digitale Gesundheitsinterventionen und ein angepasstes RE-AIM-Modell (Reichweite, Effektivität, Anpassung, Implementation und Erhaltung) identifiziert. Die Ergebnisse zeigen eine erhebliche Lücke bei validierten Bewertungsrahmen für DPHI. Um das Potenzial von DPHI voll auszuschöpfen und Probleme wie den „Digital Divide“ anzugehen, sind umfassendere Rahmenwerke erforderlich, die soziale, kulturelle und länderspezifische Faktoren berücksichtigen.</Pgraph></Abstract> <Abstract language="en" linked="yes"><Pgraph>The global pandemic accelerated the rise of digital public health (DPH), yet standardized assessment procedures for digital public health interventions (DPHIs) remain limited. This scoping review aims to identify existing assessment frameworks applicable to DPHIs. A comprehensive search was conducted across PubMed, the Healthcare Administration Database, and Google Scholar, following Joanna Briggs Institute (JBI) guidelines. English-language full-text studies published since 2020 were included. Of 1,196 studies screened, 12 met the inclusion criteria. One DPHI framework was identified, alongside three generic digital health frameworks and an adapted RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) model. Findings reveal a significant gap in validated assessment frameworks for DPHIs. Broader frameworks that consider social, cultural, and country-specific factors are needed to fully realize the potential of DPHIs and address issues like the digital divide.</Pgraph></Abstract> <TextBlock name="Introduction" linked="yes"> <MainHeadline>Introduction</MainHeadline><Pgraph>The COVID-19 pandemic has forced national healthcare systems around the world to adapt, accelerating the digital transformation of healthcare in particular <TextLink reference="1"></TextLink>. At the same time, the field of digital public health (DPH) advanced and changed the area of population-based disease prevention, control, and health promotion with digital transformation <TextLink reference="2"></TextLink>, <TextLink reference="3"></TextLink>. Digital public health interventions (DPHIs) could enable personalized and precise health solutions, improve prediction and data analytics, contribute to a shift from cure to prevention, bring care closer to people, and provide more efficient services <TextLink reference="4"></TextLink>.</Pgraph><Pgraph>Assessment methods for DPHIs have lagged behind rapid technological advancements, as evidenced by the limited number of evaluation studies <TextLink reference="2"></TextLink>. This gap has led to insufficient integration of public health perspectives in digital health intervention planning and assessment <TextLink reference="5"></TextLink>. Critical assessment frameworks are needed to ensure the effectiveness, sustainability, and safety of DPHIs <TextLink reference="6"></TextLink>. This scoping review aims to identify existing or potential frameworks applicable to DPH contexts that could support the practice of systematic and evidence-based decision-making.</Pgraph></TextBlock> <TextBlock name="Method" linked="yes"> <MainHeadline>Method</MainHeadline><Pgraph>A scoping review was conducted in August 2024 according to the Joanna Briggs Institute (JBI) scoping review methodology guidelines <TextLink reference="7"></TextLink>, using predefined search strings (e.g.("Digital health" OR "eHealth") AND "public health" AND ("assessment" OR "evaluation") AND ("framework" OR "tool")) in PubMed, Healthcare Administration Database (HAD), and Google Scholar. Inclusion criteria were studies published in English since January 1<Superscript>st</Superscript>, 2020, which describe generic assessment frameworks for digital health interventions, the generic assessment of a DPHI, and functional frameworks specific to DPH. A total of 1,196 studies were screened by title and abstract. After full-text review, twelve studies were included. An adapted EUnetHTA model <TextLink reference="8"></TextLink> was used to compare frameworks across key domains. </Pgraph></TextBlock> <TextBlock name="Results" linked="yes"> <MainHeadline>Results</MainHeadline><Pgraph>Five relevant frameworks for evaluating DPHIs were identified: DigiPHrame <TextLink reference="9"></TextLink>, NICE’s Evidence Standards Framework (ESF) for digital health technologies <TextLink reference="10"></TextLink>, Digi-HTA <TextLink reference="11"></TextLink>, Digital Health Technology Evaluation for Health Organizations (DHTEfHO) <TextLink reference="12"></TextLink>, and an adapted RE-AIM framework <TextLink reference="13"></TextLink>. Of the initiatives reviewed, DigiPHrame was uniquely designed to address the specific needs of DPHIs, while the others have a broader focus on either digital health technologies or public health. The development of the frameworks typically involved literature reviews, expert panels, and stakeholder engagement. </Pgraph><Pgraph>Assessment methods varied, including scoring systems and structured guidance through key questions. DigiPHrame demonstrated the most comprehensive coverage, particularly in addressing ethical, legal, and equity considerations. Most frameworks covered to some extent effectiveness and safety, while technical <TextLink reference="12"></TextLink>, <TextLink reference="13"></TextLink> and economic <TextLink reference="13"></TextLink> aspects were inconsistently evaluated. Social, ethical, and organizational impacts were underrepresented <TextLink reference="10"></TextLink>, <TextLink reference="11"></TextLink>, <TextLink reference="13"></TextLink>. </Pgraph></TextBlock> <TextBlock name="Discussion" linked="yes"> <MainHeadline>Discussion</MainHeadline><Pgraph>DigiPHrame includes evaluation criteria tailored to DPHIs, but its methodological foundation remains underdeveloped. It lacks standardized procedures for evidence generation, mechanisms for weighting criteria, and a quantifiable scoring system to ensure comparability, re<TextGroup><PlainText>m</PlainText></TextGroup>aining untested and unvalidated. Notably, other frameworks either neglect technical aspects such as interoperability or define country-specific technical guidelines <TextLink reference="14"></TextLink>. Overall, these assessment frameworks do not adequately address the most important, mutually reinforcing pillars of digital public health <TextLink reference="6"></TextLink>, which include ethical and social dimensions. The strengths of this review and its further results will be the focus on specific digital public health frameworks. At this stage, the review is limited to its restricted search terms and databases. </Pgraph></TextBlock> <TextBlock name="Conclusion" linked="yes"> <MainHeadline>Conclusion</MainHeadline><Pgraph>The assessment of DPHIs faces significant challenges due to their complexity, diversity of stakeholders, and dynamic development, making evidence generation and transferability difficult. There is a critical need for adaptable, context-sensitive assessment frameworks that address the evolving nature of DPHIs and support evidence-based public health strategies.</Pgraph></TextBlock> <TextBlock name="Notes" linked="yes"> <MainHeadline>Notes</MainHeadline><SubHeadline>Authors’ ORCIDs</SubHeadline><Pgraph><UnorderedList><ListItem level="1">Henrik Hülsmann: <Hyperlink href="https://orcid.org/0009-0003-5283-4488">0009-0003-5283-4488</Hyperlink></ListItem><ListItem level="1">Walter Swoboda: <Hyperlink href="https://orcid.org/0000-0002-3836-311X">0000-0002-3836-311X</Hyperlink></ListItem><ListItem level="1">Felix Holl: <Hyperlink href="http://orcid.org/0000-0002-4020-9509">0000-0002-4020-9509</Hyperlink></ListItem></UnorderedList></Pgraph><SubHeadline>Competing interests</SubHeadline><Pgraph>The authors declare that they have no competing interests. Felix Holl and Henrik Hülsmann are affiliated with the Leibniz Science Campus Digital Public Health. <TextGroup><PlainText>DigiPHrame</PlainText></TextGroup> was developed by members of the Leibniz Science Campus Digital Public Health.</Pgraph></TextBlock> <References linked="yes"> <Reference refNo="1"> <RefAuthor>Wong BLH</RefAuthor> <RefAuthor>Maaß L</RefAuthor> <RefAuthor>Vodden A</RefAuthor> <RefAuthor>van Kessel R</RefAuthor> <RefAuthor>Sorbello S</RefAuthor> <RefAuthor>Buttigieg S</RefAuthor> <RefAuthor>Odone A</RefAuthor> <RefAuthor> European Public Health Association (EUPHA) Digital Health Section</RefAuthor> <RefTitle>The dawn of digital public health in Europe: Implications for public health policy and practice</RefTitle> <RefYear>2022</RefYear> <RefJournal>Lancet Reg Health Eur</RefJournal> <RefPage>100316</RefPage> <RefTotal>Wong BLH, Maaß L, Vodden A, van Kessel R, Sorbello S, Buttigieg S, Odone A; European Public Health Association (EUPHA) Digital Health Section. The dawn of digital public health in Europe: Implications for public health policy and practice. Lancet Reg Health Eur. 2022 Mar;14:100316. DOI: 10.1016/j.lanepe.2022.100316</RefTotal> <RefLink>https://doi.org/10.1016/j.lanepe.2022.100316</RefLink> </Reference> <Reference refNo="2"> <RefAuthor>Zeeb H</RefAuthor> <RefAuthor>Pigeot I</RefAuthor> <RefAuthor>Schüz B</RefAuthor> <RefTitle>Digital Public Health – Rasanter technischer Fortschritt, aber viele offene Public-Health-Fragen</RefTitle> <RefYear>2020</RefYear> <RefJournal>Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz</RefJournal> <RefPage>135-6</RefPage> <RefTotal>Zeeb H, Pigeot I, Schüz B. Digital Public Health – Rasanter technischer Fortschritt, aber viele offene Public-Health-Fragen [Digital public health-rapid technological progress, but many open public health questions]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2020 Feb;63(2):135-6. DOI: 10.1007/s00103-020-03092-0</RefTotal> <RefLink>https://doi.org/10.1007/s00103-020-03092-0</RefLink> </Reference> <Reference refNo="3"> <RefAuthor>Zeeb H</RefAuthor> <RefAuthor>Schüz B</RefAuthor> <RefAuthor>Schultz T</RefAuthor> <RefAuthor>Pigeot I</RefAuthor> <RefTitle>Entwicklungen in der Digitalisierung von Public Health seit 2020: Beispiele aus dem Leibniz-WissenschaftsCampus Digital Public Health Bremen</RefTitle> <RefYear>2024</RefYear> <RefJournal>Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz</RefJournal> <RefPage>260-7</RefPage> <RefTotal>Zeeb H, Schüz B, Schultz T, Pigeot I. Entwicklungen in der Digitalisierung von Public Health seit 2020: Beispiele aus dem Leibniz-WissenschaftsCampus Digital Public Health Bremen [Developments in the digitalization of public health since 2020: Examples from the Leibniz ScienceCampus Digital Public Health Bremen]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2024 Mar;67(3):260-7. DOI: 10.1007/s00103-023-03827-9</RefTotal> <RefLink>https://doi.org/10.1007/s00103-023-03827-9</RefLink> </Reference> <Reference refNo="4"> <RefAuthor>Odone A</RefAuthor> <RefAuthor>Buttigieg S</RefAuthor> <RefAuthor>Ricciardi W</RefAuthor> <RefAuthor>Azzopardi-Muscat N</RefAuthor> <RefAuthor>Staines A</RefAuthor> <RefTitle>Public health digitalization in Europe</RefTitle> <RefYear>2019</RefYear> <RefJournal>Eur J Public Health</RefJournal> <RefPage>28-35</RefPage> <RefTotal>Odone A, Buttigieg S, Ricciardi W, Azzopardi-Muscat N, Staines A. Public health digitalization in Europe. Eur J Public Health. 2019 Oct;29(Supplement_3):28-35. DOI: 10.1093/eurpub/ckz161</RefTotal> <RefLink>https://doi.org/10.1093/eurpub/ckz161</RefLink> </Reference> <Reference refNo="5"> <RefAuthor>Dockweiler C</RefAuthor> <RefAuthor>Razum O</RefAuthor> <RefTitle>Digitalisierte Gesundheit: neue Herausforderungen für Public Health</RefTitle> <RefYear>2016</RefYear> <RefJournal>Gesundheitswesen</RefJournal> <RefPage>5-7</RefPage> <RefTotal>Dockweiler C, Razum O. Digitalisierte Gesundheit: neue Herausforderungen für Public Health [Digital Health: new Challenges for Public Health]. Gesundheitswesen. 2016 Jan;78(1):5-7. DOI: 10.1055/s-0041-110679</RefTotal> <RefLink>https://doi.org/10.1055/s-0041-110679</RefLink> </Reference> <Reference refNo="6"> <RefAuthor>Iyamu I</RefAuthor> <RefAuthor>Gómez-Ramírez O</RefAuthor> <RefAuthor>Xu AX</RefAuthor> <RefAuthor>Chang HJ</RefAuthor> <RefAuthor>Watt S</RefAuthor> <RefAuthor>Mckee G</RefAuthor> <RefAuthor>Gilbert M</RefAuthor> <RefTitle>Challenges in the development of digital public health interventions and mapped solutions: Findings from a scoping review</RefTitle> <RefYear>2022</RefYear> <RefJournal>Digit Health</RefJournal> <RefPage>20552076221102255</RefPage> <RefTotal>Iyamu I, Gómez-Ramírez O, Xu AX, Chang HJ, Watt S, Mckee G, Gilbert M. Challenges in the development of digital public health interventions and mapped solutions: Findings from a scoping review. Digit Health. 2022 Dec;8:20552076221102255. DOI: 10.1177/20552076221102255</RefTotal> <RefLink>https://doi.org/10.1177/20552076221102255</RefLink> </Reference> <Reference refNo="7"> <RefAuthor>von Elm E</RefAuthor> <RefAuthor>Schreiber G</RefAuthor> <RefAuthor>Haupt CC</RefAuthor> <RefTitle>Methodische Anleitung für Scoping Reviews (JBI-Methodologie)</RefTitle> <RefYear>2019</RefYear> <RefJournal>Z Evid Fortbild Qual Gesundhwes</RefJournal> <RefPage>1-7</RefPage> <RefTotal>von Elm E, Schreiber G, Haupt CC. Methodische Anleitung für Scoping Reviews (JBI-Methodologie). Z Evid Fortbild Qual Gesundhwes. 2019 Jun;143:1-7. DOI: 10.1016/j.zefq.2019.05.004</RefTotal> <RefLink>https://doi.org/10.1016/j.zefq.2019.05.004</RefLink> </Reference> <Reference refNo="9"> <RefAuthor>Pan CC</RefAuthor> <RefAuthor>Pedros Barnils N</RefAuthor> <RefAuthor>Jürgens D</RefAuthor> <RefAuthor>Muellmann S</RefAuthor> <RefAuthor>Janetzki S</RefAuthor> <RefAuthor>Kolschen J</RefAuthor> <RefAuthor>Freye M</RefAuthor> <RefAuthor>Dassow HH</RefAuthor> <RefAuthor>Lange O</RefAuthor> <RefAuthor>Rogowski W</RefAuthor> <RefAuthor>Reinschluessel A</RefAuthor> <RefAuthor>Forberger S</RefAuthor> <RefAuthor>Jahnel T</RefAuthor> <RefAuthor>Schüz B</RefAuthor> <RefAuthor>Gerhardus A</RefAuthor> <RefTitle></RefTitle> <RefYear>2023</RefYear> <RefBookTitle>Developing and Assessing Digital Public Health Interventions: A Digital Public Health Framework (DigiPHrame)</RefBookTitle> <RefPage></RefPage> <RefTotal>Pan CC, Pedros Barnils N, Jürgens D, Muellmann S, Janetzki S, Kolschen J, Freye M, Dassow HH, Lange O, Rogowski W, Reinschluessel A, Forberger S, Jahnel T, Schüz B, Gerhardus A. Developing and Assessing Digital Public Health Interventions: A Digital Public Health Framework (DigiPHrame). Version 1.1. Bremen: Open Science Framework; 2023. DOI: 10.17605/OSF.IO/UB3W4</RefTotal> <RefLink>https://doi.org/10.17605/OSF.IO/UB3W4</RefLink> </Reference> <Reference refNo="10"> <RefAuthor>National Institute for Health and Care Excellence</RefAuthor> <RefTitle></RefTitle> <RefYear>2022</RefYear> <RefBookTitle>Evidence standards framework for digital health technologies</RefBookTitle> <RefPage></RefPage> <RefTotal>National Institute for Health and Care Excellence. Evidence standards framework for digital health technologies. 2022 [cited 2024 Apr 21]. Available from: https://www.nice.org.uk/corporate/ecd7/chapter/section-a-technologies-suitable-for-evaluation-using-the-evidence-standards-framework</RefTotal> <RefLink>https://www.nice.org.uk/corporate/ecd7/chapter/section-a-technologies-suitable-for-evaluation-using-the-evidence-standards-framework</RefLink> </Reference> <Reference refNo="11"> <RefAuthor>Haverinen J</RefAuthor> <RefAuthor>Keränen N</RefAuthor> <RefAuthor>Falkenbach P</RefAuthor> <RefAuthor>Maijala A</RefAuthor> <RefAuthor>Kolehmainen T</RefAuthor> <RefAuthor>Reponen J</RefAuthor> <RefTitle>Digi-HTA: Health technology assessment framework for digital healthcare services</RefTitle> <RefYear>2019</RefYear> <RefJournal>Finn J EHealth EWelfare</RefJournal> <RefPage>326-41</RefPage> <RefTotal>Haverinen J, Keränen N, Falkenbach P, Maijala A, Kolehmainen T, Reponen J. Digi-HTA: Health technology assessment framework for digital healthcare services. Finn J EHealth EWelfare. 2019 Jan 1;11(4):326-41. DOI: 10.23996/fjhw.82538</RefTotal> <RefLink>https://doi.org/10.23996/fjhw.82538</RefLink> </Reference> <Reference refNo="12"> <RefAuthor>Digital Health Division</RefAuthor> <RefAuthor>Israel Ministry of Health</RefAuthor> <RefTitle></RefTitle> <RefYear>2021</RefYear> <RefBookTitle>Digital health technology evaluation for health organizations: an evaluation framework for early-stage technologies</RefBookTitle> <RefPage></RefPage> <RefTotal>Digital Health Division, Israel Ministry of Health. Digital health technology evaluation for health organizations: an evaluation framework for early-stage technologies. 2021 [cited 2024 Apr 24]. Available from: https://www.gov.il/BlobFolder/generalpage/digital-health-guide-062021/he/files_publications_digital_health_digital-health-hta-062021.pdf</RefTotal> <RefLink>https://www.gov.il/BlobFolder/generalpage/digital-health-guide-062021/he/files_publications_digital_health_digital-health-hta-062021.pdf</RefLink> </Reference> <Reference refNo="13"> <RefAuthor>de Moel-Mandel C</RefAuthor> <RefAuthor>Lynch C</RefAuthor> <RefAuthor>Issaka A</RefAuthor> <RefAuthor>Braver J</RefAuthor> <RefAuthor>Zisis G</RefAuthor> <RefAuthor>Carrington MJ</RefAuthor> <RefAuthor>Oldenburg B</RefAuthor> <RefTitle>Optimising the implementation of digital-supported interventions for the secondary prevention of heart disease: a systematic review using the RE-AIM planning and evaluation framework</RefTitle> <RefYear>2023</RefYear> <RefJournal>BMC Health Serv Res</RefJournal> <RefPage>1347</RefPage> <RefTotal>de Moel-Mandel C, Lynch C, Issaka A, Braver J, Zisis G, Carrington MJ, Oldenburg B. Optimising the implementation of digital-supported interventions for the secondary prevention of heart disease: a systematic review using the RE-AIM planning and evaluation framework. BMC Health Serv Res. 2023 Dec;23(1):1347. DOI: 10.1186/s12913-023-10361-6</RefTotal> <RefLink>https://doi.org/10.1186/s12913-023-10361-6</RefLink> </Reference> <Reference refNo="8"> <RefAuthor>Kolasa K</RefAuthor> <RefAuthor>Kozinski G</RefAuthor> <RefTitle>How to Value Digital Health Interventions? A Systematic Literature Review</RefTitle> <RefYear>2020</RefYear> <RefJournal>Int J Environ Res Public Health</RefJournal> <RefPage>2119</RefPage> <RefTotal>Kolasa K, Kozinski G. How to Value Digital Health Interventions? A Systematic Literature Review. Int J Environ Res Public Health. 2020 Mar;17(6):2119. DOI: 10.3390/ijerph17062119</RefTotal> <RefLink>https://doi.org/10.3390/ijerph17062119</RefLink> </Reference> <Reference refNo="14"> <RefAuthor>NHS Transformation Directorate</RefAuthor> <RefTitle></RefTitle> <RefYear></RefYear> <RefBookTitle>Digital Technology Assessment Criteria (DTAC)</RefBookTitle> <RefPage></RefPage> <RefTotal>NHS Transformation Directorate. Digital Technology Assessment Criteria (DTAC). [cited 2024 Apr 2]. Available from: https://transform.england.nhs.uk/key-tools-and-info/digital-technology-assessment-criteria-dtac/</RefTotal> <RefLink>https://transform.england.nhs.uk/key-tools-and-info/digital-technology-assessment-criteria-dtac/</RefLink> </Reference> </References> <Media> <Tables> <NoOfTables>0</NoOfTables> </Tables> <Figures> <NoOfPictures>0</NoOfPictures> </Figures> <InlineFigures> <NoOfPictures>0</NoOfPictures> </InlineFigures> <Attachments> <NoOfAttachments>0</NoOfAttachments> </Attachments> </Media> </OrigData> </GmsArticle>