Ensuring Socio-technical Interoperability in Digital Health
Innovation Processes: An Evaluation Approach
Tim Scheplitz
Research Group Digital Health, Technische Universität Dresden, Dresden, Germany
Keywords: Digital Health Innovation, Evaluation, Interoperability, ReEIF.
Abstract: Integrating Digital Health Innovations (DHI) into healthcare practice remains a challenging task for
innovators. They continuously seek for actionable ways to fulfil the complex web of requirements set by the
target environment. A socio-technical understanding of interoperability offers structurization to this
complexity and becomes a key property that innovators want to ensure during the innovation process.
Nevertheless, scientific guidance remains abstract rather than applicable. This research paper builds on this
point and follows the question how innovators can evaluate their DHI process holistically and tangibly to
promote the later integration into complex healthcare systems. It therefore presents an evaluation approach
based on the Refined eHealth European Interoperability Framework (ReEIF) and results of a qualitative
content analysis. Here, detailed descriptions of the six ReEIF levels and 181 potential parameters for a self-
assessment tool have been derived from prior literature. These findings stimulate future research on
interdependencies within identified aspects of socio-technical interoperability and promote applicable tools
for digital health innovators.
Successful implementation of Digital Health
Innovations (DHI) into daily healthcare remains a
challenging task for science and practice. Prior
research is facing this issue from different points of
view. It provides definitional relationships of key
concepts and types of digital health soultions (Iyawa et
al., 2016; Otto et al., 2018), consolidates valuable
insights of domain-specific diffusion barriers (Hobeck
et al., 2021; Otto and Harst, 2019), and derives success
factors to overcome hurdles (Kowatsch et al., 2019;
Otto, 2019). Also, such scientifically stated knowledge
already found its way into international political
programs, recommendations and interdisciplinary
frameworks (European Commision, 2019; WHO,
2015). Despite this knowledge gain, the capability to
foster DHI and to create intelligent and valuable digital
health systems varies immensely from country to
country (Prim et al., 2017).
Extending the knowledge base might also cause
an increase of complexity, that innovators have to be
aware of and that they have to manage. They face the
challenge to overview all crucial factors valid for
their context and to derive the right actions at the right
time. Practice-oriented research lacks thereby in
offering usefull supporting tools to ensure the later
integration into complex health systems and their
Health Information Systems (HIS) landscapes.
This paper adresses the practice-oriented focus
mentioned above. It follows the research question
how innovators can be supported in evaluating their
DHI process holistically to promote the later
integration into complex healthcare systems. It
presents an evaluation approach that will lead to an
evaluation tool for DHI practioners in further work.
This approach seeks to assess the integration
capability of DHI in the modern healthcare practice.
It uses a socio-technical understanding of Digital
Health (DH) interoperability, basically defined as the
ability of two or more (health information) systems to
effectively and efficiently perform tasks together
(HIMSS, 2020; HL7 International, 2021). Thus,
interoperability is required on different technical and
non-technical levels when a DHI has to be integrated
into healthcare practice. The presented approach
bases on a European consented framework of socio-
technical interoperability, the Refined eHealth
Scheplitz, T.
Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach.
DOI: 10.5220/0011009800003123
In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2022) - Volume 5: HEALTHINF, pages 264-275
ISBN: 978-989-758-552-4; ISSN: 2184-4305
2022 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
European Interoperability Framework (ReEIF), to
structure the complex requirement environment
(eHealth Network, 2015). Furthermore, this paper
presents results of a qualitative content analysis of
existing, domain-specific evaluation approaches to
operationalize the chosen key property. For all six
interoperability levels defined in the ReEIF,
comprehensive descriptions as well as a set of
potential evaluation parameters were formulated.
This paper is structured as follows: Within the
next section, foundations of this work are given by
presenting its practice-oriented motivation, the
conceptual evaluation approach as well as a socio-
technical interpretation of interoperability in the DH
domain. After presenting methodical details in
section 3, findings of the analysis are presented for
each ReEIF level as enriched descriptions. Further
thoughts on ensuring interoperability from an
innovator’s perspective are discussed before
limitations and a conclusion of this work are given.
2.1 Use Case of DHI Practice
This work is intended to stimulate both research and
practice, but focusses primarily the latter perspective
of DH innovators. In this paper, “DH innovators”
describe one or more professionals who are
responsible for the management of a DHI process
starting from defining an initial idea and ending
(hopefully) in integration of a new DH artifact into
healthcare practice. The DHI process itself might
differ due to the artifact’s specificity, intended usage
scenario and organizational circumstances but
somehow pass typical stages of idea creation,
conceptualization, requirements analysis,
development and prototyping as well as a final
integration into existing HIS landscapes. Whether a
DHI process is managed by using agile process
models like SCRUM, traditional sequential
development models (e.g., Waterfall- or V-model) or
hybrid models, evaluating the current progress with
intended objectives is always essential. Also,
evaluation is broadly used from a quality
management perspective within Plan-Do-Check-Act
cycles as stated, for instance, in DIN EN ISO 9000,
ISO/IEC 20000 or ISO/IEC 27001.
In all mentioned stages of a DHI process,
innovators are confronted with the domain’s
complexity. Various interdisciplinary requirements
have to be managed to reach the inherent goal of a
successful integration into healthcare practice. This
task becomes even more crucial as it is mandatory for
further scaling objectives. Thus, innovators shall be
supported in continuous or repeating evaluation
activities to assess how the ongoing DHI process
ensure the integration capability of their DHI artifact
in a pilot environment or, later on, in healthcare
2.2 Evaluation Approach
Based on the formulated support scenario for DH
innovators, a contextual concept of an evaluation tool
was created (see Figure 1). For completeness, this
concept is presented here in simplified form. Starting
by the target group, Innovators aim to develop one or
more DHI and integrate the new artifact(s)
successfully into healthcare practice. They are
thereby confronted with the challenge of managing
technical and non-technical requirements set by the
target environment. Here, an evaluation tool (working
title "Interoptimeter") intends to support innovators
assessing the integration capability of their
innovation (evaluation object). The “Interoptimeter”
provides self-assessment questionnaires with selected
items from different interoperability perspectives and
presents innovators a structured report. This report
includes information about how different
interoperability perspectives are already addressed
and what topics DH innovators should pay more
intention to in further work.
As motivated above, socio-technical
interoperability is used as a key property to ensure
this integration task (evaluation top criterion). It
describes the ability of two or more systems - in this
case the DHI as an artifact) and the target
environment of the digital healthcare practice - to
harmonize with each other and to perform common
tasks effectively and efficiently. For this purpose,
“socio-technical interoperability is generally
systematized via the ReEIF that defines six
interoperability levels for the DH domain (evaluation
sub-criteria). Details of this conceptualization are
presented in the following sections.
Typical for frameworks, the ReEIF systemizes
interoperability but does not provide their tangible
operationalization. Here, the presented research
wants to contribute on this need. It strives for
actionable activities, tasks, or duties that can be
reviewed within a self-assessment by innovators. The
conducted qualitative content analysis enhanced
existing descriptions and derived potential evaluation
parameters. In Figure 1 the parts of the overall
evaluation approach that are served by the results
presented here are highlighted in dark gray.
Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach
Figure 1: Paper's focus (dark gray) within concept of the “Interoptimeter” evaluation tool.
The evaluation approach also takes differences of
DHI types into account. For this purpose, established
taxonomies are to be used for systematic description
of a DHI. Such characterizations enable adaptive
evaluation activities as evaluation parameters can be
sorted or filtered by relevance. However, the design
of this functionality is the focus of future research and
is only mentioned here as a supplement.
2.3 Interoperability as Key Property
Interoperability is basically defined as the ability of
two or more application or information systems to
effectively and efficiently perform tasks together
(Gibbons et al., 2007; HIMSS, 2020; HL7
International, 2021; Zeinali et al., 2016). Following
the socio-technical understanding of HIS research,
interoperability is understood as a construct of
technical and non-technical dimensions (da Silva
Serapião Leal et al., 2019; Kuziemsky and Weber-
Jahnke, 2009). Within this paper, the attribute “socio-
technical“ is provocatively chosen to highlight the
societal dimensions besides technical interpretations.
Socio-technical interoperability is seen as a key
property for a DHIs successful integration as this
general construct comprises the ability of the DHI and
the status quo environment to commonly perform on
four general perspectives. (Figure 2).
These perspectives might be simplified as
follows: Interoperability of a DHI and its target
environment requires a symbiosis according to:
Existing Technical Systems that collaborate
directly or indirectly with a DHI
People who use a DHI or who are affected by its
usage (professionals and patients)
Organizations that manage a DHI’s operation
Regulations that define duties and limits
Figure 2: Construct of socio-technical interoperability.
2.4 ReEIF Model
In 2015, the European Commission's Working Group
“eHealth Network” published the Refined eHealth
European Interoperability Framework (ReEIF)
(eHealth Network, 2015; European Commission et
al., 2013). This unifying framework is primarily
intended to support activities in the context of
interoperability and standardization challenges. It
thereby provides a structuring benefit for
communication and decision-making processes
regarding DH solutions. In this sense, the ReEIF
serves as a consented language for the analysis of DH
solutions. It defines six technical and non-technical
levels of interoperability within the context of DH:
Legal & Regulatory, Policy, Care Process,
Information, Applications and IT Infrastructure.
Figure 3 presents the explanations of each of the
ReEIF's interoperability levels. The eHealth Network
Evaluation object
Digital Health Innovation
Evaluation tool
Evaluation top-criterion
Primary evaluators
Complex target environment
Digital healthcare practice
of Digital Health artifacts
Evaluation sub-criteria
Refined eHealth European
Interoperability Framework
has to be integrated into
structured via
enable adaptive
key property
can be
described by
Evaluation parameters
Tangible activities, tasks
and duties
provide items concretized by
Regulations Organizations
HEALTHINF 2022 - 15th International Conference on Health Informatics
Figure 3: Description of ReEIF Levels by eHealth Network, 2015.
formulates these in light of interoperability between
two or more organizations. Despite some vagueness
for the context of integrating DHI as artifacts into
practice, the explanations still allow for delineation of
relevant topics especially in the light of DHI for
interorganizational healthcare delivery.
From a top-down perspective, the ReEIF is
already part of international recommendations. The
WHO recommends its member states to adopt the
ReEIF within their eHealth strategies and action plans
to support all involved stakeholders on the way from
innovation to implementation (Peterson et al., 2016).
The eStandard initiative (2015-2017) also built on the
ReEIF conceptualization and provided, among other
outputs, the “Interoperability guideline for eHealth
deployment projects” as well as a “Roadmap for a
sustainable and collaborative standard development”
to promote cross-border interoperability, use and
evaluation of domain-specific standards and
beneficial eHealth systems for the European people
(eStandards, 2017a, 2017b; Schulz et al., 2019).
The research community applied the ReEIF in
selected contributions. Scientists from Greece
postulated an adopted framework for digital
transformation of the national health system that
facilitates especially citizen empowerment, health
process alignment and integration of information
technology (Katehakis and Kouroubali, 2019;
Kouroubali and Katehakis, 2019). Dutch researchers
contributed a reference architecture for primary care
that uses the ReEIF to define essential functionalities
which need to be ensured by future digital platforms
or ecosystems (d’Hollosy et al., 2018). Overall, there
are single contributions from science and leading
international institutions that push the adoption of the
ReEIF but, to the author’s knowledge, its transfer to
guidance for DHI practice remains uncomplete.
The presented evaluation concept requires a suitable
operationalization of socio-technical interoperability
regarding the development of an innovative DH
artifact and its integration into modern healthcare
practice. A structuring, deductive Qualitative Content
Analysis (QCA) according to Mayring was conducted
to identify actionable evaluation items for all ReEIF
levels (Mayring, 2014). Details of the research
activities are listed below.
3.1 Literature Selection
Criteria-based evaluations are often underpinned by
literature-guided definitions of the evaluation criteria
and corresponding parameters (Alalwany, 2010).
Socio-technical interoperability was chosen as the
evaluation top criterion and further structured into six
sub-criteria covered by the six levels of the ReEIF. In
order to derive adequate parameters from existing
evaluation concepts, an explorative literature review
was opened. The PubMed, ScienceDirect,
EBSCOHost, and SpringerLink databases were
searched for articles between 2009 and 2019 that
combined "digital health" or related terms [“eHealth”,
Refined eHealth European Interoperability Framework (ReEIF)
Legal &
On this level, compatible legislation and regulatory guidelines define the boundaries for
interoperability across borders, but also within a country or region.
Policy On this level, contracts and agreements between organisations have to be made. The
purpose and value of the collaboration must be set. Trust and responsibilities between the
organisations are formalised on the Policy level. In governance documents the governance
of collaboration is anchored.
Care Process After the organisations have agreed to work together, specific care processes are analysed
and aligned, resulting in integrated care pathways and shared workflows. This level
handles the tracking and management of the workflow processes. The shared workflow
prescribes which information is needed in order to deliver the integrated care.
Information This level represents the functional description of the data model, the data elements
(concepts and possible values) and the linking of these data elements to terminologies that
define the interoperability of the data elements.
Applications On this level, agreements are made about the way import and export of medical
information are handled by the healthcare information systems. The technical specification
of how information is transported is at this level (communication standards). The
information systems must be able to export and import using these communication
standards. Another aspect in this level is the integration and processing of exchanged
information in user-friendly applications.
IT Infrastructure The generic communication and network protocols and standards, the storage, backup, and
the database engines are on this level. It contains all the “generic” interoperability
standards and protocols.
Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach
“mhealth”, “telemedicine”, “telehealth”] (Otto et al.,
2020) with "evaluation" or "assessment" in the title or
abstract. Only articles that discuss evaluation in the
context of integration efforts of DHI into practice
were included. Therefore, articles have to address
DHI diffusion, adoption, implementation or
integration as their contribution objective.
Additionally, only articles that discuss DHI from a
generic point of view have been included. Thus,
articles that focus on single DHI or on DHI for a
specific medical context were excluded from
analysis. 34 contributions were finally selected (see
Appendix 1). These contributions include concepts,
methods, frameworks as well as initiatives or
programs for DHI evaluation.
3.2 Qualitative Content Analysis
The 34 relevant sources were selected as the analysis
set for a deductive, qualitative content analysis
(Mayring, 2014). A structuring approach was
implemented to detect concretizing text passages for
each ReEIF level. In this regard, the definitions listed
in Figure 3 were set as detection criteria. Passages
were assigned to a ReEIF level if they concretize that
definition for the scenario of DHI integration to a
tangible item of action or consideration.
The free online tool QCAmap was used to
perform the coding collaboratively with four research
assistants. After about 10% of the material run, a
check of the coding rules took place. At this point, we
identified the issue that a consistent degree of
abstraction is difficult to apply during coding. We
decided to continue for the moment as we could not
define a suitable rule as well as anchor examples and
added a second analysis iteration afterwards. Here,
we decided to whether a marking was suitable to
provide an actionable parameter for our evaluation
approach (low degree of abstraction) or enriched a
more detailed description of a ReEIF level (medium
degree of abstraction). Both objectives of the
conducted QCA are illustrated in Figure 4.
After completion of the material run,
approximately 4500 markings were set. The coding
results are provided as a raw data set in Appendix 2
to ensure the traceability of detected findings. As
mentioned, the markings showed differences in the
degree of abstraction but could be subsumed into 122
descriptive aspects and 181 potential parameters.
The aspects for the detailed description of the
ReEIF levels are distributed as follows: Legal &
Regulatory 8; Policy 21; Care Process 49;
Information 20; Application 7; IT-Infrastructure 17.
The detected potential evaluation items are
distributed as follows: Legal & Regulatory 13; Policy
66; Care Process 39; Information 11; Application 32;
IT-Infrastructure 20.
Figure 4: Objectives of QCA.
The extent of findings allows only a condensed
presentation of the analysis results at this place. For
each ReEIF level, detailed descriptions are given
below that are taken from those detected text passages
that represent concretizations but no actionable
parameters for evaluation. These detailed
descriptions enrich the existing explanations of
ReEIF given in Figure 3 and adopt them for the
context of integration activities of DHI into
healthcare practice. A complete list of these
descriptive aspects can be found in Appendix 3.
Within Appendix 4, all potential evaluation
parameters are formulated as self-assessment
questions and are offered specially to practice.
As mentioned above, Appendix 2 provides the
raw data set of literature markings which are source
of the following findings. In addition, Table 1
highlights those sources that particularly shaped the
findings for each ReEIF level. Due to their extent, the
reports of PAHO and WHO as well as the study by
Dattakumar et al. caused a majority of the markings
across all ReEIF levels (approximately 60%) and are
therefore listed separately. However, this observation
is purely quantitative in nature and is relativized by
the adjusting subsumption of the second analysis
iteration. The qualitative influence of articles of
smaller length is particularly highlighted for the
contributions of Lau & Price 2017 and Greenhalgh et
al. 2017, as they also push a holistic, socio-technical
approach to increase the adaptation of DHI into
healthcare practice.
ReEIF Levels
Description of
ReEIF Levels
of ReEIF Levels
Degree of
Coded Text
Degree of Applicability for Evaluation Approach
HEALTHINF 2022 - 15th International Conference on Health Informatics
Table 1: Most influential sources of each ReEIF level.
ReEIF Level Ke
Dattakumar et al., 2013; PAHO, 2016;
WHO, 2016; WHO, 2015
Legal &
Kowatsch et al., 2019; Momentum, 2012
Chang, 2015; Khoja et al., 2013;
Kowatsch et al., 2019; Lau and Price,
2017; Scirocco, 2016
Care Process
Chang, 2015; Lau et al., 2017; Lau and
Price, 2017; National Qualit
Forum, 2017
Information HIMSS Analytics, 2017; Lau et al., 2017
Chang, 2015; Kowatsch et al., 2019;
Tamburis et al., 2012
Chang, 2015; Khoja et al., 2013; Van Dyk
et al., 2012
4.1 Legal & Regulatory
This level describes fundamental as well as domain-
specific public regulations and laws at regional,
national or international level with regard to certain
rights and values, esp. equity, equality, justice,
security, liability, privacy, confidentiality and ethics.
They regulate and ensure, among other things,
personal rights, medical procedural rules, public
structures of healthcare delivery, and data processing
conditions. To ensure interoperability in this
dimension, DH innovators can take action by
identifying relevant regulations and guidelines,
ensuring compliance to them, and/or requesting of
specific consulting services and advisory.
Ensuring interoperability on the “Legal &
Regulatory” level is mostly understood from the
innovator’s perspective of a specific DHI as being
aware of and comply with the current and/or future
legal circumstances. Unlinked from a specific DHI
project, there might be also opportunities for
innovators to participate in design or reformation
processes on a legal level depending on an
innovator’s influence, position and possibilities to
invest the required amount of time.
4.2 Policy
This interoperability level includes basic as well as
specific policies or guidelines between organizations,
esp. between the organizational background of
innovators and contractually involved parties
(Clinicians, IT businesses, funding agencies and
individuals (e.g., Patient) and their compliance
(governance). Also, intraorganizational policies (for
instance of single hospitals) are of interest for
innovators, as they eventually ensure or inhibit the
adoption of a DHI within clinical practice. Those
policies need to be clarified, negotiated, documented,
communicated and fulfilled by innovators and
appropriate parties regarding aspects like:
organizational compatibility; liability; sustainability;
safety, security and privacy; competencies; quality
assurance & management; value propositions;
business principles; technical support; operations and
maintenance; working collaborations and
cooperation; education and training as well as
licenses and accreditations.
Ensuring interoperability on the “Policy” level
requires, trivially speaking, the negotiation and
confirmation of agreements between all involved
stakeholders of a DHI. Depending on the specific
DHI, its usage context and its innovational degree,
this task becomes more or less complex. As policy
activities refer to a broad variety of aspects that are
also part of other interoperability levels and due to the
unknown balance between compatibility to existing
policies and the need for new agreements, the
required efforts shall be rather over- than
4.3 Care Process
This interoperability level addresses the alignment or
reorganization of: workflows of care delivery;
business and administrative processes; healthcare
models and programs; care plans and pathways as
well as personal interaction and communication. This
includes (re)definitions and statements about:
cooperation; coordination; competences and
responsibilities; liability of practice as well as error
prevention and risk descriptions. Among others, such
alignments or reorganizations aim to ensure: quality
of care; accuracy and disease specificity; continuity;
validity; safety and security; usability, user-
friendliness, acceptance and satisfaction of patients
and professionals as well as customizability and
individualization. Innovators may support these
efforts by confirmation of effects like: clinical
effectiveness and outcomes; patient-related
outcomes; efficiency and/or quality benefits; process
measures; treatment or medication adherence. They
therefore should consider or provide: comprehensive
description of DHI functionality; guidelines and
standards of health practice; deviation in regular
practice; patient engagement, user empowerment or
education initiatives.
Summing up, the demand for interoperability at
the care process level entails a large number of
aspects that innovators should address. Based on the
Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach
concrete needs from care practice, the core process
for which a DHI offers a solution must be analyzed
intensively. Of particular interest are the questions:
How do apply which users the DHI and which people
are directly affected by it and how? How does the
DHI change the existing core process? Furthermore,
dependencies or the influence on accompanying care,
administration or business processes must be taken
into account. Innovators need to balance whether a
DHI should be designed to be compatible with
established processes or the design of a DHI and its
value proposition requires changes of the status quo.
4.4 Information
The “Information” level comprises aspects of
semantic and syntactic interoperability, esp. data
types, formats and structures; data flows; and the use
of terminologies and standards. Considered data and
information sets typically consist of general health
information, clinical data, information about
decisions, system-generated data as well as
timestamps or log files. Innovators may generally
align with existing standards or participate in
standardization initiatives to ensure: accuracy;
comparability; completeness; comprehensiveness;
consistency; relevance and value; confidentiality;
reliability as well as integrity.
Ensuring interoperability on the “Information”
level requires on the innovator’s site the balancing
task of: identification and alignment with data
models, structures and formats that are determined by
the target environment of a DHI; harmonization of
those (eventually heterogenous) compatibility
requirements with own development; identification
and re-use of consented interoperability standards
provided by relevant institutions, e.g., HL7 or IHE;
and the promotion of standard adoption or initiating
standardization processes of new specifications.
Reflecting these subtasks, interoperability on
“information” level seems to be primarily ensured by
compatibility activities especially towards existing
semantic and syntactic standards. Nevertheless, a
DHI that offers a new solution for an existing problem
will probably hit a spot where the state of practice
does not offer a health information standard. Here,
innovators are able to fill this gap with self-defined
specifications and might contribute their
achievements to the synergetic community.
4.5 Application
The “Application” level comprises agreements and
their realization according to interconnectivity of
distinguished (information and) application systems,
esp. in terms of: interconnection services and data
exchange; use of communication standards and
unified terminologies to ensure robustness of
technical interfaces, sustainability as well as usability
of technical interfaces. This generally technical
dominated interoperability level does also include
human-centered aspects like end-user satisfaction and
user acceptance but with a focus on the
interconnection of a DHI with other application
systems. While the “Care Process” level addresses
usability of a DHI itself – simplified as its use without
involvement of any other technical system this level
considers usability aspects of DHI within an
interconnected, synergetic HIS landscape. For
instance, a professional documentation tool for a
specific indication can be autonomously usable,
intuitive and, thus, valuable but if data exchange with
central Electronic Health Record systems is not
ensured then double documentation might occur and
will decrease user acceptance.
Increasing interoperability on the “Application”
level requires knowledge about (potentially)
mandatory communication scenarios of a DHI with
existing or future application systems. Definition and
prioritization of these scenarios are key tasks for
innovators before technical interface solutions can be
derived and realized. Thereby, innovators are not
exclusively responsible on the required realization
efforts as changes of the target environment could
also foster interoperability, e.g., by supporting
communication standards like HL7 FHIR.
4.6 IT-Infrastructure
Interoperability on “IT-Infrastructure” level includes
considerations of specific properties, e.g.,
availability, performance, capacity, scalability,
reliability, stability as well as safety and security of
infrastructural components, like basic infrastructure
of electricity, physical and mobile communication
networks, required hardware, distributed server
architectures and physical databases as well as
storage units. Activities that may be considered to
fulfill interoperability on this level are, among others,
the use of technical infrastructure standards and
protocols, the establishment of infrastructural data
protection measures and validation mechanisms as
well as maintenance and failure prevention activities.
Depending on the specific characteristics of a DHI,
innovators need to consider infrastructural aspects on
international, national, regional or local level. As
infrastructures do not change rapidly, innovators shall
search for a DHI design that is compatible with existing
HEALTHINF 2022 - 15th International Conference on Health Informatics
infrastructures. Thus, specifying the access to required
server structures or networks, clarifying how
continuity of operations can be ensured and
implementing mechanisms to prevent or handle
potential failure as well as IT attacks are main tasks.
5.1 Relevance
The extent of aspects for the given descriptions of the
ReEIF levels as well as for potential parameters for
the presented evaluation approach motivates the
relevance question for each item. At the highest level
of abstraction (ReEIF levels), no differentiation of
relevance can be stated in general as neglecting each
level makes the failure of a DHI integration likely.
Although the detailed descriptions are formulated
generically, the characteristics of a specific DHI, its
usage context, and the DHI project’s organizational
circumstances may assign a single aspect more or less
relevant. These three influencing factors require an
individual assessment of relevance at the level of the
formulated parameters, which cannot be provided in
a blanket manner within this paper. In order to take
this sensitivity into account, the presented approach
comprises a selection of relevant parameters for a
concrete evaluation instance based on a previous DHI
characterization (Figure 1).
5.2 Critique on ReEIF
Generally, the ReEIF suited the task of systemizing
interoperability from a socio-technical HIS point of
view. Nevertheless, from the author’s perspective,
two themes could be assigned within the ReEIF but
do not match a level’s intention perfectly and, thus,
should be highlighted more explicitly.
As the ReEIF is originally focused on
interoperability between organization, the usage of a
DHI or the user itself is not prominently represented.
Especially findings regarding usability have to be
assigned to theCare Process level, as a DHI
generally intends to support healthcare activities, or
to theApplication level, as data exchange within
interconnected HIS components might be crucial for
usability to ensure continuity of information flow.
Considering the extent of the “Care Process” level
presented here, it might be valuable to distinguish
user-centered topics (“Use of DHI”) from process-
centered topics. Other authors promote a similar
separation of a DHI’s usage without any
communication scenario to other technical systems
from the alignment and continuity of process
landscapes in a target environment of connected HIS
(van Mens et al., 2020).
Another vagueness occurs while placing aspects
about required data for a DHI’s functionality into the
right ReEIF levels. Especially in the light of data-
centered DHI and the progress of AI application in
healthcare, valid access to required data sources
becomes a central topic for innovators. Thereby,
“required data” rather combines all three technical
ReEIF levels than perfectly fit into a single one. Even
though the interplay of syntax and semantics
(Information), technical system interfaces and
communication standards (Application) as well as
appropriate connection to networks, server
architectures and databases is implicated, it shall be
highlighted for future data-centered DHI.
5.3 Dominance in Interoperability
Interoperability, in its technical and non-technical
manner, is a property that targets two or more systems
as a unit, not as single parts. Ensuring interoperability
in the context of this paper depends therefore on the
constitution of both the DHI (as an artifact) and the
target environment. Reflecting the findings against
this background, it might be valuable for innovators
to differentiate the way of how they should act to
ensure interoperability: by alignment and providing
compatibility or by declaration of requirements on the
target environment’s site. Simplified, when a DHI
takes its place within an existing target environment,
three principles of dominance in interoperability
might occur (Figure 5): I. Dominance of DHI the
DHI stimulates changes in the target environment
which ensure interoperability; II. Dominance of target
environment the target environment declares
mandatory requirements that have to be aligned with;
and III. Interdependent adjustments – interoperability
ensured by coordinated changes on both sites.
Figure 5: Principles of Dominance in Interoperability.
Digital Health
I. DHI stimulates changes in target environment
II. Target environment declares mandatory requirements
III. Interdependent adjustments
Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach
The findings as well as the dependency on a
specific DHI indicate that these principles should not
be seen as absolute categories. Rather, it shall be
understood as a continuum within interoperability
efforts can be assessed from an innovator’s
perspective. Regarding the findings, an innovators
opportunity to ensure interoperability on “Legal &
Regulatory” aspects as well as in existing “IT-
infrastructures” tend to compatibility activities to the
status quo (II). On the other hand, DHI that provide
new solutions for healthcare practice and new
beneficial value propositions will influence the way
how healthcare is delivered and how “Care
Processes” are conducted (I). “Policy”, “Information”
and “Application” level require a balance of activities
striving to changes within established structures and
the alignment with mandatory conditions (III).
5.4 Limitations
The scope and quality of the presented results have to
be assessed under consideration of some limitations.
In particular, inherent constraints on objectivity due
to the qualitative, interpretive research approach
result in three aspects that are stated here and
motivate future research.
Degree of Abstraction. The chosen differentiation of
two types of analytical findings (detailed descriptive
aspects vs. potential evaluation parameters) as well as
the high rate of subsumption (4500 initial markings to
300 finally used descriptive aspects and evaluation
items) point to one issue: the definition and
application of a common degree of abstraction as a
coding guideline. Despite the fact that the coding
rules were adjusted for comprehension within the
material run, a relatively high variance had to be
handled during the interpretation cycles. The decision
whether an "actionable" evaluation item was detected
could not be made for all markings without doubt. In
front of this circumstance, the two types of findings
were defined. The created detailed descriptions of the
ReEIF levels could be used as a starting point for
argumentative-deductive derivation of further
actionable evaluation items to improve completeness
and fit for different DHI types.
Suitability. One of the guiding motivations of this
work is making knowledge about DHI integration
into healthcare practice accessible and actionable for
innovators. For this purpose, no restrictions were
made with regard to DHI types, neither in the design
of the evaluation approach nor during the
parameterization of socio-technical interoperability.
No artifact classes were explicitly excluded or
prioritized. The scope of detected aspects achieved in
this way was purchased with an initial lack of general
fit of the individual item. For example, the question
about confirmation of positive effects on patient self-
management (CP-13) is irrelevant for DHI without
patient involvement. Other items, such as the question
about mechanisms to prevent system overload (ITI-
02), may have universal relevance. Against this
background, the detected items are to be assessed in
terms of fit for different DHI types along established
taxonomies in order to correspond to the adaptive
character of the underlying evaluation approach
(Figure 1).
Fuzziness. The method-related limitation of
objectivity as well as the interrelation of detected
aspects causes a certain fuzziness between separately
listed evaluation items or gives the impression of
redundancy in certain cases. For example, the items
CP-10 ("Is continuity of care ensured?") and CP-34
("Are seamless transitions between tasks of care
ensured?") differ only slightly in their different
perspectives (patient-centered vs. professional-
centered) on continuous, trouble-free care processes.
Despite this limitation, the results presented benefit
from the diversity of perspectives gained as well as
from the breadth of detected aspects. Further
investigations could contribute to an improved
distinction of the evaluation items, for example, by
using a matrix structure.
Additionally, the limited topicality of this work
has also to be named. This analysis started in 2020
and included only articles published until 2019. Due
to the Covid-19-Pandemie and other circumstances,
conduction, documentation and publishing of this
work were delayed. Therefore, chosen literature data
bases have been checked for additional resources, but
the extent of articles matching the inclusion criteria is
scarce. Nevertheless, three articles are mentioned
here for completeness that generally confirm
motivation and presented findings (Bashi et al., 2020;
Guo et al., 2020; Villumsen et al., 2020). Bashi et al.
reviewed science articles about the development of
DH frameworks for chronic healthcare scenarios and
recommend the re-use of frameworks for evidence-
based DHI processes including evaluation activities.
Guo et al. see the need of more pragmatic DHI and
evaluation approaches to face the “no evidence, no
implementation no implementation, no evidence”
paradox in DH. They highlight the awareness of
socio-technical requirements faced by different
stakeholders and call for new approaches to facilitate
responsible growth of the DH domain. Villumsen et
HEALTHINF 2022 - 15th International Conference on Health Informatics
al. provide “an overview of the predominant
approaches and methodological recommendations to
national and regional monitoring and evaluation of
eHealth”. Even though their main perspective
addresses policy makers and appropriate initiatives,
they recommend continuous, transparent monitoring
and evaluation to facilitate learnings and
implementation progress.
5.6 Further Research
The given results are currently being accompanied by
an ongoing expert study. In 1-to-1 interview sessions,
the experiences of experts from various professions
(science, medicine, management and IT) are being
collected in order to investigate the following
questions, among other: How should differences in
ReEIF levels in terms of relevance and criticality be
assessed for definable DHI types? How shall
interdependencies between ReEIF levels as well as
between items taken into account? How can
evaluation parameters be linked to action items and
their termination within typical innovation phases?
This research paper addresses the challenge of
innovators to fulfill the complex, interdisciplinary
web of requirements for a successful integration of a
DHI into modern healthcare practice. It presents an
evaluation approach based on the key property of
interoperability in a socio-technical manner. Along
six interoperability levels defined by the ReEIF, this
paper explores potential evaluation parameters for a
self-assessment tool and provide detailed descriptions
of ReEIF levels. While the organizational intended
ReEIF generally suits the scenario of integrating a
DHI into healthcare practice, the framework could
benefit from little adjustments by a sound
distinguishment of usability facets and the
consideration of dominance in interoperability. The
findings enrich both further research and practice to
support innovators handling the complexity of
domain specific target environments and, thus, to
increase successful integration rates of future DHI.
This research is part of the EFRE-funded research
project “Häusliche Gesundheitsstation”. I thank all
colleagues, partners and supporters who made this
research possible.
Alalwany, H., 2010. Cross disciplinary evaluation
framework for e-health services (PhD Thesis). Brunel
University Brunel Business School PhD Theses.
Bashi, N., Fatehi, F., Mosadeghi-Nik, M., Askari, M.S.,
Karunanithi, M., 2020. Digital health interventions for
chronic diseases: a scoping review of evaluation
frameworks. BMJ Health Care Inform. 27, e100066.
Chang, H., 2015. Evaluation Framework for Telemedicine
Using the Logical Framework Approach and a
Fishbone Diagram. Healthc. Inform. Res. 21, 230.
d’Hollosy, W.O.N.–, van Velsen, L., Henket, A., Hermens,
H., 2018. An Interoperable eHealth Reference
Architecture for Primary Care, in: 2018 IEEE
Symposium on Computers and Communications
(ISCC). Presented at the 2018 IEEE Symposium on
Computers and Communications (ISCC), pp. 01090–
01095. https://doi.org/10.1109/ISCC.2018.8538576
da Silva Serapião Leal, G., Guédria, W., Panetto, H., 2019.
Interoperability assessment: A systematic literature
review. Comput. Ind. 106, 111–132. https://doi.org/
Dattakumar, A., Gray, K., Jury, S., Biggs, B.-A., Maeder,
A., Noble, D., Borda, A., Schulz, T., Gasko, H., 2013.
A unified approach for the evaluation of telehealth
implementations in Australia.
eHealth Network, 2015. Refined eHealth European
Interoperability Framework. European Commission,
eStandards, 2017a. Interoperability guideline for eHealth
deployment projects.
eStandards, 2017b. Roadmap for a sustainable and
collaborative standard development: recommendations
for a globally competitive Europe.
European Commision, 2019. MAFEIP Monitoring and
Assessment Framework for the European Innovation
Partnership on Active and Healthy Ageing [WWW
Document]. URL https://www.mafeip.eu/
European Commission, Directorate-General for the
Information Society and Media, Deloitte & Touche,
2013. eHealth European Interoperability Framework
study report. Publications Office, Luxembourg.
Gibbons, P., Phd, N.A., Flewelling, T., Jepsen, T., Md,
D.K., Larson, J., Ritter, J., Md, M.R., Selover, S.,
Stanford, J., 2007. Coming to Terms: Scoping
Interoperability for Health Care Health Level Seven
EHR Interoperability Work Group.
Guo, C., Ashrafian, H., Ghafur, S., Fontana, G., Gardner,
C., Prime, M., 2020. Challenges for the evaluation of
digital health solutions—A call for innovative evidence
generation approaches. NPJ Digit. Med. 3, 110.
Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach
HIMSS, 2020. What is Interoperability in Healthcare?
[WWW Document]. URL https://www.himss.org/
HIMSS Analytics, 2017. Electronic Medical Record
Adoption Model: EMRAM [WWW Document]. URL
HL7 International, 2021. General Information about HL7
[WWW Document]. URL http://www.hl7.org/about/
Hobeck, R., Schlieter, H., Scheplitz, T., 2021. Overcoming
Diffusion Barriers of Digital Health Innovations
Conception of an Assessment Method, in: Proceedings
of the 54th Hawaii International Conference on System
Sciences. p. 3654.
Iyawa, G.E., Herselman, M., Botha, A., 2016. Digital
Health Innovation Ecosystems: From Systematic
Literature Review to Conceptual Framework. Procedia
Comput. Sci. 100, 244–252. https://doi.org/10.1016/
Katehakis, D.G., Kouroubali, A., 2019. A Framework for
eHealth Interoperability Management. J. Strateg. Innov.
Sustain. 14, 51–61.
Khoja, S., Durrani, H., Scott, R.E., Sajwani, A., Piryani, U.,
2013. Conceptual Framework for Development of
Comprehensive e-Health Evaluation Tool. Telemed. E-
Health 19, 48–53. https://doi.org/10.1089/tmj.20
Kouroubali, A., Katehakis, D.G., 2019. The new European
interoperability framework as a facilitator of digital
transformation for citizen empowerment. J. Biomed.
Inform. 94, 103166. https://doi.org/10.1016/j.jbi.20
Kowatsch, T., Otto, L., Harperink, S., Cotti, A., Schlieter,
H., 2019. A design and evaluation framework for digital
health interventions. It - Inf. Technol. 61, 253–263.
Kuziemsky, C.E., Weber-Jahnke, J.H., 2009. An
eBusiness-based Framework for eHealth
Interoperability. J. Emerg. Technol. Web Intell. 1, 129–
136. https://doi.org/10.4304/jetwi.1.2.129-136
Lau, F., Hagens, Simon, Zelmer, Jennifer, 2017. Benefits
Evaluation Framework, in: Handbook of EHealth
Evaluation: An Evidence-Based Approach [Internet].
University of Victoria.
Lau, F., Price, M., 2017. Clinical adoption framework, in:
Handbook of EHealth Evaluation: An Evidence-Based
Approach [Internet]. University of Victoria.
Mayring, P., 2014. Qualitative content analysis: theoretical
foundation, basic procedures and software solution.
Momentum, 2012. Momentum European Momentum for
Mainstreaming Telemedicine Deployment in Daily
Practice [WWW Document]. URL http://www.teleme
National Quality Forum, 2017. Creating a Framework to
Support Measure Development for Telehealth 1–53.
Otto, L., 2019. Implementing and Scaling up Telemedicine
Initiatives: Beyond User-Centeredness, in: 2019
IEEE/ACS 16th International Conference on Computer
Systems and Applications (AICCSA). Presented at the
2019 IEEE/ACS 16th International Conference on
Computer Systems and Applications (AICCSA), IEEE,
Abu Dhabi, United Arab Emirates, pp. 1–6.
Otto, L., Harst, L., 2019. Investigating Barriers for the
Implementation of Telemedicine Initiatives: A
Systematic Review of Reviews, in: AMCIS 2019
Proceedings. Presented at the AMCIS 2019, Cancun.
Otto, L., Harst, L., Schlieter, H., Wollschlaeger, B., Richter,
P., Timpel, P., 2018. Towards a Unified Understanding
of eHealth and Related Terms - Proposal of a
Consolidated Terminological Basis. Proc. 11th Int. Jt.
Conf. Biomed. Eng. Syst. Technol. 2, 533–539.
Otto, L., Harst, L., Timpel, P., Wollschlaeger, B., Richter,
P., Schlieter, H., 2020. Defining and Delimitating
Telemedicine and Related Terms-An Ontology-Based
Classification. Stud. Health Technol. Inform. 268, 113–
PAHO, 2016. Framework for the Implementation of a
Telemedicine Service. Pan American Health
Organization, Washington DC.
Peterson, C.B., Hamilton, C., Hasvold, P., 2016. From
innovation to implementation: eHealth in the WHO
European region. WHO Regional Office for Europe,
Copenhagen, Denmark.
Prim, A.L., Filho, L.S., Zamur, G.A.C., Di Serio, L.C.,
2017. The relationship between national culture
dimensions and degree of innovation. Int. J. Innov.
Manag. 21, 1730001. https://doi.org/10.1142/S13639
Schulz, S., Stegwee, R., Chronaki, C., 2019. Standards in
healthcare data. Fundam. Clin. Data Sci. 19–36.
Scirocco, 2016. Scirocco Maturity Model for Integrated
Care [WWW Document]. URL https://www.scirocco-
Tamburis, O., Mangia, M., Contenti, M., Mercurio, G.,
Rossi Mori, A., 2012. The LITIS conceptual
framework: measuring eHealth readiness and adoption
dynamics across the Healthcare Organizations. Health
Technol. 2, 97–112. https://doi.org/10.1007/s12553-
Van Dyk, L., Schutte, C.S., Fortuin, J., 2012. Development
of a Maturity Model for Telemedicine. South Afr. J.
Ind. Eng. 23 (2), 61–72.
van Mens, H.J.T., Duijm, R.D., Nienhuis, R., de Keizer,
N.F., Cornet, R., 2020. Towards an Adoption
Framework for Patient Access to Electronic Health
Records: Systematic Literature Mapping Study. JMIR
Med. Inform. 8, e15150. https://doi.org/10.2196/15150
Villumsen, S., Adler-Milstein, J., Nøhr, C., 2020. National
monitoring and evaluation of eHealth: a scoping
review. JAMIA Open 3, 132–140. https://doi.org/
WHO, 2016. Monitoring and evaluating digital health
interventions: A practical guide to conducting research
and assessment. World Health Organization.
HEALTHINF 2022 - 15th International Conference on Health Informatics
WHO, 2015. The MAPS Toolkit: mHealth Assessment and
Planning for Scale 106.
Zeinali, N., Asosheh, A., Setareh, S., 2016. The conceptual
model to solve the problem of interoperability in health
information systems, in: 2016 8th International
Symposium on Telecommunications (IST). Presented
at the 2016 8th International Symposium on
Telecommunications (IST), IEEE, Tehran, Iran, pp.
684–689. https://doi.org/10.1109/ISTEL.2016.78819
All appendices are available via the following link:
Ensuring Socio-technical Interoperability in Digital Health Innovation Processes: An Evaluation Approach