Evaluate digital patient experience
This section will increase your chances of successfully improving digital patient experiences and convincing your clients, stakeholders, and end-users to accept your design outcomes.
*The information of this page is based on:
Wang, T., Giunti, G., Melles, M., & Goossens, R. (2022). Design-Relevant Factors Affecting the Patient Experience in Digital Health: Preliminary Results of an Umbrella Systematic Review. Studies in health technology and informatics, 290, 862–866. https://doi.org/10.3233/SHTI220202
Wang, T., Giunti, G., Melles, M., & Goossens, R. (2022). Digital patient experience: umbrella systematic review. Journal of medical Internet research, 24(8), e37952.
Why to measure?
Clarify evaluation objectives
Why to measure? Think about who may care about your evaluation results. You can choose between framing your own evaluation purpose or selecting from the below five common ones.
To broaden understanding, guide evaluation research and practice (Researchers)
To improve design, development, and implementation (Designers)
To achieve evidence-based clinical usage, and increase adoption and uptake (End-users)
To drive ongoing investment (Investors)
To inform health policy practice (Policy-makers)
When to measure?
Select evaluation timing
Now that you know why and have decided to make an evaluation plan, what is the appropriate stage for you to evaluate your digital health solution?
You can select one from the table below.
Step 1
Product-related timing
Aligned with your digital health design, development, and implementation process.
After design & development
Assess effectiveness, whether your digital health achieves your intended results in the real world after the design or development.
Implementation
Assess the uptake, institutionalization, and sustainability of your digital health after the implementation stage, in terms of policies and practices in the real world.
Step 2
Interaction-related timing
Prior to, during or following patients' touches with your digital health.
Before Interaction
Perform a pre-test before individual patients use your digital health to assess their initial health status or anticipated perception of your digital health.
During Interaction
Conduct an evaluation when patients use your digital health to monitor their real-time feedback and reactions.
After Interaction
Perform a post-test right after or a long time after patients use your digital health to assess their experiences and changes.
Step 3
Patient-related timing
Collect qualitative or quantitative data on how patients experience your digital health.
Immediate
You can collect “real-time” data on patients’ experiences, such as short-term patient satisfaction, during or immediately after their usage of your digital health.
Delayed
You can obtain more substantial responses, such as long-term quality of life, after patients have used your digital health for a long period of time.
Momentary
You can collect transient information, such as emotions lasting only seconds, from patients at a specific moment.
Continuous
You can gather sustained feedback, such as moods lasting for hours, from patients at different points along the care pathway.
What to measure?
Choose evaluation criteria
What to measure? Combining your design and evaluation objectives, you can use either influencing factors or evaluation indicators as your evaluation criteria.
The influencing factors are more appropriate for formative assessment during the design and development process.
The evaluation indicators are more suitable for summative assessments during and after the implementation process.
Evaluation indicators
All
Intervention Outputs
Patient Outcomes
Healthcare System Impact
How to measure?
Determine evaluation approaches
Evaluation approaches refer to three aspects: study design, data collection, and data analysis.
Study designs
Study designs are the set of methods and procedures used to collect and analyze data in a study. Study designs often affect the strength of your evidence and determine your data collection and analysis methods. As a digital health designer, you are expected to know more about it to inform your conversations with healthcare experts.
We summarized these study designs in below. Please read our publication. If you want to know more about these study designs.
Mode of inquiry
Qualitative research (Phenomenology, and Ethnography).
Mixed-methods research and multiple qualitative methods
Nature of the investigation
Experimental studies (Randomized controlled trials, and Non-randomized trials.)
Descriptive studies (Case reports, Case series, Cross-sectional).
Analytical studies (Case-control, Cohort studies.)
Number of contacts
Before-and-after
Reference period
Prospective designs
Retrospective study designs
The design study
User research study
Participatory design or contextual design
Design sessions
Qualitative methods are more appropriate for collecting in-depth experience data for a smaller sample size in the early digital health design and development stages.
Quantitative methods are more suitable for investigating experience data on a large scale or comparing it with others during or after implementation stages. Behavioral data, such as dropout rates, may provide stronger evidence than opinion data, such as perceived value. Examples of data collection methods
Examples of data collection methods
Observations
Log data
Usability testing
Think-aloud
Examples of data collection instruments
Patient Activation Measure
Patient Health Questionnaire
Beck Depression Inventory
Data analysis
The type of your collected data often determine the methods that you used for data analysis.
Qualitative data consists of text, images, or videos instead of numbers. You can use content analysis, thematic analysis, and discourse analysis to analyze this type of data.
Quantitative data is based on numbers. You can use simple math or more advanced statistical analysis to discover commonalities or patterns in the data.
Examples of data analysis methods
Grounded theory
Heuristic analysis
Cost analysis
Failure analysis
Inductive analysis
Deductive analysis
After measurement
Present evaluation results
Evaluation reports
Why did you do the evaluation?
What did you expect?
How did you do (what, when, where, who)?
What did you find?
What do the findings mean?
Who can use what findings, and how?
Theoretical or practical implications
Researchers: what research questions need to be considered for the digital patient experience?
Designers: what has to be improved, and how, in the digital patient experience?
End-users: shall we start or continue using which digital health interventions and why
Investors: shall we start or continue investing in which digital health interventions and why？
Policymakers: where can we find the digital-health-interventions-related opportunities for informing future policy?
