Methods in Biomedical Informatics

Online Qualitative Methods Training Module

Alex Cheng, Biomedical Informatics, working with
Kim Unertl, Assistant Professor of Biomedical Informatics

Abstract

Despite its importance in biomedical informatics, the availability of online educational materials for qualitative methods is low compared to online courses in data science. We developed a highly distributable video observation teaching module for clinical process improvement and validated its use in a class of biomedical informatics graduate students. Our results show that the module was successful in improving students’ observation skill and confidence.

Introduction

Education in qualitative methods has been repeated cited as essential for biomedical informatics training1-3. Online learning in higher education is also growing to increase access of materials to more students. While there are many online courses in statistics and data science, there are few in qualitative methods that offer both instruction and practical experience. The field of biomedical informatics needs learning modules in qualitative methods that are rigorously evaluated4. One of the challenges with online training in qualitative methods such as process observation, interviews, or focus groups5 is that it requires physical presence in healthcare settings. In this study, we develop and evaluate an online learning module for qualitative methods in process improvement.

Methods

The objective of this module was to teach biomedical informatics graduate students to make observations in clinical areas with a focus on process improvement. We tested the module in a class segment which was part of a larger graduate-level biomedical informatics methods course. The module and instructor feedback for this portion of the course was delivered online. Our module consisted of two simulated video observation exercises from an infusion preparation process at a pharmacy. Students also performed an in-person observation in a clinical area as part of the course between the two video exercises. There were two sources of data for evaluating the module. One source was a survey that asked students about their educational background, experience working and observing in clinical areas, and attitudes toward qualitative and quantitative analysis. We conducted this survey before and after module completion to observe whether students’ opinions changed after completing the module. The second source of data was the notes from the observation exercises. When completing the assignment, students took notes on the pharmacy process with a spreadsheet with automatic timestamps.

The instructors coded each line of notes as describing tasks, environment, technology, communication, or anomalies. We evaluated each student’s notes by measuring the deviation from the instructor average in each category as a percentage of the instructors’ average. We provided written feedback to each student as well as data showing where their notes differed from the instructors’. We then assigned a second video observation in the same pharmacy but with an altered process and compared the deviation between the first and second assignments.

Figure 2: Course Module Progression. * denotes data collection

Results and Discussion

There were six students in the biomedical informatics class where we tested our module. All six consented to participating in the study. All of the students had bachelor’s level degrees while three also had doctorate level degrees.

Figure 3: Students participating in the study by highest level of education

We observed that students with more clinical experience had less deviation from instructors across categories than students with less clinical experience in the first assignment. Between the first and second assignment, five out of six students had more lines of observations, more total words, and less line deviation from the instructors. All six students had less deviation in word count in the second assignment. There was no reduction in the dispersion index across categories in the second assignment. This result was expected since the dispersion index for the instructors actually increased in the second assignment indicating that the distribution of noteworthy events was more lopsided across categories in the second video than in the first.

Figure 4: Average deviation in lines of observations before and after the module.

Table 1: Summary of change in various performance metrics

Figure 5: Example of feedback given to a student

From the pre and post module attitudes surveys we observed that students had mixed opinions on whether or not the online video helped improve their confidence in making conclusions in process improvement and their comfort in observing in clinical areas. Out of a score of 100, students rated the helpfulness of the video module at an average score of 55. However, all six students felt more confident in making conclusions about process improvement after the module, suggesting that the learning module as a whole was effective. Only two students believed that qualitative methods were more important after the module. However, five students believed that qualitative methods were more important than quantitative methods after the module.

Figure 6: Student response to whether the simulated video observation exercises helped improve observational note taking (out of 100)

Figure 7: Change in attitudes toward qualitative versus quantitative methods in biomedical informatics before and after module.

Figure 8: Change in self-assessed confidence in making process decisions and comfort in observing in clinical areas.

One limitation of this study was that the class size was small and may not have been representative of a large scale implementation of the learning module. Additionally, the method of measuring deviation assumes that the instructors’ distribution of observation categories is ideal, based on the extensive experience of instructors. Furthermore, attitudes surveys may have had some response bias with students answering questions based on what they believed the instructors wanted to hear. Future work may include more detailed analysis into how the content of observations and conclusions change for students before and after completing the learning module. This evaluation framework could also be useful for assessing student competency and attitudes in learning other methods in biomedical informatics.

Conclusions

We developed a highly distributable online teaching module for clinical process observation. The module was effective in improving students’ observation skills and confidence in making process improvement conclusions.

References

1. Huang QR. Competencies for graduate curricula in health, medical and biomedical informatics: a framework. Health Informatics Journal. 2007 Jun 1;13(2):89–103.

2. Kulikowski CA, Shortliffe EH, Currie LM, Elkin PL, Hunter LE, Johnson TR, et al. AMIA Board white paper: definition of biomedical informatics and specification of core competencies for graduate education in the discipline. Journal of the American Medical Informatics Association. 2012 Nov;19(6):931–8.

3. Mantas J, Ammenwerth E, Demiris G, Hasman A, Haux R, Hersh W, et al. Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics: First Revision. Methods of Information in Medicine. 2010 Jan 7;49(2):105–20.

4. Patel VL, Yoskowitz NA, Arocha JF, Shortliffe EH. Cognitive and learning sciences in biomedical and health instructional design: A review with lessons for biomedical informatics education. Journal of Biomedical Informatics. 2009 Feb;42(1):176–97.

5. Meyer J. Qualitative research in health care: Using qualitative methods in health related action research. BMJ: British Medical Journal. 2000;320(7228):178.