Disaster Medicine Online: evaluation of an online, modular, interactive, asynchronous curriculum
Adam Lund, MD; Kenneth Lam, MD; Paul Parks, MD
Disaster Medicine Subsection, Division of Emergency Medicine, University of Alberta, Edmonton, Alta.
Canada has no formal training program in disaster medicine for health care professionals. The University of Alberta's Division of Emergency Medicine has developed a means to fill the gap. Disaster Medicine Online (DMO) is an Internet-based, interactive, facilitator-guided distance-learning course on the fundamentals of disaster medicine. The 3-week pilot of DMO was offered in March 2002 and taken by a multidisciplinary group of 22 health care professionals, including resident and attending physicians, paramedics and nurses. Evaluation of the learning materials and educational methodology by experts and learners demonstrated a high degree of satisfaction with the Web interface, site usability, lesson content and format, and the interactive components of the online course. Learners reported spending a mean of 11.2 hours (range = 5-20) over the 3-week course period. Twenty of 22 learners completed the final assignment, and all 20 were successful in passing the course. Overall, 95% of learners said they would pursue another module if offered, and 100% would recommend DMO to their colleagues. DMO is a viable option for health care professionals who would like to pursue continuing medical education in this area without having to take time out of their personal and professional lives to travel to a face-to-face, traditional educational program.
Le Canada n'a pas de programme officiel de formation en médecine de catastrophe pour les professionnels de la santé. La division de médecine d'urgence de l'Université de l'Alberta a créé un moyen de combler cette lacune. Disaster Medicine Online (Médecine de catastrophe en ligne) (DMO) est un cours interactif à distance par Internet portant sur les principes de base de la médecine de catastrophe. Le pilote de DMO d'une durée de trois semaines a été offert en mars 2002 et a été suivi par un groupe multidisciplinaire de 22 professionnels de la santé, notamment des résidents et des médecins traitants, des techniciens ambulanciers et des infirmières. L'évaluation du matériel d'apprentissage et de la méthodologie par des experts et par les apprentis indiquait un haut degré de satisfaction quant à l'interface du site web, à la facilité d'utilisation du site, au contenu et à la présentation des leçons et aux composantes interactives du cours en ligne. Les apprentis dirent avoir consacré en moyenne 11,2 heures (éventail = 5-20) au cours pendant la période de trois semaines. Vingt des 22 apprentis complétèrent le dernier devoir, et les 20 réussirent le cours. Globalement, 95 % des apprentis dirent qu'ils prendraient un autre module s'il était offert et 100 % d'entre eux recommanderaient Disaster Medicine Online à leurs collègues. DMO est une option viable pour les professionnels de la santé qui souhaitent poursuivre leur éducation médicale continue dans ce domaine sans avoir à perturber leur vie privée et professionnelle pour se déplacer vers un programme d'enseignement traditionnel.
There is currently no nationally recognized disaster medicine (DM) curriculum in Canada; yet physicians, nurses and paramedics are expected to be trained in DM and able to respond to disasters in their communities. Offering a traditional face-to-face course is impractical, given the expense and difficulty in assembling (scarce) faculty. Such courses are unable to provide training to the numbers of health care workers required and carry a significant financial burden for the learner: tuition, travel, accommodation and lost wages while attending a course.
Disaster Medicine Online (DMO) (email@example.com; www.disastermedicine.ca/) is a curriculum designed for health care professionals to develop their knowledge and skill-set in disaster planning and response. It has been designed using the principles of distance education to create an Internet-based, modular, interactive, asynchronous course. At present, this is a format that is not highly utilized in continuing medical education. The advantage of distance education over traditional face-to-face courses is the flexibility it offers to participants. Learners access the materials when most convenient for them and proceed at their own pace. Although learning from a book offers similar flexibility, distance education goes further -- it incorporates interaction with other learners and faculty and structures activities and assignments to facilitate the meeting of learning objectives. Tools such as computer-mediated conferencing boards and email encourage discussions in an asynchronous fashion (learners need not log in at any specific time); therefore, learners can participate regardless of their geographic location, time zone or work schedules. Participants weave learning into their lives over a longer period (a few hours per week for 3 weeks), rather than taking time out of their lives for a "bolus" of learning delivered in a concentrated manner over several full days.
The DM Subsection of the Division of Emergency Medicine at the University of Alberta has completed the first module of the DMO curriculum. In addition, a preparatory module is available to teach health care professionals the computer skills and provide the necessary software to participate in online distance education. Our objective was to evaluate the effectiveness of this online distance education course in Disaster Medicine, and this paper focuses on the pilot course provided in March 2002.
Prior to the 3-week pilot course, learners received login instructions and a course overview. The first week of the pilot was referred to as the "Welcome Week." Its purpose was to orient learners to computer-mediated conferencing (CMC), audio files, video files, downloading articles, and using email attachments. It also provided a venue for the facilitator and fellow learners to introduce themselves to the group and prepare for the remainder of the course. The latter 2 weeks of the course, "DM 101: Fundamentals of Disaster Medicine," consisted of 7 online lessons (see Appendix 1). Learners logged in every day or two over the 3 weeks to access the lessons, participate in online discussions and complete the assignments. An online facilitator coordinated the learners, oriented them regarding course access, led discussions on the CMC board and provided guidance on activities and assignments. At the end of the course, a summative assignment was submitted to the online facilitator, leading to a certificate of completion.
The target population for the course includes resident and attending emergency physicians, family physicians practising emergency medicine, nursing staff, and emergency medical services (EMS) personnel. The pilot learners were drawn from a targeted convenience sample of residents (n = 10), emergency and family physicians (n = 5), nurses (n = 1), paramedics (n = 5) and a medical student (n = 1). The pilot group was diverse with respect to gender, computer experience, computer access and previous experience with distance learning and DM.
The curriculum development strategy relied on the "Training for Improved Performance Series" (TIPS) model developed by Athabasca University,1 which requires a prospective evaluation plan. Readers are referred to this reference for details of course development and Web implementation, which are beyond the scope of this article. Evaluation mechanisms within the scope of this article include expert evaluation, usability evaluation and learner evaluation.
To assure confidentiality, participants submitted surveys anonymously to an undisclosed member of the Course Development Team. Data were then encoded and evaluated in SPSS version 8.0, student edition. Ethics approval was not pursued for this evaluation, and participant consent was obtained upon their agreement to participate in the pilot.
Ten external reviewers with expertise in DM, medical education or distance education were invited to evaluate the Web site and learning materials prior to the pilot.
Usability testing to assess the functionality of the Web site's interface was undertaken prior to initiating the pilot. Seven prototypical users who had a wide range of (self-assessed) comfort with computers were selected from a convenience sample of emergency medicine residents, paramedics and nurses (www.useit.com/alertbox/; accessed 2002 Sept 6).2,3 These subjects were asked to complete a standardized list of lesson activity tasks (see Box 1) using their home or work computers. They then evaluated 4 usability categories: ease of navigation, readability, page load speed and technical problems (i.e., "bugs" not discovered during development and beta-testing) (www.useit.com/alertbox/; accessed 2002 Sept 6).2,3 Usability was concurrently measured by a silent observer, who recorded the time required to perform a task, errors observed during the task and "unperformable" tasks. After task completion, subjects provided additional narrative comments by filling out a standard questionnaire.
Learners were evaluated at the time of course registration, through online surveys built into the Web site and through separate surveys following the 1st and 3rd weeks of participation. Facilitators tracked participation and progress of all learners on electronic CMC boards. Those not participating by (each) midweek were contacted to determine whether they required help. Formative evaluation of the learners was accomplished through the use of built-in pretests (self-marked) to some lessons, and through facilitated activities and exercises on the CMC boards. Summative evaluation of the learners was based on a final assignment focusing on the DM 101 learning objectives (Appendix 1). Evaluation of the facilitator was incorporated into the online surveys.
Reviewers identified minor typographical and usability errors. No major content or conceptual problems with the lessons were identified. A correctable platform/browser conflict was identified by 2 of our reviewers who used Macintosh systems with Netscape Navigator.
Usability testing revealed no major navigation errors. The 7 test users found the site easy to navigate and "friendly" in tone. Minor changes as a result of the usability testing included clearer labeling of "pop-up" graphics, end of lesson navigation icons, increased conformity with the use of typography (e.g., bold, italic) and the addition of a lesson on "refreshing" a Web browser.
Table 1 summarizes learner feedback on the Web site's lesson content. Users reported a high level of satisfaction (95%-100%) with Web site usability, layout, use of casual language and functionality. Some indicated that the audio and video elements were interesting but did not contribute to their ability to meet the learning objectives. Nineteen respondents felt that the CMC boards were useful, and 21 of 22 learners participated in the online conferences. A few expressed dissatisfaction when other learners dominated online discussions, leaving less opportunity for contribution by subsequent learners. Most learners considered the role of the facilitator valuable, and 85% indicated that they had sufficient guidance. Learners felt that the final assignment was appropriate and that they were well prepared by the preceding lessons.
Learners reported spending a mean of 11.2 hours (range: 5-20) over the 3-week course period. Twenty of 22 learners completed the final assignment and all 20 were successful in passing the course. Overall, 95% of learners said they would pursue another module if offered, and 100% would recommend DMO to their colleagues.
Physicians, nurses and paramedics are expected to establish disaster plans, train responders and coordinate the disaster response and care for disaster victims, yet few have the training, knowledge or experience to do so. A face-to-face teaching strategy that would meet Canada's educational needs in DM would be an enormous undertaking. We have shown that DMO is a viable option for teaching the essentials of disaster response to health care professionals who have computer and Internet access.
Although we launched the DMO project in January 2000, the events of Sept. 11, 2001, brought DM to the forefront of the national CME agenda. Our goal was to prospectively evaluate the learning materials and format of the online, modular, interactive, asynchronous curriculum developed in the DMO project. The DMO's evaluation system was designed to assess the satisfaction with and the educational effectiveness of the learning materials by learners and faculty. The DMO evaluation system was designed with both short-term and long-term goals in mind. In the short term, the focus was on formative evaluation through data from learners, faculty and staff to facilitate improvement of the learning materials. Long-term evaluation activities will include ongoing monitoring of the instructional systems design process for revising course materials. New material and learner support systems will be introduced, responding to formative feedback from learners and faculty.
Expert reviewers found no major "holes" in our course content or methodology, although minor usability and typographical errors were identified and corrected. Usability studies are performed to understand the way subjects interact with an educational product. The study was an important step in the evaluation of this Web-based curriculum because developers tend to lack objectivity when critiquing their inventions and the assumptions they used while designing. A key component of the usability assessment was to "watch" subjects working with the system because watching often reveals problems that users fail to report verbally or in a survey (www.useit.com/alertbox/; accessed 2002 Sept 6).2,3 Although all study subjects reported that the DMO Web site was easy to navigate, several had problems with pop-up screens used to demonstrate how to use the CMC boards. The demonstration images (screenshots) were sometimes mistaken for real Web pages, and the course development team subsequently pasted a "Demo" tag on them.
The pilot study was the ultimate test of the Web site and learning materials. During this phase, learners reported high satisfaction with the usability of the Web site, the quality and length of the lessons, the appropriateness of the assignment, the value of peer group interaction on the CMC boards, and the involvement of the facilitator. Some learners felt that some assignment questions were not challenging enough, and that the audio and video clips added little educational value beyond the text and still pictures in the lessons. Others pointed out that learners who "got to the boards first" covered most discussion points, leaving less to talk about for those visiting the CMC boards later. In future courses, the designers will adopt alternate strategies to create more inclusive discussions. These may take the form of questions covering more topics, or more questions exploring the same topics in greater detail posted continually throughout the module. Future assignments and exercises may explore questions in more breadth and depth to satisfy the diverse backgrounds and interests of the target audience. The designers will also have to closely examine individual multimedia elements to ensure that their merits deserve the extra bandwidth required.
All 22 learners said they would recommend DMO to their colleagues, and 20 of 22 (91%) completed the course. This is considered excellent because completion rates for distance education courses typically range from 40%-80% (Dr. Mohamed Ally, Athabasca University: personal communication, March 2002). Of the 22 learners in the pilot, all but one said they would pursue future modules in an online format. The one learner who would not, indicated that he missed the ability to see the other learners in the same room.
Several factors might limit the generalizability of our evaluation. The pilot group was not sampled randomly. Rather, an open invitation aimed at the target audience was put out, with a limit on class size. Due to the timing of the announcements, a number of spots were protected for EMS personnel to ensure their representation within the pilot group. Although this contrives the group mix, it is representative of the overall target audience. We do not know how the course would be perceived if the class was made up of entirely one profession. Further, much of our advertising was done through email, which undoubtedly selected for learners with at least basic computer skill levels. Some of the target audience for DM training will likely be uncomfortable with computers. Finally, unlike other content areas in the health care fields, DM does not lend itself to practical evaluation of learners. Unfortunately the geographical dispersion of learners in a distance education course is also not conducive to mock disaster scenarios or large tabletop exercises. Therefore, DM 101 learner evaluation was centred on case-based questions. Almost all the learners felt this was an appropriate method for evaluation. Further, the value of "mock disaster scenarios" has been questioned because they often put health care professionals into roles that they would not be expected to fill in a real disaster scenario (e.g., physicians performing search and rescue) (Dr. Garnet Cummings, University of Alberta: personal communication, March 2000).
Disaster medicine is an area that health professionals are expected to be knowledgeable in, yet there is no obvious means for learners to access this training in Canada. DMO represents a viable option through distance education for health care professionals interested in pursuing this training.
Information concerning Disaster Medicine Online can be obtained through the DMO Web site (www.disastermedicine.ca), by email (firstname.lastname@example.org) or by contacting Vera Klein at the CAEP Head Office: email@example.com
- Athabasca University. Training for Improved Performance (TIP) Series. Athabasca University Publication, Athabasca, Alberta, Canada. 1990.
- Nielsen J. Usability metrics. In: Alertbox: current issues in web usability. Jan 21, 2001. Available: www.useit.com/alertbox/20010121.html (accessed 2002 Sept 6).
- Nielsen J. First rule of usability? Don't listen to users. In: Alertbox: current issues in web usability. Aug 5, 2001. Available: www.useit.com/alertbox/20010805.html (accessed 2002 Sept 6).
Dr. Adam Lund, 10708 69th Ave., Edmonton AB T6H 2E1; 780 430-9554, fax 780 407-3314, firstname.lastname@example.org