On September 9, 2015, in Phoenix, Arizona, the APSF convened an experts’ workshop entitled Implementing and Using Emergency Manuals and Checklists to Improve Patient Safety. The background for this conference addressed the need for anesthesia professionals and other members of the perioperative care team to move towards the acceptance of cognitive aids (emergency manuals, checklists) and away from the traditional reliance on memory and the cultural perception of individual perfection. Cognitive aids include a variety of physical and electronic representations of knowledge “in the world” designed to assist those responsible for perioperative care in executing complex decision-making in dynamic settings. The reality is that no one can function as the lone expert recalling every procedure and drug dose from memory. Successful care of the patient in the perioperative period, particularly during critical events, has previously been considered to be the exclusive responsibility of an individual’s knowledge and skill. This is now being recognized as not optimal because human memory is limited and fallible, especially under stress. A recognized principle of human factors research is the use of both knowledge “in the head” (memory) and “knowledge in the world” (presented externally) combined with inter-professional teamwork. The combination of these may allow the best opportunity for optimal outcome, particularly in crisis situations.
The goals of this conference focused on the practical aspects of systematically implementing Emergency Manuals/Cognitive Aids and Checklists in the in the perioperative setting. With Dr. Robert Stoelting, APSF President, and Dr. David Gaba serving as conference co-moderators, Dr. Stoelting opened the conference, welcoming the over 100 participants representing diverse stakeholders including anesthesiologists, CRNAs, anesthesia associates, surgeons, OR nurses and technicians, insurance providers and several healthcare companies with anesthesia interests.
The program began with a series of informational podium presentations, followed by panel discussions and culminating in small group breakout sessions, which led to a number of recommendations.
Dr. Gaba, who provided background information addressing why crisis management cognitive aids are needed in anesthesia and perioperative care, delivered the first presentation. Some reasons included that “even smart people need help in dynamic settings” and that oftentimes factors such as subconscious complacency and premature closure may play roles in limiting optimal performance when dealing with diagnosis, planning and treatment. Dr. Gaba also provided perspective obtained from Dr. Gawande’s Checklist Manifesto, simulation-based studies of manual use and the role of the reader, and the Emergency Manual Implementation Collaborative (EMIC). Following Dr. Gaba’s presentation audience response data demonstrated that 82% of participants felt that every site of perioperative care should have one or more emergency manuals (EMs) readily accessible.
Historical Background Provided
David Hepner, MD, MPH, from the Brigham and Women’s Hospital in Boston, presented a review of relevant literature. He noted a 1924 historical report of such a manual by Babcock. Yet, it was many years until the concept gained serious interest in hospitals. Runciman, in 2005, introduced the idea of a crisis management algorithm based on studies of anesthesia critical events. A Stanford team led by Gaba and colleagues in 2006, reported on the use of cognitive aids on laminated cards. Those who used cognitive aids during simulated emergencies were more effective in performing critical steps compared with those who did not use such aids. Dr. Hepner provided a number of examples of studies demonstrating the value of cognitive aids, including a 2012 study by Dr. Joseph Neal demonstrating improved trainee performance during a simulated episode of local anesthetic toxicity using the ASRA checklist. Dr. Hepner also discussed how Pronovost reported on the use of checklists to reduce central line infections in the ICU setting. In addition, Zieacz et. al., published their study of the use in simulation of a surgical crisis checklist, which garnered national attention.
Hard Copy Manuals Tend to Disappear
Dr. Steven Howard, from Stanford and a co-author of the 1994 book that holds what may be the first extensive compilations of emergency procedures in anesthesia, described the advantages of using hardcopy versions of an EM. Hard copies are familiar to all clinicians, relatively easy to create, don’t require special revisions dependent on a specific platform or operating system (vs. electronic systems), and can be easily modified by adding or replacing pages. In addition hard copy can withstand power failures and Wi-Fi outages. However, he noted that there are problems intrinsic to the hard-copy model, including the need to identify an obvious place to put them where they are accessible, yet not in the way. Most challenging is that hard copies can and do easily disappear from the OR. The concept of hard copy vs. electronic was studied in simulation. Interestingly, one third of people didn’t use the cognitive aid despite good instruction in its use. The hard copy seemed to be favored somewhat over the electronic version. Dr. Howard also shared his perspective that other issues are much more important than the format of the tool. These important issues include “training with the tool,” cultural acceptance, practitioner acceptance, and determining how best to use while recognizing potential pitfalls. Audience participation results revealed that 84% of participants felt that hard copy manuals had several advantages over electronic versions. In addition, 92% of participants believed that more studies are necessary to determine how to best utilize emergency manuals.
Pros and Cons of Electronic EM Explored
Laurie Schleelein, MD, from the Children’s Hospital of Philadelphia, described how electronic apps could be used on personal devices for accessing an emergency manual. Advantages of electronic versions include the ability to interact with the user, for which some improvement in performance during resuscitations has been reported. The interaction can include a form of checklist that branches depending on what is done or observed. An electronic version also allows the input of patient specific information, such as entering the patient’s weight, to allow a weight-based dose. It can include a clock to enable decision-making based on elapsed time and warn when critical times are passed. An app can be used to record what has been done, which can be used for debriefing and to facilitate learning from prior events.
Emergency manual apps can be updated automatically via a server. Mobile phones and tablets are now widely used, so downloading to a very large population of users is possible. That is already happening based on the world map of downloads of the application demonstrated by Dr. Schleelein. Electronic platforms have disadvantages as well. Technology failures are possible, it can be difficult to navigate, easy to accidently jump to the wrong place and struggle to get back, and the display size can be limited. Electronic apps are expensive to make and need to be operating system compatible. It can be cumbersome to hold a device, which might be overcome with a reader. Research about electronic vs. hard-copy aids is conflicting, so it’s currently unclear which is the most desired approach. Audience participation results showed that 41% of participants agreed and 40% disagreed that the advantages of electronic apps far outweigh the disadvantages, while 74% of participants felt that there is a risk cognitive aids will distract the team from the emergency situation.
Matti Lehtonen of GE Healthcare subsequently discussed how emergency manuals could be embedded in clinical equipment. This has the advantage of enabling patient-specific data being accessible as part of the EM algorithm. He thinks it may be possible to build some intelligence into technology that can actually give early warning of an impending crisis to avoid it. His engineering team has many pilots; they study what’s been learned and what’s being done in aviation. Embedded, paperless versions of checklists are being integrated into new aircraft. This potential technology raises several questions. If the checklist is integrated, where should it be displayed? Is a central display for everyone to see or a personal device preferred? These questions remain unanswered. Preliminary work suggests many advantages to an embedded EM, e.g., the interface will be familiar, and real-time patient data can be accessed. But such systems could limit individual adjustments to EMs if they are standardized by a manufacturer. The audience strongly agreed that a central visible display (83%), context-sensitive information (85%), and predictive algorithms (85%) would be beneficial in crisis management situations. In addition, 86% of the audience believed that combining electronic patient information with caregiver input would allow more appropriate and efficient checklists. 62% of the audience felt that the primary barrier to utilizing EMs and checklists is cultural, while only 9% believed barriers to be technological. 76% of the audience thought that a trained reader of the manual should be designated during the time-out.
EM Use Can Have Pitfalls
Following a short break, APSF Executive Committee member Maria van Pelt, PhD, CRNA, moderated the second session, which began with Dan Raemer, PhD, from Massachusetts General Hospital (MGH) who spoke about the pitfalls and risks associated with the use of emergency manuals. He explained that he had been a proponent of EMs for many years; however, he has shifted his thinking, having seen many real and simulated anesthesia crises and instances where the EMs have not been as helpful as hoped and perhaps harmful. He acknowledged that he did not have data to support his concerns, rather anecdotes from his observations.
Dr. Raemer illustrated pitfalls with examples he has seen. In one case of septic shock, the team perseverated on a diagnosis of malignant hyperthermia (MH). This occurred after someone suggested MH as a possibility and turned to that page in the manual. This was an example of a fixation by being on the wrong page. In another case of a mixed diagnosis with possible components of anaphylaxis and/or transfusion reaction, the team went back and forth between pages and to other pages without getting to a correct course of treatment. In yet another case, the correct diagnosis was septic shock, for which there is no page in the manual. This team became distracted and did not provide appropriate treatment.
A relevant audience response question revealed that 99% of participants believed that the introduction of EMs, like any new technique or technology in medicine, presents unanticipated risks and potential complications.
Amanda Burden, MD, of Cooper Medical School of Rowan University, made the next presentation. Dr. Burden described ways to mitigate the risks of using checklists and the role of a reader during crisis management. She reviewed the history of checklists in aviation, beginning around WWII. Despite the evidence of the effectiveness of checklists, pilots were resistant to them because of their culture of independence and reliance on their skills. She went on to review how stress and expansion of knowledge impact the ability of physicians to do the right things during critical situations. The kinds of stresses and challenges in aviation and perioperative medicine are similar and the resistance to using checklists was the same during the introduction in aviation, as they appear to be now in health care. In aviation, the solution was to do research and training to both optimize their use and teach people to use them effectively. She presented a number of pitfalls and possible solutions including the development of better checklists, the use of a reader, use of crisis resource management (CRM) skills, team training and a supportive culture.
The audience response indicated that only 19% of participants believed that limitations of checklists must be overcome before their use should be widely adopted. Similarly, 92% of respondents disagreed with the position that if teams and individuals practice CRM, checklists are not necessary.
Consensus Building is Not Easy
Dr. William Berry, one of the early pioneers in introducing the concept of checklists into perioperative care, addressed the question if a standardized EM should be developed. He described how he and his colleagues got involved with the topic of checklists based on an adverse surgical event experienced by a surgical colleague. He took the opportunity to describe the efforts of EMIC. He acknowledged that, currently, there are many different tools from different groups; there is no consensus on what items should be included in an EM, on the format, or on needed training. The process of implementing a checklist locally is a great benefit for getting people to consider how to prepare for, and how to manage emergencies. He gave an example of how one hospital developed their own new checklist for bronchospasm, which was not in some other manuals. Much more innovation and creativity are needed to optimize the use of EMs. It would be very challenging to create a standard manual, to reach consensus on what belongs in such a manual, the steps for each situation, and then be able to maintain the product. He does not feel the time has come to do that, nor believe that it may ever come to fruition. 87% of the audience agreed that there are a number of questions that need to be answered prior to creating a single standardized manual.
Dr. Paul Preston from the Permanente Medical Group took a somewhat different position from Dr. Berry. He asked what would it take to standardize? His rationale was, in part, that there is a cost to each hospital creating its own manual. It would be easier to implement if less time was needed to select an EM and thus more time could be available for training. Having standardization would enable EM integration into an electronic record, current vendors of which do not allow much customization. He also sees a great advantage for those who travel between institutions, although of course some local information, such as phone numbers, would be different. In audience response, 86% agreed that it would be helpful to have a standardized set of EMs, which could be tailored and used for regular emergency drills. Only 35% of the audience felt it would be important for each institution to design its own set of EMs to reflect their clinical situation.
Dr. Sara Goldhaber-Feibert from Stanford shared her expertise and extensive experience in implementing and studying EMs. She elucidated some of the best practices for implementation. Champions, leadership buy-in and local teams are keys to success. Local adaptation and customization has been shown to be vital, as well as learning from what others have done. She stressed that it is important to avoid aiming for perfection. It is important to synergize the EM implementation with other patient safety goals and context. Having the EM can be an incentive to beginning full OR teamwork training. Getting input and buy-in from multidisciplinary leaders is another piece of the successful implementation puzzle. Success is also aided by a diverse training plan that includes getting buy-in to do it, what goes into it and then how to use the manual and implement it. Encouraging self-review and educational use is also very effective. She also stressed the importance of publicizing success stories and bright spots in anthropological terms. It was made clear that there is much to be learned in the growing field of implementation science. The steps noted above are generally used in other fields so there is evidence that they have some science behind them. There are also many reports in health care regarding successful implementations of patient safety and quality interventions. 96% of the audience agreed that there are many steps between an individual downloading a useful EM and an institution effectively implementing it clinically. 66% agreed that literature on EM implementation and clinical use is accurately described as “nascent.” Interestingly, only 31% of respondents agreed that the greatest barrier to implementing EMs and checklists seems to be the belief that their use denotes some sort of failure on the part of the anesthesia professional.
Dr. Bill Paulsen moderated the final series of presentations, which began with Dr. Alex Hannenberg describing ideas for how to make a “team sport” of the use of EMs. Dr. Hannenberg is a member of the board of directors of the Council on Surgical and Perioperative Safety, which is a multidisciplinary coalition of 7 associations representing professionals involved in surgical care. He sees simulation as being a vehicle to introduce emergency manuals to all the players on the team. In his experience, the process of introducing EMs locally can help identify system weaknesses. Each discipline sees weaknesses that are invisible to the others. He emphasized the importance of involving all of the specialties in the process of creating, editing, implementing, and training for EM introduction and use. In this way, a “team of champions” is created, all of who own the product. Dr. Hannenberg advised that we create a “we use checklists” mentality, the idea being that good clinicians use cognitive aids. He wants the nurse to be able to say, “Dr. Hannenberg, do you want me to bring the code cart in here, do you want me to get the Emergency Manual?” He told the story of how one of his senior surgical colleagues, when asked how he’d feel about being asked to read the manual, replied, “You have no idea how it feels when your patient is dying on the table and there’s nothing you’re involved in doing about it. I’d happily read out the checklist.” Dr. Hannenberg quoted a 2012 Study from the Annals of Surgery demonstrating that the success of implementation of a safety checklist was improved when a multidisciplinary team led the process, rather than when a single staff member led it. Audience response revealed that 79% agreed that anesthesia professionals should lead the development of the content for cognitive aids for OR emergency management; however, 96% disagreed that only anesthesia professionals should call for the use of an EM.
Matt Weinger, MD, Vanderbilt University professor and Director of the Center For Research and Innovation in Systems Safety, spoke to the question of what research is needed to evaluate the safety and effectiveness of EM use. He presented preliminary results from a large study of board-certified anesthesiologists who were participating in MOCA simulation courses at 10 different U.S. sites. In a highly organized, controlled study, experts rated 364 CRM scenarios. Only 74% of all critical items were actually performed. For some critical actions, as few as 7% of anesthesiologists performed a task that experts had determined was critical in that situation. Only 54% actually used the MH protocol during the simulated MH event. For overall performance, the average score was in the middle of the scale, or just average performance. These results were supportive of the need for significant improvement and the potential benefit of training with, and implementation of, emergency manuals and CRM.
Not Feasible to Conduct Ideal Study
Dr. Weinger generated numerous questions that that needed to be addressed with further research. He focused on the design of an ideal outcome study, introducing cognitive aids into practice and measuring patient outcome. With an estimated event rate and effect size, he estimated it would require 1.6 million patients and over three years to conduct such a study. His conclusion is that it just is not feasible to conduct such a study for EMs. Conversely, such a study could be done for the WHO checklist introduction, since it is used in all cases, not just rare emergencies. Dr. Weinger recommends that large prospective studies have to focus on teamwork interventions and that cognitive aids need to be used, but not studied as the cause of the outcomes. There was strong agreement with Dr. Weinger’s conclusion that studying outcome improvement resulting from EM introduction is not likely useful.
Dr. John Eichhorn’s presentation followed, in which he suggested actions APSF could take to further implement using EMs to improve patient safety. Dr. Eichhorn, as founding editor of this newsletter, recommended that the APSF should work to dispel the message that the use of cognitive aids is a weakness, but rather is a strength. He also noted that the APSF had been highly influential in the success of widespread use of simulation. In discussing how the APSF could effect change, he suggested that efforts should primarily be via education and advocacy. Dr. Eichhorn reminded the audience that the APSF does not set standards; rather this organization can spread the word via the APSF Newsletter, which is the most widely disseminated anesthesia publication in the world. He also suggested videos and visibility in the APSF booths at the ASA and AANA annual meetings. Further, the APSF could lobby its constituents. He noted that APSF recommendations about audible alarms were adopted by the ASA. The question of supporting a new standard of care would necessitate the ASA as the best pathway. It’s not likely that new standards will be created, but guidelines are possible.
A large majority of the audience felt that the APSF should take a leading role in promoting EMs; however, the audience was split on the question of whether EMs should ever become a standard of care with only 67% voicing that opinion. Many of these issues were discussed and revisited during the final panel discussion.
Finally, the small group breakout sessions resulted in very creative and constructive ideas and recommendations. Some examples follow:
- Create an APSF education/advocacy package, including a video and a PowerPoint about implementation and use of EMs along with a “toolkit” for interested individuals/champions.
- Develop, implement, and maintain a strong social media presence for proper use of EMs.
- Recommend that a part of the pre-surgical timeout should consist of the verification of the presence of an emergency manual, to remind the team that anyone can suggest its use, and that a reader be designated as appropriate to the situation.
- Invest in research and career development that improves implementation science related to the use of checklists.
- Advocate for the processes and education it takes to successfully engage local environment teams to implement the use of unified checklists.
- Use research to determine how one could design an EM that is so simple that no training is required.
- Use research to examine the effect of a manual or checklist focused on a single more common event (intraoperative hypotension) on composite patient outcomes.
- Determine if the APSF should play a leading role in developing and testing a new national crisis event management curriculum intended for all perioperative learners.
The conference demonstrated that we are on the right track, but much work needs to be done.
Dr. Morell is the Senior Co-editor of the APSF Newsletter, a member of the APSF Executive Committee, and a private practice anesthesiologist in Niceville, FL.
Jeffrey B. Cooper, PhD, is Professor of Anaesthesia at Harvard Medical School, Department of Anesthesia, Critical Care & Pain Medicine, Massachusetts General Hospital, and Executive Director at the Center for Medical Simulation, Boston, MA.