Do Quality Assurance programs make anesthesia safer? There is little direct evidence on which to base an answer only intuition and opinion. Academic and community anesthesia practitioners were recently surveyed about their attitudes toward such program. (Chapman GM M.D., personal communication, December 1990). Their comments ranged from, “have been extremely pleased with the program to date” to “…Q.A. a big game… quality of anesthesia not improved… does not get inside the head of a practicing anesthesiologist.” In between these extremes, there was a broad spectrum of opinion. Such opinions must be taken seriously before the anesthesia community uncritically adopts one or another specific Quality Assurance (QA) program.
An analysis of QA programs requires two assumptions: first, that any mechanism intended to assure better quality of anesthesia care at least includes the aim of improving patient safety and, second, that all anesthesiologists welcome objective information about their own performance so they can try to improve the care they give.
An examination of three well-known QA programs may reveal whether they are likely to help or hinder anesthesiologists in these endeavors. In Table I the philosophy behind each program and the methods used are classified. All three programs either state or imply that their aim is to make unbiased assessments of competence.
The incentive for developing these formal and, in some cases, commercially available programs, presumably is the following statement in the JCAHO Manual on Monitors of quality and Appropriateness of Patient Care: “The following screening criteria will be utilized to identify possible variations for review”. There after is a list of adverse outcomes such as cardiac arrest, peri-operative myocardial infarction, variations of blood pressure beyond certain ordained limits, etc.
Does the word “variations” mean mathematical demonstration of a number of adverse events beyond a predetermined acceptable frequency? It appears that the authors of these QA programs mistakenly concluded just that from the coincidence of three factors: fast, the belief that the nature of anesthetic practice is such that high quality is associated only with absence or decrease in number of adverse events; second, the looming demand from various authorities for recredentialling seems to require placing a numerical value on an anesthesiologists competence; third, the belief that norms of competence can be established for groups of anesthesiologists. Surgeons and patients constantly place new challenges before us. When it is assumed that a particular rate of adverse outcomes is the acceptable one, it is like trying to measure the height of a sand dune in the middle of a blizzard. Two recent examples of these challenges involve the complexities of taking care of patients having insertion of automatic implantable cardioverter defibrillators and the difficulties associated with anesthetizing young children undergoing scans in the highly magnetic environment of the MRI machine.
Summary of Three Quality Assurance Programs
Philosophy: Determining clinical competence involves decisions made by fallible humans and the best competence indicator is outcome
Method: Depends on outcome analysis which must be continual and collective, conditional, (i.e. under circumstances of usual practice), compared with peers, and include established minimal acceptable levels of performance.
Evaluation: Individual negative outcome scores compared to those of “competent” members of department.
Philosophy: Evaluate by absence of adverse patient occurrences.
Method: Evaluate adverse occurrences as “avoidable” and “unavoidable”. Care in avoidable cases divided into appropriate and inappropriate. Evaluation: Each department must set its own criteria. Revise QA program periodically.
Philosophy: Plan corrective action for practitioners with higher complication rates.
Method: Collect annual number of adverse incidents for the practitioner.
Evaluation: Compare to departmental mean.
Types of Programs
Do these types of QA programs justify their use of mathematical comparisons of incident rates?
Program A describes minimal performance levels for practitioners who are expected to anesthetize healthy patients without permanent damage, institute appropriate life-sustaining actions in life-threatening situations and, lastly, display insight when involved in an important error. It is impossible to quarrel with these guidelines, but diem are uncertainties about the validity of placing a numerical score on breaches of these guidelines. Even though cases are reviewed anonymously by a member of the departmental QA Committee, bias seems inevitable. Caplan et all, using cases drawn from the ASA closed claims database, showed that a large group of peer reviewers exhibited biases in a situation analogous to
Program A. They demonstrated an association between rendering judgements that care was inappropriate and the fact that the patients’ injuries were permanent and more severe By presenting identical case scenarios to the reviewers with alternating plausible but opposite outcomes, they demonstrated that the judgements were independent of the details of the cases.
Program B is much less certain as to whether competence can be enumerated. This uncertainty is illustrated by the following statements: “if the practice profiles of the providers consistently meet established standards, then it is likely the standards are set too law,” and later, “the acceptable rate for various adverse patient occurrences is not universally agreed upon!” However, it still leans heavily on the concept of counting adverse outcomes.
Program C boldly generates the incidence of critical events per 1000 anesthetics per physician and compares this with the departmental mean value. It emphasizes that such numbers are likely to be accepted by Government agencies that might establish a recredentialling process. This is a simplistic response to a bureaucratic demand. It is quite possible that more conscientious anesthesiologists do more difficult cases and voluntarily report more adverse outcomes (which could lead to their being sanctioned under this system).
All three programs invoke the use of widely accepted “standards of care” to support their methods and imply that the proposed QA mechanisms will lead to measureable improvement in anesthesia outcomes. The link to standards of cam seems somewhat tenuous. An example from the ASA Closed Claims Study is relevant. The ASA Closed Claims database contains 168 cases drawn from the files of the Massachusetts Joint Underwriters Association (JUA) which was established in 1975 and has consistently insured an average of 350 anesthesiologists. On July 1, 1987 all anesthesiologists insured by the JUA became eligible for a 20% “risk management” discount in their malpractice insurance premium if they signed a document agreeing to abide by the ASA Monitoring Standards and use pulse oximetry and capnography wherever applicable and also to submit to random audit on these points. Between 1975 and December 1984, there were 49 deaths which led to malpractice
Claims against JUA insured anesthesiology (2). Since the wide spread application of the discount on July 1, 1987, there has been no JUA suits brought for death associated with anesthesia in which practitioners were observing the rules of the risk management agreement. Nor were there any such deaths reported to the JUA occurring between January 1, 1985 and July 1987. To suggest similar results from a proposed QA program requires essentially “leap of faith” assumptions.
Anesthesia Hard to Study
Quality assurance activity in anesthesia is inherently more difficult than in other specialties in which studies of outcome are derived from controlled comparisons between alternative therapies. The study by Roos et al (3) comparing mortality and reoperation rates after transurethral and open prostatectomy is a good example. Since anesthesia per se is almost always non-therapeutic, its outcome can only be quantified in one way -fewer adverse outcomes. If I were to have no pneumothoraces following central venous catheterization for ten years (not true), does this prove that my technique and selection of patients for this procedure is period? I do not know. But I do know that since my patients have suffered this complication twice, I am more cautious than ever. If my colleague has caused three pneumothoraces in the same time period, this gives me no comfort as to the possibility that my technique is better.
Can this objection be overcome by pooling of data to compensate for the low frequency of adverse outcomes? This would require all anesthesiologists in a vast number of hospitals to practice in only one way, in comparable environments, with equally sick patients, working with measurably equally competent surgeons, and they submit themselves to a single peer review mechanism. These difficulties can be profitably compared to those experienced by the Health Care Financing Administration since 1986 while trying to use hospital mortality data as an adequate measure of quality of care’. Neither adjustment for length of follow-up nor severity of illness has yet been able to produce comparisons between hospitals that are valid or worth anything at all.
Feedback from Data
These criticisms of anesthesia QA programs do not deny that they may have other uses. One can learn from them. Few anesthesiologists have the time to record all the outcomes for all their patients, so data storage through a QA mechanism assists s&-analysis. This is the “feedback” benefit of data collection. Good large-scale examples of this are the Maternal Death Reports from the United Kingdom (5) which led to a substantial changeover from general to regional anesthesia in obstetrics and the New South Wales Mortality data (6) which contributed to a 5-fold decrease in anesthesia associated mortality between 1960 and 1987. A more specific study of the effect of instituting a QA program by Schreider et al (7) also suggests that feedback from QA may well contribute to improved care.
In all the welter of discussion about quality assurance, pea review, standards of cam, and practice parameters, one rarely heard the word “conscience” applied to the care of patients in the operating room. That is, until Rhoton et al (8) published their study of the behavior of anesthesia residents, which strongly suggested that conscientiousness (among other non-cognitive variables) was a powerful predictor of overall performance assessments and its absence played a crucial role as a predictor of critical incidents. The anonymous responder in the survey described above said functionally the same thing: “QA programs … do not get into the head of the practicing anesthesiologist.”
Suppose a quality assurance program demonstrates to an anesthesiologist that his patients have an unusually large number of perioperative M.I.’s. If one assumes that our criteria for selecting residents for anesthesia training are adequate, that review of training programs appropriately strict, and that the supervision and teaching of residents is conscientious, then this anesthesiologist will turn to Mangano’s article on Perioperative Cardiac Morbidity (PCM) (9) and will discover that some predictors of PCM have been clearly identified and others are suspected and he will adjust his thinking and practice accordingly. He does not need a QA committee to tell him to do this. If the reader considers this scenario naive, then he or she does not have an ideal view of the role of the anesthesiologist as physician.
Can others who are interested in the quality of care in medicine help answer these questions? Eddy (10) in a series of articles on clinical decision making says the assumption that all physicians’ decisions are correct is being increasingly challenged and that this challenge is justified by a concern for quality. He asks whether pre-guessing or second-guessing each physician’s decisions can be counted on to make the ultimate choice commit? He answers that, while these mechanisms are well-intentioned, while they are steps in the right direction, and while they may the best we can do right now, they will by no means prevent or correct every error. There are three main problem. First, who can say that a guideline developed by an expert panel is correct? Second, many medical decisions are inherently too subtle be made at a distance(eg. my colleagues rarely have time to watch me insert central venous catheters). Third, even if these quality assurance mechanisms worked in the sense of “correcting” incorrect decisions over the long haul, they are cumbersome, expensive, and potentially demoralizing for physicians.
Group Not Always Right
Most telling of all, Eddy states these mechanisms depend on a Questionable assumption. Virtually all current quality assurance mechanisms assume there is 11 accuracy in numbers”. In other words, if the decisions of individual physicians cannot be busted, can the collective decision of a large number of physicians be trusted? He concludes that the solution is not to remove the decision-making power from physicians through extreme imposition of substituted judgments, but to approve the capacity of physicians to make better decisions.
In the article “Continuous Improvement as an ideal in Health Care”, Berwick says it best: “We are wasting our time with the Theory of Bad Apples and our defensive response to it in health care today, and we can best begin by freeing ourselves from the fear, accusation, defensiveness and naivete of an empty search for improvement through inspection and discipline.”
Can QA programs improve patient safety? Yes; but only if they are used to give the individual practitioner concrete information about how to improve his own specific practice.
Dr. Zeitlin is a Senior Staff Anesthesiologist at the Lahey Clinic in Burlington, MA and an Associate Editor of the APSF Newsletter.
1. Caplan RA, Posner K, Cheney FW. Effect of outcome on physicians’ judgments of appropriateness of care. Anesthesiol 1990;73:AI247
2. Zeitlin GL. The Massachusetts anesthetic malpractice experience: Comparison with the American Society of Anesthesiologist Closed Claims Study. Poster presentation at 8th European Congress of Anesthesiology, Warsaw, Sept. 1990
3. Roos NP, Wennberg JE, Malenka DJ. Mortality and reoperation after open and transurethral resection of the prostate for benign prostatic hypertrophy. N. Engl. J. Med. 1989; 320:1120-1124
4. Green J. Wintfield N, Sharkey P, Passman LJ. The Importance of severity of illness in assessing hospital mortality. 1990; 263: 241-246
5. Morgan M. Anesthetic contribution to maternal mortality. Brit J Anesth. 1987; 59; 842-855
6. Holland R. Anesthetic mortality in New South Wales. Brit. J Anaesth. 1987; 59: 834-841
7. Schreider B, Roizen MF, Lichtor L, Roberts R, Keany M, Polk S, Delisi M, Holmes R. Does a quality assurance program improve patient care? Anesthesiol. 1988;693A:A499
8. Rhoton W, Barnes A, Flashburg M, Ronai A, Springman S. Noncognitive variables: primary determinants of clinical performance and precursors of critical incidents in five anesthesiology training programs. Anesthesiol. 1990; 73: A1056
9. Mangano DT Perioperative Cardiac Morbidity Anesthesiol. 1990; 72: 153-184
10. Eddy DM. Clinical Decision Making: From Theory to Practice The Challenge. JAMA. 1990; 263: 287-290
11. Berwick DM. Continuous improvement as an ideal in health cam N. Engl. J. Med. 1990; 320: 53-56