Monday, May 21, 2012

Eliminate, don't benchmark.

In several earlier posts, I have talked about how the use of benchmarks can be inimical to clinical quality improvement, stating a preference instead for absolute targets, like zero or 100%.  I understand this to be a controversial view, and you can see an excellent discussion on the topic in this post, with thoughtful comments offered by Marya Zilberberg and others.

It is reassuring to me, then, when I see respected health care systems adopting the more targeted approach -- and getting consistently great results.  An example is the University of Michigan Health System, whose Surgical Intensive Care Unit (SICU) and Trauma Burn Intensive Care Unit (TBICU) will receive an outstanding achievement and leadership award for eliminating ventilator-associated pneumonia (VAP).  The award is co-sponsored by the U.S. Department of Health and Human Services (USDHHS) and the Critical Care Societies Collaborative (CCSC), which is composed of the American Association of Critical-Care Nurses, American College of Chest Physicians, American Thoracic Society and Society of Critical Care Medicine.

The award requires consistent performance -- 25 months or longer -- accompanied by national leadership in sharing evidence-based practices with others. Well beyond two years, the U of M teams have shown results for more than 9 years.

The philosophy that is in place is set forth in the University's press announcement.
At any given time in the United States, 1 in 20 inpatients have an infection associated with health care they have received, and every year about 99,000 people die from a healthcare-associated infection (HAI).

Can HAIs be eliminated altogether? The University of Michigan says yes.

Transparency is part of this success. Here's the chart showing compliance with the bundle of steps taken to avoid ventilator associated pneumonia in all the ICUs, not just the two mentioned above.  This and other metrics are available for the world to see on the system's website.

Modesty must be inherent in this kind of transparency.  After all, you can't just show good results.  You have to show all results.  The point of transparency is to hold yourself accountable to the standard of care you say you believe in.  You learn from both successes and failures, and the learning is shared broadly throughout the institution. It takes leadership at all levels in the organization to pull this off because there are always folks who will want to retrench and avoid the perceived potential of embarrassment when results are other than stellar (not to mention lawyers who overstate the legal risks of such disclosure.)

As the U of M has shown, this is all worth it.  Hundreds of lives have been saved, people whose deaths previously would not even have been reportable as adverse events.  Congratulations!


Anonymous said...

Actually the target of zero is becoming less controversial as it is demonstrated to be achievable. Googling 'target of zero hai' yields numerous references to this effect, including the National Action plan of HHS, which states:

"F. Setting Elimination as an Aspirational Goal

Since the publication of To Err is Human in 1999, experts in the field of patient safety have discussed the concept and feasibility of not just reducing, but eliminating HAIs. Scientific advances, a changing culture of safety within the healthcare system, and the proven success of smaller scale prevention initiatives have now led to a consensus of the foremost experts in the field that the elimination of HAIs is a worthy aspirational goal and that the elimination of HAIs is possible.[13],[14]"

What disturbs me is the number of people, mostly MD's, who quibble with the idea of elimination as 'impossible' and 'doomed to fail', risking getting lost in the numbers and fear of blame, instead of understanding the point - which, as I see it, is to make large and sustainable improvements and keep making them - if you get right to zero, that's wonderful, but you certainly have not 'failed' if you don't hit zero the first time around, have you? This is intellectual arrogance and defeatism at its worst. How often, unfortunately, has our profession sat in this seat.

nonlocal MD

Bruce Ramshaw said...

I completely agree with the effort to continuously improve (rather than attempt to beat benchmarks that usually reflect mediocrity at best). I am also very supportive of radical transparency and agree the legal risks are paradoxically much less when done to genuinely improve care. However, I would note that the examples of eliminating central line infection and ventilator associated pneumonia are an incomplete application of the principles of clinical quality improvement (focused on only a part of the whole process to be improved). To truly improve care and decrease costs, the entire process (in this case a patients entire cycle of care for their problem) needs to be defined, measured and improved for definable patient groups. The difficult part is that this will likely require a change in the system structure for care delivery to allow for these concepts to scale. I believe this is why the valiant efforts of yourself, Atul Gawande, Peter Pronovost and many others have not spread as common sense would dictate they should.

Marya Zilberberg said...

Paul, great post, thanks again! And it was fun to re-read our volley in your previous post. I am going to take my prerogative to disagree again -- I still worry very much that there is a lot of wheel-spinning to show improvements that actually are not real. VAP in my mind is a really good example of what lurks beneath this apparent transparency. VAP is such a nebulous concept that it is extremely easy to disappear it. It is very dependent on how it is defined, and, as the invisible gorilla experiment showed us, we see what we want to see. I have written extensively on this issue both in peer-reviewed literature and on my blog, and finally it seems that the powers that be are in agreement with me (see this CDC doc The real proof will be in the pudding, so-to-speak, when we reduce the catastrophic numbers of mortality, morbidity, and abntimicrobial resistance (no study has convincingly shown that the vent bundle reduces the use of antibiotics). I agree with Bruce above -- we need a radical shift in financing, incentives and the whole delivery cascade. And that might give us data that we can believe.

Dan Diekema said...

I agree that UM personnel are doing excellent HAI prevention work.

By clicking on the "Outcomes" tab, you can review their CLABSI and VAP data in detail, with explanatory information. Two things become clear from this review:

1. They haven't eliminated (i.e. reached a "zero" target)

2. They benchmark (and they explain why they benchmark).

Please dissuade me from sending them a strongly-worded letter for violating both imperatives in the title of your post.

Paul Levy said...

Yes, they show the comparison to the benchmark, but they don't use it as their target. And, yes, they don't always reach the target, but they try.

Dan Diekema said...

Right, and I think this describes most hospitals: they haven’t achieved elimination (except for limited time periods in individual units), they very much would like to, and they still find benchmarks to be useful. I’ve practiced infectious diseases and clinical microbiology for 15 years, and I have yet to meet a hospital epidemiologist or infection preventionist who doesn’t want to prevent every HAI, or who would be satisfied with achieving the “median”. I also haven’t met one who believes that every single HAI is preventable. So whatever “controversy” exists around this issue seems to me to be based upon straw men. If it helps your QI efforts to set zero as your aspirational goal, then by all means do so. No one is preventing you, certainly not the CDC. Our hospital regularly celebrates units that have prevented a specific HAI for x months in a row. But making a fetish of zero can bring with it some unintended adverse consequences, and these should also be recognized and monitored. The goal, after all, is to provide the safest environment and the best outcome for each patient. Fudging definitions, overusing antibiotics, changing diagnostic approaches to reduce HAI rates, spending hours “adjudicating” HAIs, de-emphasizing HAI research (because we already know how to prevent HAIs!), none of these practices help patients.

Paul Levy said...

Wow, you went from setting a goal of zero to: "Fudging definitions, overusing antibiotics, changing diagnostic approaches to reduce HAI rates, spending hours “adjudicating” HAIs, de-emphasizing HAI research (because we already know how to prevent HAIs!), none of these practices help patients." Talk about creating a straw man!

I see that we don't have a simple disagreement about motivational tactics. You have aimed your howitzers at this one, asserting that there all kinds of misdeeds employed in trying to accomplish certain results. I don't know where you work or where you have seen that, but I certainly have not. What I see are well intentioned people trying to eliminate preventable harm, and often succeeding for extended periods of time.

You choose to damn all such efforts with the kind of assertions you make by calling it a fetish. It is not a fetish. It is a legitimate approach to process improvement.

By the way, I have met people in hospitals who are content with reaching the median -- as an ultimate goal -- and I will assert that having that attitude is equivalent to being content to harm and kill patients.

Anonymous said...

Reading Dan's/Marya's exchanges with Paul exposes the Jekyll and Hyde of modern medicine - there is the 'right' way to improve and the gamesmanship that goes along with the 'wrong' way. Dan, Marya and I have all experienced that gamesmanship in our hospitals, and I sense in their comments a clear concern that increasing emphasis on carrot and stick methods, as is happening today, will simply increase the motivation to game the system for the rewards/penalty avoidance. But Paul, here and in the past, has given clear evidence that culture change can lead us in the 'right' way.

My question to Dan and Marya is, if you don't like Paul's idea, how would you accomplish the goal instead?


Marya Zilberberg said...

Hi, nonlocal, I appreciate that you have asked the pivotal question here. Indeed, what do we do while the measures are imperfect? Well, I think the root of the problem needs to be addressed much more upstream from where the problem becomes visible. That is, the culture change that is needed is in the direction of "less is more" very early on. This may save a certain percentage of people from becoming critically ill, or even from becoming patients. For those who cannot avoid critical illness, however, we really do need to do something.

I am much more apt to believe the CLABSI data than the VAP data. CLABSI is a less elusive diagnosis, so more difficult (though not impossible) to game. There are other considerations, and I have laid them out repeatedly in my review papers on the economics in the ICU (you can Medline them). As for VAP, it seems like a witch hunt for something that is not even all that definable at the moment. (Again, you can find a lot of my take on it on my blog). This is the case of taking a deep breath and trying to gather more data. The elements of the MV bundle for the most part are just good medicine, but still need to be applied on an individual basis. For example, the head of the bed elevation rec comes from a single small RCT from Spain, where the VAP rates were high (I have also reviewed this at length on my blog), and the study conduct and reporting make the effect much less clear than the guideline acknowledges (again, look at my blog for details). There are also potential issues with GI prophylaxis --evidence for an increase in pneumonia risk (oh, the irony!) and a risk for C diff.

So, what would I do? I would strive to do good medicine and structure strategies to emphasize local needs, strengths and weaknesses, rather than one-size-fit-alls. I would definitely focus on process measures locally, rather than the outcomes, as the latter depends on too many random events and needs much higher multi-center numbers.

As for evidence of gaming, the recent pneumonia paper in JAMA that came from my very smart collaborators in Springfield, MA, suggests that to get better pneumonia outcomes, you can just upcode your pneumonia to sepsis (JAMA. 2012;307(13):1405-1413. doi:10.1001/jama.2012.384). I am not suggesting that all of this is done with malice; it is merely human nature at work.

Anyway, sorry for the long post. Leave it to Paul to come up with a post that stirs up such a terrific discussion!

Paul Levy said...

Thanks, Marya. A friend has privately written to suggest that an underlying concern here among clinicians might be that my approach could be viewed as gung-ho cheerleading parade by well-meaning but naive people which will lead to it being politically incorrect to admit to having not reached zero - thus driving mistakes underground and subverting the whole effort. S/he suggests that this might represent a perceptual divide between 'scientific' and 'corporate' approaches to achieving a goal - one believes in measured incremental progress and the other is more inspirational and aspirational to provide motivation.

I would respond that such could be the case except for one thing: The approach adopted by us at BIDMC and by U of Michigan and other places was, in fact, designed by the clinicians themselves, not by administrators. I know, speaking for myself, and taking the chance of speaking for people like CEO Ora Pescovitz at U of M, that we view our job as encouraging and empowering the clinician leaders in our midst. This is not, to quote an old ad, "your father's Oldsmobile," in terms of corporate values. It is an approach based more on principles of servant leadership.

And, to the extent that an institution adopts a Lean philosophy, it is an approach designed to produce zero defects -- not a level of defects based on some industry average.

If Marya is correct that some elements of the ventilator bundle are not the best way to go, it is by application of the scientific method to the process of clinical care that will uncover such defects. But in the meantime, we must pursue zero defects as a goal when we think about preventable harm. No other target is intellectually defensible.

Dan Diekema said...

Yikes. I didn’t think I was aiming howitzers here, I was merely describing what is actually happening in some hospitals, and there are published data to back this up. I also was not damning zero by calling it a fetish, I was saying that making it a fetish can lead to unintended adverse consequences. Nonlocal, I’m off to clinic now, but perhaps can answer your question in more detail later—but in short, the goal of driving infections to an “irreducible minimum” is achieved by changing the culture, in most hospitals unit-by-unit, so that it is easiest to do the right thing and no longer acceptable to do the wrong thing. This can involve rewards and celebrations when zero infections are achieved for periods of time, but I also think it is healthy to be honest about what we don’t know (and what we can’t currently prevent), and I still think gathering accurate data and using appropriate benchmarks can be useful.

Mark Graban said...

Benchmarking can be helpful when it's done for the purposes of learning from other organizations... not with the goal of blindly copying some trick or method that we saw at some other hospital (because those tricks tend to not work, in isolation, in our own organization).

Benchmarking is more harmful, I think, when it's just about comparing numbers and data.

Back in the 1980s, Ford wouldn't have been served well by benchmarking against GM and Chrysler. Their quality and performance was pretty abysmal across the board. Then, came Honda and Toyota, setting the bar higher. But, there was still much denial in Detroit (as executives assumed the Japanese must have been cheating the numbers, somehow).

When most modern hospitals have the same quality and patient safety performance (because they have basically the same processes), it's easy to get complacent comparing yourselves to the mean or a peer group that has equally poor quality and productivity.

So you're in the 80th percentile of an underperforming industry... who cares?

If you're going to benchmark against anybody, it should be those Hondas and Toyotas of healthcare - the organizations that have dramatically improved quality and patient safety... hopefully that inspires more improvement than lots of data and percentiles.

Jeff Thompson said...


I agree with Mark's comments that Benchmarking can be helpful for purposes of learning. In healthcare, we need stop worrying about comparisions for the sake of patting ourselves on our back. We should be setting the bar high to reach an ideal state and challenge each other to transform our delivery system.

Second, I agree with Bruce's comment that we need to move from
"single point improvements" to improving care across the entire patient care continuum which requires changes to the delivery model structure as well as the financial models for reimbursing our care.

Jeff Thompson
Healthcare Engineer Analyst at Mayo Clinic in Rochester, MN

"The views expressed on this [blog; website] are my own and do not reflect the views of my employer."

Paul Bataldan said...

Seems like three terms are in use that I understand a little differently than some of this commentary.

1. Benchmarking. A popularized translation of "dantatsu" -- Japanese for "best of the best." When Robert Camp of Xerox introduced the term, he detailed a process for conducting benchmarking (of process and outcome). As originally used the process invited re-imagining process design for better outcomes. Quickly the term became newspeak for comparison studies, a substantial degradation of Camp's original work.

2. Working at the theoretic limit. Paul O'Neill's innocent-sounding question, "Are you working at the theoretic limit?" in my mind is another invitation for critical process design thinking. If the current process was not designed with the theoretic limit in mind, or if the process performance data is not close to the theoretic limit, it stands as an invitation for re-design.

3. Arbitrary numeric goals invite gaming. Deming was quite clear about this point (I think it was number 11 of 14) and advised system literacy which included knowledge of the process and outcome. This was not a very popular recommendation as numeric targets were coming into their full flowering with MBO and other management programs and some didn't want to be troubled by the work of understanding the systems involved.

Anonymous said...

Point #11:

a. Eliminate work standards (quotas) on the factory floor. Substitute with leadership.

b. Eliminate management by objective. Eliminate management by numbers and numerical goals. Instead substitute with leadership.

If only it were so easy to 'substitute with leadership.' That's the whole point - gaming comes from lack thereof......

Anonymous said...

I just want to comment on the leadership point. Dare I say it? Patient-centered leadership, or any leadership really, in hospitals stinks. If good leadership existed, we wouldn't need to have half of these discussions. Perhaps that is one of the 'root causes' we should be attacking while we are imagining our redesign of medicine.

nonlocal MD

Bonnie Brossart said...

I agree completely with Paul when he says: "in the meantime, we must pursue zero defects as a goal when we think about preventable harm. No other target is intellectually defensible".

Last week, I had the privilege of learning from another workplace that saves lives - Autoliv - a company that build airbags and other safety systems for automobile manufacturers. When I spoke with the associates who work there - there was no question or debate - zero defects is their north star. (

It disheartens me to hear the "yeah buts" in the commentary. It disheartens me to hear the "yeah buts" in the commentary. Instead why can’t our collective energy and focus be on learning how and getting to zero. The people that use our health care systems deserve nothing less.

John Toussaint said...

Paul, the target should be zero, the goal should improvement.Benchmarking does nothing for either.Public reporting on the other hand, holds one accountable to improve as we have experienced at The Wisconsin Collaborative for Healthcare Quality

Sami Bahri said...

The goal should, no doubt, be zero mistakes. In a lean management system it is not even a subject for debate!

As for benchmarking, it is good if seen as an intermediate process that would lead to the “True North” of “Zero Defects.” In our quest for perfection we are always seeking new ideas and benchmarking could certainly be one source of ideas. If, however, benchmarking is seen as an end in itself, I would see it as a limitation to people’s potential.

To support this kind of reasoning, here is quote from Yoshihito Wakamitsu’s book, “The Toyota Mindset: the ten commandments of Taiichi Ohno: “The secret behind the Toyota system lies in the endless benchmarking techniques. Benchmarking promotes a way of growth in which you are compared to the best process or people and continually adapt such best practices to achieve progress.”p.38.