Have You Seen This Pessimist’s Guide To Benchmarking In Healthcare?

By:  DMKashmer MD MBA FACS (@DavidKashmer)

LinkedIn Profile here.

 

It sure sounds like a good idea to measure our healthcare processes against standards from other centers, right? It seems like pretty obvious logic that if we benchmark ourselves against how other organizations and professional societies want us to do (or how they perform) that we’ll be better off in the end. Doesn’t it sound straightforward that we should have an external benchmark that we compare to our processes?

 

Guess what? It’s not, and here’s why. You probably have a long way to go before you benchmark.

 

Thirty five healthcare quality projects in the last three years have reinforced this simple truism for me:  don’t benchmark at first. Why? There is usually a lot more you have to do before you look to some external agencies for a benchmark.  Here are some of the items that probably need doing before you scoop and apply an external measure to your system.

 

You Don’t Have A Clear, Usable Definition of What You’re Measuring

 

For example, your healthcare system probably lacks a clear operational definition of the metrics it wants to measure.  Will you use a definition for VAP (Ventilator Assisted Pneumonia) from the CDC or some other definition?  Does everyone who is performing data collection have the same definition?  Truth is, unfortunately, when you scratch the service…they probably don’t.

 

You Don’t Know The Voice of the Customer…Or Even Who The Customer Is (!)

 

You may not even know the voice of the customer (VOC) and key process indicators for your various systems.  Who exactly is receiving output from this system of yours?  And what do they (not you) want?  Get over yourself already and go find who is on the receiving end of your system and what they expect from the system.  You may even need to get out of the building to find out.  (Shudder!)

In other words, until you have a clear definition of what you’re measuring, a way to measure it, and a knowledge that it will significantly impact what you’re doing, you have a long way to go before you benchmark. Let me tell you more. One of the common areas we make with healthcare statistical process control and other quality projects is that we fumble at the one yard line. I mean that we don’t have a sense of a clear definition for what we are measuring or how we are going to measure it. How can we benchmark against an external measure before we even know what we are talking about? All too often, this is exactly what happens.

 

Consider this story of woe that owes itself to the problems we discussed above.

 

A Cautionary Tale:  VAPs in the ICU

 

Once upon a time there was an intensive care unit that wanted to benchmark its performance with ventilator associated pneumonia versus external organizations. (By the way, this is NOT the organization I work for!) It looked around and found typical rates of ventilator associated pneumonias as determined from other organizations. It seemed to make a lot of sense to do this. After all, they could bring their expected performance in line with other organizations. Of course, they wanted to have zero ventilator associated pneumonias as their real goal. What were the problems?

 

First, they had a non-standard definition of ventilator associated pneumonia. In fact, the operational definition they chose of VAP did not square with the definition of ventilator associated pneumonia from other centers. What did this cause? This caused all sorts of misguided quality interventions.  Alas, they didn’t discover this until a lot of work had been done.

 

For example, the team adopted a VAP bundle, which also makes a lot of sense. It then went on to perform no less than 12 other interventions in order to achieve quality improvement. Some of these decreased the VAP rate and some (many) did not.  The team spun its wheels and fatigue and staff churn quickly set in.

 

Another problem with external benchmarking? The team did not have the infrastructure to determine if they were doing significantly better or not. This is a common danger of benchmarking. The fact that the operational definitions did not align made the team add layer after layer of complexity and friction for dubious outcomes in quality. Worse yet, this wild goose chase caused an increase in worse outcomes owing to the variations that all of the ineffective changes caused in the system.  Because quality teams often lack sophistication to do statistical testing and to protect against tampering / type 1 errors, the wild goose chase in healthcare (sometimes from inappropriate benchmarking) really hurts!

 

I see this all the time and it’s very challenging to avoid in our current healthcare climate. For example, it is always hard to argue against doing more. Intuitively, who wouldn’t want to do more to make sure their patients were safe?  It’s an easy position to support, akin to “putting more cops on the street” promises from politicians.  Who could disagree!

 

However, it turns out, that when we make too many changes, or changes that do not result in significant improvement, we can unfortunately increase variation in our processes and obtain paradoxically worse outcomes. Processes can become cumbersome or resource intensive, whether that be in terms of manpower or other sources of friction. This is very difficult to guard against.

 

Learn from this instructional fairy tale: Align the operational definition you are working with, with your benchmark. Or better yet, don’t benchmark at first.

 

Important Thoughts on Benchmarking

 

So, if I’m telling you not to benchmark first, what is there to do? My recommendation is to follow the DMAIC process where there is a clear definition and those definitions are measured in rigorous statistical ways. This means having a team together that adopts a standard definition of the item that is being studied. I can’t say enough about that.

 

The operational definition for your particular item must align with the eventual item you want to benchmark. Typically in non-rigorous healthcare quality projects, this does not happen. Before you go on to accept the benchmark that you so badly want to look toward, make sure that this definition can be measured in adequate ways.

 

A measurement systems analysis and other measurement vagaries can really throw off your quality project. You can end up forever chasing your tail or the benchmark if your measurement system is not statistically rigorous or useful. Does the outside institution obtain the benchmark rate from retrospective cleaned data warehouses? Or did they obtain it prospectively right from the process? These are things you’ll have to wrestle with and it may make a difference in the benchmark you accept and what you think represents quality.

 

If the benchmark you are looking toward is a zero defect rate or some similar end point that’s one thing. However, typically we use benchmarks to get a sense of what a typical rate of performance is. As taught to me by experience and Lean and Six Sigma coursework: don’t benchmark until you have rigorously improved your process as much as possible. And when you do benchmark, I recommend that you have carefully aligned your operational definition, measurement system, and even the control phase of your project with this eventual benchmark.

 

Do you have thoughts on benchmarks? You probably feel, like I do, that used properly benchmarks can be very useful for quality projects…but when used carefully!  Have you ever seen a benchmark used inappropriately or one that caused all of the issues raised above? If you have, let me know, because I would love to discuss!

The Healthcare Quality Podcast: Mistakes With Control Charts

 

By:  DMKashmer, MD MBA (@DavidKashmer) with Vivienne Neale (@SupposeIAm)

 

Have you seen these common issues with control charts in healthcare?  Thanks to Vivienne and the podcast team at The Healthcare Quality Podcast for helping explore the use of control charts in healthcare!

 

 

Hi, and welcome to DDD, which is Data Driven Decision Radio, episode two, if you didn’t know. My name is Vivienne Neale and I’m delighted to be back with you. For those who have asked, my background is in education, training, broadcasting and social media. Of course, I am also a sometime patient, curious to know what decisions are being taken in my name that might just affect me and thousands of people just like me. So, this week, once again I am joined by David Kashmer, Chair of Surgery at Signature Healthcare. David is an expert in statistical process control, including Lean and Six Sigma. He has a special interest in new tools to improve healthcare, like gamification. David also edits and writes for a blog called SurgicalBusinessModelInnovation.com. Hi David and welcome back.

 

Vivienne, its great to be here again with you, and let me share that is been quite a week in the news for patient quality and satisfaction. We tend to talk, when we get together, about some of the various news items that we’ve seen and I have one to share with you and the listeners for this week. This one comes to us from ModernHealthcare.com and that well known magazine has a highlighted article this week about ‘bad metrics that put patients at risk and prevent providers from improving’, and as you know, this is something that is near and dear to our heart in healthcare and the focus on healthcare quality improvement. Their article goes on to say that hospitals most often penalised by the centres for Medicare and Medicaid are typically ones that do well on other publically reported quality measures and ones that are typically accredited by the joint commission, as that US crediting body for hospitals. Vivienne, to my mind, what this really highlighted is the tension between different metrics that we see in healthcare now. As you and I have discussed before, one of the challenges that we have in healthcare is this glut of data that we typically see as we try to do the best we can for patients and as we focus on quality. So, in today’s episode, I know we are going to go on to have a conversation about some of the specific tools, tips and techniques we can use to find meaning with certain quality tools. This article for ModernHealthcare.com really resonates with that tension that sometimes exists between accreditation and a true focus on quality. Really just fascinating stuff this week.

 

Brilliant. I think you are right to point out about the different metrics that are available to all of us and this glut of data that, in the end, we can’t see the wood for the trees and it’s quite difficult to make real progress.

 

Vivienne, I think that’s well said and we are seeing this all around healthcare now, for those of us who focus on being quality professionals and clinical practitioners. So, I’m excited to keep you and the listeners up to date on some of the different things we see and to focus on the tools that we are here to discuss today.

 

Right, indeed. Well, now, let me share something with you that I discovered in the UK press a couple of days ago. It was an item from the BBC and it’s about the importance of speaking out. although we all feel it’s our democratic right to be heard, sometimes the hierarchy of an organisation can make it actually particularly difficult for those lower down the pecking order to speak out. Do you have any experience of this?

 

Vivienne, as recently as a week ago, while in the operating room, I took a moment to actually solicit the thoughts of those around me and it’s something we do routinely in certain cases and situations, and it is a fascinating thing that when we review issues that we’ve had, cases that have a quality issue or something else, that there is often someone in the room who had a different thought or had a different perspective and they didn’t share it, for whatever reason. So, at certain points in certain cases, we actually take a moment to actively solicit everyone’s thoughts, even when we are on comfortable footing and we know what we are, where we are, what we’re doing, and we know it well, we still take a moment at those decision points in a case to sometimes solicit actively the thoughts of those around us. So, yes, I have seen it. That staff are sometimes uncomfortable speaking out. It’s a fascinating experience that we see commonly in healthcare. So, yes, it happens all the time.

 

Yes. Well, hmm. Interesting. I don’t know if many people do as you do. It has been pointed out that hospitals and airlines are highlighted as being areas of specific concern. So, one of the ways airlines are trying to reduce potentially fatal errors occurring is to use psychological techniques to break down that hierarchical structure and encourage people at all levels to highlight if something is about to go wrong, and guess what? Medicine, in general, is starting to follow suit. The aviation industry has embraced what is known as ’just culture’, where reporting errors in encouraged to prevent mistakes turning into tragedies. They discovered this, of course, through painful and tragic events, and that many people found it hard to speak up in front of senior colleagues, even when it was a matter of life or death. It’s something that can get in the way of openly pointing out errors. So, what’s worth noting is that even when teams are working very closely together, like the crew on an aeroplane, junior staff have been known to keep quiet in an emergency rather than question the actions of a pilot. I guess you can see just how that happens. So, surgical teams now hope to learn from years of research in aviation psychology, which have made crashed a rarity and may offer some pertinent lessons to medical staff. A guy called Matt Linley, who flies jumbo jets, but also now trains doctors in safety, recalls a case where a surgeon was preparing to operate on a child’s hand. A junior member of staff noticed they were about to operate on the wrong hand, but her fears were dismissed… her team finally realized they had operated on the wrong hand.  She tried again. Matt was adamant about what goes on at this point and he said, it’s quite usual, a lot of people just simply back down after the first time they’re not acknowledged, and she was told quite bluntly just to be quiet. Interestingly, the team finally realised they’d operated on the wrong hand about ten minutes into the procedure. Afterwards, the junior doctor said she actually felt guilty, but also that she didn’t have the skills to make herself heard.

 

Vivienne, what solution did the authors offer or describe that may help us in situations like these?

 

Well, actually, it’s making use of assertive phrases and using certain trigger words. Phrases like, ‘I’m concerned…’ or, ‘I’m uncomfortable…’ even, ‘this is unsafe’, or, ‘we need to stop’. I think no matter what position you are in the pecking order, to ignore those four trigger phrases would be very, very difficult, don’t you think?

 

I agree.

 

Most doctors say they’ve had a lightbulb moment when they’ve finished the course that this Matt Linley actually runs, and he says many say, “why am I doing this course when I’ve been a doctor for 25 years? I should have actually done it on day one”. So, if you want to more about this, we’ll add the link to the show notes. So, let’s get down to the business of the week’s show. David, what do you have for us this week?

 

Well, Vivienne, first I’ll share with you a personal note. I think that your comments on the importance of staff speaking up and trigger words can be overstated, and I will share with you that as a surgeon there comes a moment in every case that even when we are completely confident that we are in exactly the right spot doing exactly the right thing at exactly the right moment, we sometimes encounter quality issues, whether that’s a laparoscopic cholecystectomy or some other procedure, and I encourage mu surgical colleagues that even when we think that asking staff will be low yield, or even when we think that it would make us seem more apt, to appear that we don’t know where we are, which is a concern for surgeons. I say, even in those moments, it is worth our time to actively solicit from our staff, even if it adds five minutes to our case. Are there any concerns at this point? It is the critical part of the procedure… and that can really help with patient safety. So, the article you discuss really resonates with me.

 

I’m glad to hear that. Also, it makes us think about something that is termed ‘conscious incompetence’. Do you know what I mean?

 

I do. I think exploring that phrase is key. I think it reminds us that we can be conscious of those moments where there are high risk events occurring and we can leverage certain techniques to increase the probability that things go the right way. We can be conscious of those moments that are high risk and those quality issues can occur at different points in the procedure. So, I think that phrase is well taken and important.

 

Yes, I agree. So, come on, tell me about what you’ve got on your mind this week.

 

Well, this week Vivienne, I think we should explore control charts as a specific quality technique.

 

Right. So, are you talking about the kind of thing that’s used to determine how well a system is performing? The sort of things you might see outside operating rooms, administrator’s offices, and even hospital cafeterias. Am I right?

 

That’s exactly what I mean, Vivienne. Each location uses a control chart with the idea that the tool will show them when the data and the associated system are out of control.

 

Strange that, because I would have thought they were an easy visual tool that keeps everyone up to date with what’s going on.

 

Well, sadly that’s the oversimplified message of control charts and it actually leads us to apply control charts improperly. So, I want to flag this issue so that anyone who uses control charts to follow performance can be sure that they are using them the right way, so that they can actually get the accurate message from the data. In fact, I’m sharing that these can cause real problems if they’re not used properly. There are some major mistakes that are commonly made with control charts.

 

So, can you give us some examples please?

 

Well, first off Vivienne, a classic issue is choosing the wrong type of control chart for the data that we have and that can be problematic.

 

So, are you saying… is that something like understanding the type of control charts you should use with your data varies with the type of data that you have to hand?

 

Well, exactly. For example, in healthcare, one of the most useful charts that we can use is called the IMR chart, and IMR stands for Individuals Moving Range chart. They are particularly useful when an individual patient or an item moves through a system just one item at a time, and one of the first problems I see with the use of control charts is that staff often pick the wrong type of control chart and apply it to their data. If you go to our blog, and that’s surgicalbusinessmodelinnovation.com, you will see some charts that will help you get to grips on this and select the proper type of control chart for your data.  (Click here for that chart.)

 

I guess that’s easily done, really, when you’re in a hurry. It’s not an excuse, but I guess it does happen. I suppose there is also a tendency to create a control chart before you have the context to save time.

 

Well, Vivienne, you really picked up on it. That’s a fact, especially when the voice of the customer or the patient is missing. What I mean by that is, the point of a control chart is to get a sense of when the data are working with an expected variation according to the tolerances set by the voice of the customer of the process, or of the patient, or of the regulatory agency. So, if we apply a control chart too quickly, we see all kinds of issues and that kind of failure can be avoided.

 

So, David, what do you see as the main point of a control chart then?

 

Well, the control charts are utilized, Vivienne, so that you can recognize when a process is out of control, when it’s beyond expectations for what a system should do or when it’s tending to become out of control. The chart may highlight cases or values or outcomes that are causing problems much more quickly than other techniques. Sometimes control charts can even tell us things like whether the central tendency of the process is shifting.

 

Do you have any examples or can you explain this further?

 

Sure. For example, one of the treatments we typically use for trauma and acute care surgery involves fresh frozen plasma, a blood product or a blood related product. So, the issue is that we may feel like for a particular patient it took ‘way too long’ to thaw and deliver fresh frozen plasma, or FFP, to a patient. One of the first ways we can fail in this quality circumstance is not establishing the context for how the system functions overall. What the distribution of times typically is to deliver FFP for a patient. One of the next ways we can sort of fail in understanding and improving the system is to not have that voice of the customer or that tolerance for how long should it take? What is the upper limit of how long we are willing to wait for this medication? Vivienne, all those roll into the issue with the control chart because without that context should we go to apply a control chart, it won’t have meaning. We won’t know until we have a reference for the control chart, for how long it should take. We won’t be able to adequately understand, is this process performing as expected? Is it tending to become out of control over time? really applying not just the wrong type of chart, which we’ve already said can happen, but applying the chart too soon before having context also really impacts its ability to tell us what it needs to. So, really, with that said, you may think that a control chart tells you whether a process is being effective. That’s a typical mistake made, and guess what, control charts in no way tell you whether a process is adequate, useful or performing well. They only tell you whether a process is in statistical control, and that’s only if you select the correct type of control chart. A control chart will answer the question, are these data within expectations for the process I already have? Really, that’s all they do.

 

So, there is quite a lot to think about, and I guess you’re saying that relying on statistical controls only will not help either patients or staff.

 

Well, Vivienne, they can be very valuable, but they need to be applied just so. Meaning, just selecting a control chart or saying we’re going to put our data on a control chart, that actually will not typically have meaning for the way we want things to go for our patients and staff. So, rather than saying that they won’t help patients or staff, I would say the, I guess, more subtle message is that they have to be applied just so, in just the right way, in the just the right, at just the right context and then they can really have value for patients.

 

So, the unsubtle headline would be, do not use control charts until you have improved your process as much as possible.

 

Absolutely, and I would also suggest that departments ensure their process, the one that they’re looking at with the control chart, really satisfies the voice of the customer or the voice of the patient or the voice of the regulatory authority that they’re targeting. On our blog, we show a sample of a control chart and it shows that no individual case was out of expectation for the process, and to an administrator that often sounds great, right?

 

Yep.

 

Unfortunately, a control chart can look great and yet can demonstrate a process that is in no way adequate. Why? The figures demonstrate the process does not satisfy the VOC, or voice of the customer, when it’s applied. This brings up the interesting situation where a process is completely in control and yet wholly inadequate. We see this all the time in healthcare where run charts and control charts may be misapplied. If you do it wrong, the data will look in control and just great, yet the process will remain completely inadequate and both you and your audience will be fooled.

 

Right. So, David, you can have an IMR with data in control, but actually be running an inadequate system as a consequence? Am I right?

 

Well said, Vivienne. A control chart can show you that all data are in control, as if to say, “Nothing special here, so go about your business”, and yet the process hums along on its merry way to making outcomes that are completely unacceptable. That’s the danger of misapplying a control chart. Doing so makes us miss the whole point.

 

Ah, so I’m guessing the take away here is before you apply a control chart, make sure you’ve improved your data and try the best you can to become compliant with the voice of the customer.

 

Well said, and yet I would also add, there are other restrictions or important points that also go into what we have to do before we use this tool of control charts. You have to ensure that the department does not apply a control chart that’s based on the wrong underlying data distribution.

 

In your experience, is that common?

 

To be honest, most control charts that I see commonly used assume that data are normally distributed and that’s a classic fail. In fact, much data for health systems, including things like patient time in the emergency department, length of stay for patients and many other examples are often non normal data. So, applying a control chart, a specific type with its assumption that the data are normally distributed, is a nonstarter.

 

So, what do you think, David, is a potential solution to this?

 

It’s straightforward. We need to use control charts based on the distribution that we know the data follow. That’s why it’s important to get a sense, Vivienne, before we apply a control chart, of what data distribution we have, which one we’re looking at. If they are not normally distributed and in fact if they are some non-normal distribution, it makes no sense to apply a control chart that requires the assumption that the data follow the normal distribution, as it’s called. As a matter of fact, again, the misapplication of that control chart will mislead us.

 

A classic case of garbage in/garbage out model?

 

Yes, indeed. In my practice, I have found that the items outlined today have helped keep me out of trouble as I’ve gone to apply control charts for statistical process control projects in healthcare. Remember, if you haven’t improved the process and you haven’t placed it in a context already, it makes no sense and really is of little value often to apply the control chart. The control chart will only tell you whether the process is performing within some typical zone and not whether it is good enough. A control chart simply can’t tell you, when used improperly, that things are just fine.

 

Well, David, I think that’s given us sufficient food for thought and good enough is never appropriate in matters of patient or clinical safety. So, thank you very much, David, and if you want to keep up to date with David Kashmer’s approach to quality and statistical process control, business model innovation and critical practice, do join us for the next programme. In fact, we are very interested to hear what innovative practices are being undertaken in your health provision. If you’d like to appear on the show, contact us through our website and we’re looking forward to hearing from you. Meanwhile, if you’ve liked the show, do leave us a rating on iTunes. It’s one way we can ensure the word is spread, and we look forward to being with you next time. So, bye for now!