By: DM Kashmer MD MBA MBB FACS (@DavidKashmer)
Sometimes, you can see the train coming but can’t get out of the way fast enough. Whack! The train gets you despite your best efforts. Wouldn’t have been great to start to get out of the way earlier? In this entry, let’s focus on how to identify, as early as possible, four types of bad metrics in healthcare so that we can run away from that particular train as early as possible. After all, the sooner we flee from these bad actors the more likely we are to avoid being run over by them.
Truth is, you’ve probably seen the train of bad metrics before. After all, you know that all sorts of things are getting measured in our field nowadays and, sometimes, certain endpoints don’t feel particularly helpful and (in fact) seem to make things a lot worse.
First, a disclaimer: this entry does not argue with metrics that the government mandates. There are some things that we measure because we have to for reimbursement or other reasons. However, if you believe (like me and other quality professionals) that a focus on reducing defects eventually impacts all sorts of quality measures (even mandated ones), then this is the entry for you! This work does not focus on arguing or pushing back against those things that we must measure owing to regulation. Now, on with the show…
Let’s explore four broad categories of bad metrics and how to avoid them.
#1 Metrics for which you cannot collect accurate or complete data.
It can be very challenging, in hospitals, to collect data. Often, data collection is frowned upon, or is even thought of as an afterthought or imposition. So, as we launch in here, remember: saying that you can’t collect complete or accurate data is not the same as actually being unable to.
Colleagues, listen: if you think you can’t afford the time to collect good data, let me tell you that you can’t afford not to collect and use data.
When I’m working with a team that’s new to Lean or Six Sigma and we discuss data collection, the team often balks and focuses on the fact that no one is available to measure data, that we don’t have data collection resources or that, even if we had resources, we can’t get data.
I usually start with a quote: “If you think it’s tough to get data, remember how tough it is to not get data.” (Split infinitive included for drama’s sake.)
Then we go on to explore together how there are several techniques we can use to make gathering data much easier so that we can avoid the “easy out” of “we can’t collect data about this and so it’s not a useful metric”. In fact, most projects we do require data collection for 1-2 seconds per patient at most. And that’s for prospective data collection. (Want more info about how to make data collection easy, email me at firstname.lastname@example.org and I’ll pass it along.)
However, in healthcare, we have all seen projects where data collection is arduous and so we react against data collection when we hear about it.
Sometimes, teams focus on using retrospective data. Of course, using retrospective data is much better than using no data. However, retrospective data has often been cleaned via editing or in some other way that makes it less valuable. Raw data that focuses on the specific operational definition of what you’re looking at tends to have the most value.
Sometimes, you have no way to measure a certain metric or concept and yet the team believes that concept to be very valuable. Take, for instance, a team that focused on scheduling patients for the operating room. The team felt that many patients were not prepared adequately before coming to the holding room. This included all sorts of ideas such as not having consent on the chart or some other issue. The team decided to measure this prospectively and found that only about one third of patients were completely prepared by the time they came to the pre-operative holding area. This was measured prospectively with a discrete data check sheet.
Let me explain that, sometimes, the fact that something hasn’t been measured previously means that the organization has not had that concept on its radar previously. This goes back to the old statement that if it is measured it will be managed and its corollary that if an endpoint is not measured, it is very hard to manage that endpoint.
To wrap this one up: it is important to mention that one category of bad data or a bad metric is a metric that you cannot measure. However, it is important to realise that just because you haven’t measured it before doesn’t mean that you absolutely cannot measure it. Sometimes, if the idea or concept is important enough, you should develop a measure for it. We discuss how to develop a new end point in the entry here. That said, if it is absolutely impossible or arduous to collect accurate or complete data, the metric is much less likely to have value…but don’t just let yourself off the hook! If you think something is important to measure, learn that there are ways to collect data that require only four or five seconds per patient!
#2 Metrics that are complex and difficult to explain to others.
If a metric gives a result that people can’t feel or conceptualize it’s just plain less valuable. Take, for example, a metric for OR readiness. In the month of April the operating room scored a very clear score on this metric. That score was “pumpkin”.
“Pumpkin?!”…Well, pumpkin doesn’t mean much to us in terms of operating room readiness. For that reason, you may want to measure your OR preparedness with a different metric than the pumpkin. Complex and difficult metrics that lack tangible meaning should be avoided. Chose something that tells a story or evokes an emotion. One upon a time, a center created (and validated) a “Hair On Fire Index” to indicate the level of emergent problems and crazy situations the operating room staff encountered in a day to indicate how stressed the OR staff was that day. Wonder how they did it? Look here.
#3 Metrics that complicate operations and create excessive overhead.
This type of metric is especially problematic. If a metric is difficult to measure and requires an incredible level of structure / workload to create it, it may not be useful.
Imagine, for example, a metric to predict sepsis that requires a twelve part scoring system, multiple regression, and the computing power of IBM’s Watson. This may not be a useful day to day metric for quality or outcome. Metrics that complicate operations and create excessive difficulty should be avoided. When you see that type of metric coming, jump out of the way of the train.
#4 Metrics that cause employees to ‘make their numbers’.
This is similar to problem metric number two. When staff can’t feel the metrics that we describe, or see how they affect patient care, it can be very hard to mentally link what we do every day to our quality levels. That can lead to situations where employees are acting just to ‘make their numbers’. That type of focus is difficult and makes metrics less useful.
It’s important to have metrics that we perceive as having a tangible relationship to patients and their outcomes. We are so busy in healthcare that often if staff can fudge a metric, complete a form just to say it’s done, or in some other way ‘make numbers’, well, we often see that’s what happens. (That effect may not just be confined to healthcare of course!) It can be very challenging to create a metric that very clearly indicates what we have to do (and should be doing) rather than one that is sort of an abstract number we ‘have to hit’.
Take Aways, Or How To Avoid Being Hit By The Train Of Bad Metrics
In conclusion, there are at least four types of bad metrics and very clear ways to avoid them. Take a moment to try to see these trains coming from as far away in the distance as possible so that you can quickly get off the tracks unscathed.
We need metrics that we can feel and that tell a story of our patient care. We need ones that, whether government mandated or not, seem to relate to what we do everyday. We need ones that are easily gathered and tell the story of our performance clearly to both us as practitioners and staff who review us. Sometimes, we are mandated to collect certain end points yet, over time, I have come to find that when we do a good job with metrics that have meaning, we often have less defects and see better outcomes in all the metrics…whether we are mandated to collect a particular metric or not.
As part of your next quality project and how you participate in the healthcare system, take a minute to focus on whether the metrics you’re using are useful and, if not, how you can make them better. Be the first to sound the alarm if you see the train of bad metrics on the track to derail meaningful improvement for our patients.