David Kashmer (@DavidKashmer)
Of the many barriers we face while trying to improve quality in healthcare, none is perhaps more problematic than the lack of good data. Although everyone seems to love data (I see so much written about healthcare data) it is often very tough to get. And when we do get it, much of the data we get are junk. It’s not easy to make meaningful improvements based on junk data. So, what can we do to get meaningful data for healthcare quality improvement?
In this entry, I’ll share some tools, tips, & techniques for getting meaningful quality improvement data from your healthcare system. I’ll share how to do that by telling a story about Super Bowl LI…
The Super Bowl Data Collection
About ten minutes before kickoff, I had a few questions about the Super Bowl. I was wondering if there was a simple way to gauge the performance of each team and make some meaningful statements about that performance.
When we do quality improvement projects, it’s very important to make sure it’s as easy as possible to collect data. I recommend collecting data directly from the process rather than retrospectively or from a data warehouse. Why? For one, I was taught that the more filters the data pass through the more they are cleaned up or otherwise altered. They tend to lose fidelity and a direct representation of the system. Whether you agree or not, my experience has definitely substantiated that teaching.
The issue with that is how do I collect data directly from the system? Isn’t that cumbersome? We don’t have staff to collect data (!) Like you, I’ve heard each of those barriers before–and that’s what makes the tricks and tools I’m about to share so useful.
So back to me then, sitting on my couch with a plate of wings and a Coke ready to watch the Super Bowl. I wanted data on something that I thought would be meaningful. Remember, this wasn’t a DMAIC project…it was just something to see if I could quickly describe the game in a meaningful way. It would require me to collect data easily and quickly…especially if those wings were going to get eaten.
Decide Whether You’ll Collect Discrete or Continuous Data
So as the first few wings disappeared, I decided about what type of data I’d want to collect. I would definitely collect continuous data if at all possible. (Not discrete.) That part of the deciding was easy. (Wonder why? Don’t know the difference between continuous and discrete data? Look here.)
Ok, the next issue was these data had to be very easy for me to get. They needed to be something that I had a reasonable belief would correlate with something important. Hmmm…ok, scoring touchdowns. That’s the whole point of the game after all.
Get A Clear Operational Definition Of What You’ll Collect
As wings number three and four disappeared, and the players were introduced, I decided on my data collection plan:
- collect how far away each offense was from scoring a touchdown when possession changed
- each data point would come from where ball was at start of 4th down
- interceptions, fumbles, or change of possession (like an interception) before 4th down would NOT recorded (I’ll get to why in a minute.)
- touchdowns scored were recorded as “0 yards away”
- a play where a field goal was attempted would be recorded as the where the ball was on the start of the down
Of course, for formal quality projects, we would collect more than just one data point. Additionally, we specify exactly the operational definition of each endpoint.
We’d also make a sample size calculation. Here, however, I intended to collect every touchdown and change of possession where a team kicked away on fourth down or went for it but didn’t make it. So this wasn’t a sample of those moments. It was going to be all of them. Of course, they don’t happen that often. That was a big help here, because they can also be anticipated. That was all very important so I could eat those wings.
Items like interceptions, fumbles, and other turnovers can not be anticipated as easily. They also would make me have to pay attention to where the ball was spotted at the beginning of every down. It was tough enough to pay attention to the spot of the ball for the downs I was going to record.
With those rules in mind, I set out to record the field position whenever possession changed. I thought that the position the offense wound up its possession at, over time, might correlate with who won the game. Less overall variance in final position might mean that team had less moments where it under-performed and lost possession nearer to its own endzone.
Of course, it could also mean that the team never reached the endzone for a touchdown. In fact, if the offense played the whole game between their opponents 45 and 50 yard line it would have little variation in field position…but also probably wouldn’t score much. Maybe a combination of better field position (higher median field position) and low variation in field position would indicate who won the game. I thought it might. Let’s see if I was right.
Data Collection: Nuts and Bolts
Next, I quickly drew a continuous data collection sheet. It looked like this:
Sounds fancy, but obviously it isn’t. That’s an important tool for you when you go to collect continuous data right from your process: the continuous data collection sheet can be very simple and very easy to use.
Really, that was about it. I went through the game watching, like you, the Patriots fall way behind for the first two quarters. After some Lady Gaga halftime show (with drones!) I looked at the data and noticed something interesting.
The Patriots data on distance from the endzone seemed to demonstrate less variance than the Falcons. (I’ll show you the actual data collection sheets in a moment.) It was odd. Yes, they were VERY far behind. Yes there had been two costly turnovers that lead to the Falcons opening up a huge lead. But, strangely, in terms of moving the ball and getting closer to the endzone based on their own offense, the Patriots were actually doing better than the Falcons. Three people around me pronounced the Patriots dead and one even said we should change the channel.
If you’ve read this blog before, you know that one of the key beliefs it describes is that data is most useful when it can change our minds. These data, at least, made me unsure if the game was over.
As you know (no spoiler alert really by now) the game was far from over and the Patriots executed one of the most impressive comebacks (if not the most impressive) in Super Bowl history. Data collected and wings eaten without difficulty! Check and check.
Here are the final data collection sheets:
Notice the number in parenthesis next to the distance from the endzone when possession changed? That number is the possession number the team had. So, 52(7) means the Falcons were 52 yards away from the Patriots endzone when they punted the ball on their seventh possession of the game. An entry like 0(10) would mean that the team scored a touchdown (0 yards from opposing team’s endzone) on their tenth possession.
Notice that collecting data this way and stacking similar numbers on top of each other makes a histogram over time. That’s what let me see how the variation of the Patriot’s final field position was smaller than the Falcon’s by about halfway through the game.
Anything To Learn From The Data Collection?
Recently, I put the data into Minitab to see what I could learn. Here are those same histograms for each offense’s performance:
Notice a few items. First, each set of data do NOT deviate from the normal distribution per the Anderson-Darling test. (More info on what that means here.) However, a word of caution: there are so few data points in each set that it can be difficult to tell which distribution they follow. I even performed distribution fitting to demonstrate that testing will likely show that these data do not deviate substantially from other distributions either. Again, it’s difficult to tell a difference because there just aren’t that many possessions for each team in a football game. In a Lean Six Sigma project, we would normally protect against this with a good sampling plan as part of our data collection plan but, hey, I had wings to eat! Here’s an example of checking the offense performance against other data distributions:
Just as with the initial Anderson-Darling test, we see here that the data do not deviate from many of these other distributions either. Bottom line: we can’t be sure which distribution it follows. Maybe the normal distribution, maybe not.
In any event, we are left with some important questions. Notice the variance exhibited by the Patriots offense versus the Falcons offense: this highlights that the Patriots in general were able to move the ball closer to the Falcons endzone by the time the possession changed (remember that turnovers aren’t included). Does that decreased variation correlate with the outcome of every football game? Can it be used to predict outcomes of games? I don’t know…at least not yet. After all, if stopping a team inside their own 10 yard line once or twice was a major factor in predicting who won a game, well, that would be very useful! If data is collected by the league on field position, we could apply this idea to previous games (maybe at half time) and see if it predicts the winner routinely. If it did, we could apply it to future games.
In the case of Super Bowl LI, the Patriots offense demonstrated a better median field position and less variation in overall field position compared to the Falcons.
Of course, remember this favorite quote:
All models are wrong, but some are useful. — George E.P. Box (of Box-Cox transform fame)
Final Recommendations (How To Eat Wings AND Collect Data)
More importantly, this entry highlights a few interesting tools for data collection for your healthcare quality project. At the end of the day, in order to continue all the things you have to do and collect good data for your project, here are my recommendations:
(1) get data right from the process, not a warehouse or after it has been cleaned.
(2) use continuous data!
(3) remember the continuous data check sheet can be very simple to set up and use
(4) when you create a data collection plan, remember the sample size calculation & operational definition!
(5) reward those who collect data…maybe with wings!
David Kashmer is a trauma surgeon and Lean Six Sigma Master Black Belt. He writes about data-driven healthcare quality improvement for TheHill.com, Insights.TheSurgicalLab.com, and TheHealthcareQualityBlog.com. He is the author of the Amazon bestseller Volume To Value, & is especially focused on how best to measure value in Healthcare.