How To Collect Quality Improvement Data Without Putting Down Your Wings During the Superbowl

David Kashmer (@DavidKashmer)

Of the many barriers we face while trying to improve quality in healthcare, none is perhaps more problematic than the lack of good data.  Although everyone seems to love data (I see so much written about healthcare data) it is often very tough to get.  And when we do get it, much of the data we get are junk.  It’s not easy to make meaningful improvements based on junk data.  So, what can we do to get meaningful data for healthcare quality improvement?

In this entry, I’ll share some tools, tips, & techniques for getting meaningful quality improvement data from your healthcare system.  I’ll share how to do that by telling a story about Super Bowl LI…

The Super Bowl Data Collection

About ten minutes before kickoff, I had a few questions about the Super Bowl.  I was wondering if there was a simple way to gauge the performance of each team and make some meaningful statements about that performance.

When we do quality improvement projects, it’s very important to make sure it’s as easy as possible to collect data.  I recommend collecting data directly from the process rather than retrospectively or from a data warehouse.  Why?  For one, I was taught that the more filters the data pass through the more they are cleaned up or otherwise altered.  They tend to lose fidelity and a direct representation of the system.  Whether you agree or not, my experience has definitely substantiated that teaching.

The issue with that is how do I collect data directly from the system?  Isn’t that cumbersome?  We don’t have staff to collect data (!) Like you, I’ve heard each of those barriers before–and that’s what makes the tricks and tools I’m about to share so useful.

So back to me then, sitting on my couch with a plate of wings and a Coke ready to watch the Super Bowl.  I wanted data on something that I thought would be meaningful.  Remember, this wasn’t a DMAIC project…it was just something to see if I could quickly describe the game in a meaningful way.  It would require me to collect data easily and quickly…especially if those wings were going to get eaten.

Decide Whether You’ll Collect Discrete or Continuous Data

So as the first few wings disappeared, I decided about what type of data I’d want to collect.  I would definitely collect continuous data if at all possible.  (Not discrete.)  That part of the deciding was easy.  (Wonder why?  Don’t know the difference between continuous and discrete data?  Look here.)

Ok, the next issue was these data had to be very easy for me to get.  They needed to be something that I had a reasonable belief would correlate with something important.  Hmmm…ok, scoring touchdowns.  That’s the whole point of the game after all.

Get A Clear Operational Definition Of What You’ll Collect

As wings number three and four disappeared, and the players were introduced, I decided on my data collection plan:

  1. collect how far away each offense was from scoring a touchdown when possession changed
  2. each data point would come from where ball was at start of 4th down
  3. interceptions, fumbles, or change of possession (like an interception) before 4th down would NOT recorded (I’ll get to why in a minute.)
  4. touchdowns scored were recorded as “0 yards away”
  5. a play where a field goal was attempted would be recorded as the where the ball was on the start of the down

Of course, for formal quality projects, we would collect more than just one data point.  Additionally, we specify exactly the operational definition of each endpoint.

We’d also make a sample size calculation.  Here, however, I intended to collect every touchdown and change of possession where a team kicked away on fourth down or went for it but didn’t make it.  So this wasn’t a sample of those moments.  It was going to be all of them.  Of course, they don’t happen that often.  That was a big help here, because they can also be anticipated.  That was all very important so I could eat those wings.

Items like interceptions, fumbles, and other turnovers can not be anticipated as easily.  They also would make me have to pay attention to where the ball was spotted at the beginning of every down.  It was tough enough to pay attention to the spot of the ball for the downs I was going to record.

With those rules in mind, I set out to record the field position whenever possession changed.  I thought that the position the offense wound up its possession at, over time, might correlate with who won the game.  Less overall variance in final position might mean that team had less moments where it under-performed and lost possession nearer to its own endzone.

Of course, it could also mean that the team never reached the endzone for a touchdown.  In fact, if the offense played the whole game between their opponents 45 and 50 yard line it would have little variation in field position…but also probably wouldn’t score much.  Maybe a combination of better field position (higher median field position) and low variation in field position would indicate who won the game.  I thought it might.  Let’s see if I was right.

Data Collection:  Nuts and Bolts

Next, I quickly drew a continuous data collection sheet.  It looked like this:

Figure 1: Example of a continuous data collection sheet.


Sounds fancy, but obviously it isn’t.  That’s an important tool for you when you go to collect continuous data right from your process:  the continuous data collection sheet can be very simple and very easy to use.

Really, that was about it.  I went through the game watching, like you, the Patriots fall way behind for the first two quarters.  After some Lady Gaga halftime show (with drones!) I looked at the data and noticed something interesting.

The Patriots data on distance from the endzone seemed to demonstrate less variance than the Falcons.  (I’ll show you the actual data collection sheets in a moment.) It was odd.  Yes, they were VERY far behind.  Yes there had been two costly turnovers that lead to the Falcons opening up a huge lead.  But, strangely, in terms of moving the ball and getting closer to the endzone based on their own offense, the Patriots were actually doing better than the Falcons.  Three people around me pronounced the Patriots dead and one even said we should change the channel.

If you’ve read this blog before, you know that one of the key beliefs it describes is that data is most useful when it can change our minds.  These data, at least, made me unsure if the game was over.

As you know (no spoiler alert really by now) the game was far from over and the Patriots executed one of the most impressive comebacks (if not the most impressive) in Super Bowl history.  Data collected and wings eaten without difficulty!  Check and check.

Here are the final data collection sheets:

Figure 2: Data collection sheet. Patriots offense performance.
Figure 3: Data collection sheet. Falcons offense performance.

Notice the number in parenthesis next to the distance from the endzone when possession changed?  That number is the possession number the team had.  So, 52(7) means the Falcons were 52 yards away from the Patriots endzone when they punted the ball on their seventh possession of the game.  An entry like 0(10) would mean that the team scored a touchdown (0 yards from opposing team’s endzone) on their tenth possession.

Notice that collecting data this way and stacking similar numbers on top of each other makes a histogram over time.  That’s what let me see how the variation of the Patriot’s final field position was smaller than the Falcon’s by about halfway through the game.

Anything To Learn From The Data Collection?

Recently, I put the data into Minitab to see what I could learn.  Here are those same histograms for each offense’s performance:

Figure 4: Summary of Patriots offense performance.
Figure 5: Summary of Falcons offense performance.

Notice a few items.  First, each set of data do NOT deviate from the normal distribution per the Anderson-Darling test.  (More info on what that means here.) However, a word of caution:  there are so few data points in each set that it can be difficult to tell which distribution they follow.  I even performed distribution fitting to demonstrate that testing will likely show that these data do not deviate substantially from other distributions either.  Again, it’s difficult to tell a difference because there just aren’t that many possessions for each team in a football game.  In a Lean Six Sigma project, we would normally protect against this with a good sampling plan as part of our data collection plan but, hey, I had wings to eat!  Here’s an example of checking the offense performance against other data distributions:

Figure 6: Testing Patriots offense performance to determine whether it deviates substantially from these known distributions.

Just as with the initial Anderson-Darling test, we see here that the data do not deviate from many of these other distributions either.  Bottom line:  we can’t be sure which distribution it follows.  Maybe the normal distribution, maybe not.

In any event, we are left with some important questions.  Notice the variance exhibited by the Patriots offense versus the Falcons offense:  this highlights that the Patriots in general were able to move the ball closer to the Falcons endzone by the time the possession changed (remember that turnovers aren’t included).  Does that decreased variation correlate with the outcome of every football game?  Can it be used to predict outcomes of games?  I don’t know…at least not yet.  After all, if stopping a team inside their own 10 yard line once or twice was a major factor in predicting who won a game, well, that would be very useful!  If data is collected by the league on field position, we could apply this idea to previous games (maybe at half time) and see if it predicts the winner routinely.  If it did, we could apply it to future games.

In the case of Super Bowl LI, the Patriots offense demonstrated a better median field position and less variation in overall field position compared to the Falcons.

Of course, remember this favorite quote:

All models are wrong, but some are useful.  — George E.P. Box (of Box-Cox transform fame)

Final Recommendations (How To Eat Wings AND Collect Data)

More importantly, this entry highlights a few interesting tools for data collection for your healthcare quality project.  At the end of the day, in order to continue all the things you have to do and collect good data for your project, here are my recommendations:

(1) get data right from the process, not a warehouse or after it has been cleaned.

(2) use continuous data!

(3) remember the continuous data check sheet can be very simple to set up and use

(4) when you create a data collection plan, remember the sample size calculation & operational definition!

(5) reward those who collect data…maybe with wings!


David Kashmer is a trauma surgeon and Lean Six Sigma Master Black Belt.  He writes about data-driven healthcare quality improvement for,, and  He is the author of the Amazon bestseller Volume To Value, & is especially focused on how best to measure value in Healthcare.




Do You Know About Boston Snow & Special Cause Variation?

By:  David Kashmer, MD MBA (@DavidKashmer)

Lean Six Sigma Master Black Belt



Extreme Weather & The Six M’s




This is a recent photo of my car. Yes, I live and work in the greater Boston area.  Guess what–after this latest snowfall things only got worse.  That car is really buried now!


This latest snowfall gave me the opportunity to consider some of the interesting points about statistical process control and how to create systems in healthcare that work no matter the hour or condition–even when that condition is so bad that it spawns Twitter hashtags like #BosNOW & #bostonsnow.





Talk About An Organization Dedicated To Patient Safety (!)


During this latest winter storm, the organization in which I work has done remarkable things. Every morning, there is a group meeting which focuses on safety.  This daily meeting is built into the system, and happens even when there isn’t snow or some “hot topic” issue.  Lately, the meeting has included what we are doing to get our patients into and out of the hospital despite their comorbidities and the extreme weather. Every morning, as part of this safety huddle, each department reports off any safety concerns.


This focus on safety made me spend some time applying quality tools to how I get to work in the morning.  (Yes, you can do that!) Let me share with you some of the techniques I used to try to make sure I could effectively, and safely, get to work.  Let’s use these tools from the Six Sigma toolbox to highlight how they relate to extreme weather as a cause of variation.


The FMEA Highlights Things We Wouldn’t Have Thought About


First, I used the failure mode effect analysis (or FMEA) to figure out what all the ways were in which I could fail to get to work. The FMEA pointed out to me several ways in which I could be unable to do my job right at the onset of my time with the organization.  (I’m new to working here.)


As you know, coming into a new organization is a key time to build a team and quickly learn about the how and why regarding why things are the way they are. My personal failure mode affects analysis showed me that extreme weather was going to be a significant consideration.  Months ago, when I planned where to live, the FMEA made me recognize that weather was a significant concern that could impact my ability to perform.  Although snow was infrequent on a day to day basis, it could be severe.  The FMEA told the story.


How did I respond?  I made sure to find a place to live that had covered parking. I chose a place that was as close to the hospital as was practical. I also changed my car & purchased a used (pre-owned always sounds so much nicer but let’s call it what it is) four wheel drive vehicle. The FMEA really pointed me toward some key changes to make sure that I could even get to work. So, to sum up my personal portion of the story, the FMEA was very useful in pointing out to me some things that were necessary for me to even be able to get to work in the first place.  I wouldn’t have thought of these without the tool.


The Ishikawa, 6M’s, & Snow (!)


The FMEA is not the only useful tool of the component that can help us to design quality systems in health care. (And make sure we get to work!) Consider, again, the six sources of special cause variation. These are often called the six M’s or five M’s and one P. The six M’s include man (or people), materials (in the world of healthcare these are often patients with their attendant comorbidities), machine, method, Mother Nature, and management. Previously, I have discussed the six causes of special cause variation here.


Now, let’s take a moment to focus on Mother Nature. When we do a root cause analysis or similar quality meeting, we often make an Ishikawa or fishbone diagram. (More about those here.) The fishbone encompasses each of the factors in special cause variation. Again, one of these is Mother Nature. As mentioned previously here, we described (that once we’ve completed a fishbone diagram) we usually go back and label parts of it that we can and cannot influence. Parts that we can influence we label as C for control and parts that we cannot we label as N for noise (non-controllable). What is useful about this is that we can make a multiple regression analysis, and it can show us how much of the variation in the system we can directly control. (More on that here.)


As interesting as that may be, we should take a moment to describe that, even if we can’t control the weather, we can definitely plan for it. For example, throughput in the hospital changes greatly when people are unable to enter (and exit) the hospital.  This organization’s patient safety focused response plan is to be commended.


These Tools Answer Important Questions


Additionally, we may look to data and see how often we have weather extremes such as snowfall, heat, and other extreme variations in temperature and climate.  Important questions, such as “Does the hospital fill up with backlog?” or “Do patients not show up?” or even “Do both happen, and, if so, which effect wins out?” can be asked and answered with historic data.


We can then plan accordingly and make intelligent choices about what to do exactly when weather strikes. Again, I take a moment to compliment the organization of which I am a part. The team has a clear, patient-safety focused initiative to help patients enter and exit the hospital safely.


Utilization of a Y=f(x) (aka fishbone or Ishikawa) diagram may help us learn how weather impacts things such as hospital throughput. Insights like these come about when we take the time to understand how things like weather often lay behind other things we want to measure in our system.  A broad focus on all six causes of special cause variation yields insight that can allow us to help decrease outliers for issues like throughput.


I am confident that this health system and I will remember the impact of snow on throughput for a long time to come. For your organization it may be worthwhile, depending on where your organization is geographically, to focus on seemingly uncontrollable influencers that contribute to special cause variation.  Again, we may not be able to influence the weather, but we can definitely plan for it and see its effects in our models.


So, greetings from the greater Boston area and hope you are warm and safe wherever you are in the country. For questions, comments, or thoughts let me know beneath.

Call For New Authors



Hello All,


You may have noticed a recent update that explained how we are bringing on new writers to the blog.  Well, here’s your chance!


We have several new voices coming on board (you may have already met @GenYSurgeon) and more will follow soon.


If you’d like to add your voice to ours, we’re interested in talking with you.


Our team is particularly interested in once to twice weekly posts on these subjects:


(1) healthcare business model innovation

(2) personal stories of systems issues in healthcare

(3) statistical process control, Lean / Six Sigma, and other data-driven process improvement initiatives.

(4) “Big Data” and its applications in healthcare


The blog now has hundreds of unique readers each day, and we appreciate all our readers out there very much.


We now have the opportunity to add your voice (anonymously if you’d like) to the already excellent team at the blog.


Let us know if you’re interested in adding this forum to help spread your unique voice.  Don’t worry if you’re new to all this–we’ll help edit and format your work for posting.


Contact me at because our 5 open spots are filling soon.


Fresh Voices Welcome,




Magnet Designation Associated With Improved Hospital Outcomes: EAST Day Three

Naples continues to be sunny and filled with powerful talks from influential trauma surgeons.  Today’s EAST conference saw Dr. Tracy Evans, from Lancaster General Hospital, deliver a clear message regarding nursing Magnet designation and an association with mortality data.  Dr. Evans and colleagues showed data to support their contention that Magnet designation is associated with improved mortality at hospitals with existent trauma center designation.  Dr. Evans, at the morning scientific


paper session, clearly demonstrated data to substantiate her, and her colleagues’, experiences to date.


Dr Evans specifically touched on the the forces of magnetism, and described the impact of nursing care on trauma hospitals.  Dr. Evans highlighted data from 27 trauma centers in Pennsylvania that had been reviewed by the Pennsylvania Trauma Systems Foundation.  An important inclusion criterion from the study involved taking data only from level 1 and 2 centers with admissions greater than 500 qualifying patients per year.  Seventeen of these accepted centers were non-magnet hospitals and this portion of the group represented 42,000 patients.  Ten other centers were magnet designated and contributed 30,000 patients to the sample.  Dr. Evans and her team then used a multi-variate analysis to demonstrate 20% less mortality in magnet designated centers, which represented a statistically significant difference between the two types of centers.  Evans stressed that this mortality reduction is on-par with that seen among hospitals with trauma center designation versus those without.


Clearly, this fine study demonstrates correlation rather than causality.  Are we able to extrapolate that nursing magnet designation is causal for better quality? We are not.  However, we have learned that magnet designation is one more way in which participants in the healthcare system can get a sense of overall quality in a hospital type institution. Again, we are unable to tell whether magnet designation causes improved quality but we can readily think that having magnet designation is associated with the presence of increased quality.  This represents nice work by Dr Evans and her colleagues.imagejpeg_0


In attendance, as an impressive show of support, was the advanced practitioner team from Lancaster General and Dr. Fredrick Rogers, Trauma Medical Director at Lancaster General.  Dr. Evans, with this abstract submission and presentation, will be moving toward the “trifecta” for publication and awaits, with good probability of success, journal acceptance and publication.  This represents one more piece of evidence linking nursing standards to quality of care provided by an institution.  I, and other members of the blog team, look forward to the continued academic strength and data coming out of Lancaster General Health under the guidance of Dr. Rogers in association with his team.



Trauma & Emergency Surgeons Told To Include Themselves With General Surgery: EAST 2014 Day Two

Today’s 2014 EAST conference in Naples, Florida saw an improvement in the weather and continued excellent talks from recognized leaders in the field of Trauma and Acute Care Surgery.  Dr. David V. Feliciano gave the Scott B. Frame memorial lecture and advocated a return to previous practice principles. Included in his talk were the conservative ideas of representing trauma and acute care surgery as “no different than general surgery” and an admonishment to young trauma surgeons to avoid “separating themselves from general surgery”.  These and other conservative comments were signposts as Dr. Feliciano recreated the mental map he and others have used to achieve success in academic trauma and emergency surgery.


Dr. Feliciano advocated several useful ideas including the calculation of each individual’s Hirsch index (or H index).  The H index represents a measure of how many times the individual has been cited by others in publications.  He contrasted this against the typical impact factor for journals and said that the H index is utilized by promotions committees at academic centers.  In addition to useful specifics, Dr Feliciano gave other advice to young academic surgeons, including ways to avoid unproductive overcommitment to activities that do not add value to their careers.


Some of Dr. Feliciano’s other ideas emerged as he reported on a lifetime of career success. He advocated that young clinicians start their collections of famous papers early, and that the way ahead in terms of publication was the “trifecta” which includes:  submitting an abstract to a conference, a podium presentation of that work, and, finally, journal publication of that same research.


In the last 5-10 years, as we know on the blog, there has been an explosion of innovative online publication techniques.  Dr. Feliciano’s classic rendition of the pathway ahead for young surgeons did not focus on these newer models or disruptive business models for Surgery as a whole.  This left questions for some young surgeons such as whether the way ahead in years previous will continue as the way ahead in the future.  Again, some young surgeons were considering the new classic quotation that we must “evolve or die”.  How do the innovative systems that influence Surgery so much fit into the path for young surgeons?  Will promotions committees ever consider, as the Twitterverse has wondered, social media and associated publications as an influencer for promotion?


Regardless of the specific reaction to Dr. Feliciano’s excellent lecture, it is challenging to argue with a career and experience as successful as his has been given his lifetime of prominence in trauma and emergency surgery.  The young surgeons look forward to determining whether the way ahead in years previous will continue to be the way ahead he described given the new techniques, tools, and in light of the American College of Surgeons’ recent focus on innovation in Surgery.  Will the models for research and publication described in this year’s Frame lecture persist into the next decade, or will new techniques including business model innovation move regularly to the forefront?  Only time will tell, and, as for now, we are left with an exposition of a classic pathway ahead in academic surgery that leads to both academic and clinical strength.  For that we can all appreciate Dr. Feliciano’s excellent talk and look forward to EAST day 3.

2014 EAST Conference Begins In Naples, Florida


The Eastern Association for the Surgery of Trauma commenced its 2014 membership assembly in beautiful Naples, Florida, January 15th.  Popular topics this year included the immunomodulatory effects of red blood cells as one of the scientific, plenary presentations.  Interestingly, exhibitors this year varied from fledgling device companies to surgical staffing solutions.  Featured prominently was Emergency Surgical Staffing, LLC, a staffing solution meant to displace locum tenens surgery.  The EAST foundation president, Dr Fred Luchette gave his introductory speech which highlighted EAST’s various contributors.  The day saw numerous examples of scientific and business model innovation including new ideas in the world of basic science, and, as mentioned, device manufacturing. 


At this year’s conference, EAST continued the tradition of a conference-associated application for iPad and iPhone.  The app features the conference schedule and even photos of the scientific posters.  This year’s conference, in sunny Naples, has all the makings of another successful EAST endeavor.


Next year’s conference will be at Disney’s Contemporary in Orlando Florida.  We will be blogging daily from this year’s EAST conference, so join us for updates on the weather, scientific highlights and images of the people seen and heard around the EAST venue.