Without Data, You Just Have An Opinion

By:  David M. Kashmer MD MBA MBB (@DavidKashmer)


Do you agree with the thought that Six Sigma is 80% people and 20% math?  Whether or not you do, it’s important to realize that the 20% of the process which is math is VERY important.  As we discussed in other posts, the virtues of basing decisions on good data rather than your gut, social pressure, or other whims can’t be overstated.  As usual, we’re not saying that “feelings” and soft skills are unimportant; in fact, they’re very important.  Just as data alone isn’t enough (but is a key ingredient in consistent improvement) so too are feelings/intuition not enough when applied on their own.  Here, let’s explore an example of what good data analysis looks like–after all, without the engine of good data analysis, the quality improvement machine can’t run.


Starts With Good Data Collection

If the situation of your quality improvement project is not set up properly–well, let’s just say it’s unlikely to succeed.  We’ve discussed, here, the importance of selecting what data you will collect.  We’ve referenced how to setup a data collection plan (once over lightly) including sample size and types of endpoints.


It’s possible that the importance of setting things up properly can be overstated–but I think it’s very unlikely.  The key to the rest of the analysis we will discuss is that we have a good sample of appropriate size that collects data on the entire process we need to represent.  Yes, colleagues, that means data from the times it’s tougher to collect as well such as nights and weekends.


Requires A Clear Understanding Of What The Data Can (and Can’t) Say

The ball gets dropped, on this point, a lot.  In an earlier entry, we’ve described the importance of knowing whether, for example, your continuous data are normally distributed.  Does it make a difference?  No, it makes perhaps the difference when you go to apply a tool or hypothesis test to your data.  Look here.


Other important considerations come from knowing the limits of your data.  Were the samples representative of the system at which you’re looking?  Is the sample size adequate to detect the size of the change for which you’re looking?


You need to know what voices the data have and which they lack.


Nowadays, Often Requires Some Software

I’m sure there’s some value to learning how to perform many of the classic statistical tests by hand…but performing a multiple regression by hand?  Probably not a great use of time.  In the modern day, excellent software packages exist that can assist you in performing the tool application.


WARNING:  remember the phrase garbage in, garbage out.  (GIGO as it is termed.) These software packages are in no way a substitute for training and understanding of the tools being used.  Some attempt to guide you through the quality process; however, I haven’t seen one yet that protects you completely from poor analysis.  Also, remember, once the tool you are using spits out a nice table, test statistic, or whatever it may show:  you need to be able to review it and make sure it’s accurate and meaningful.  Easily said and not always easily done.


Two of the common, useful packages I’ve seen are SigmaXL and Minitab (with its quality suite).  SigmaXL is an Excel plug-in that makes data analysis directly from your Excel very straightforward.


Means You Need To Select The Correct Tool

We explored, here, the different tools and how they apply to your data.  (There’s a very handy reference sheet at the bottom of that entry.) If you’ve done the rest of the setup appropriately, you can select a tool to investigate the item on which you want to drill down.  Selecting the correct tool is very straightforward if the data setup and collection are done properly, because it’s almost as if you’ve reverse engineered the data collection from what it will take to satisfy modern statistical tools.  You’ve made the question and data collection which started all of this into a form that has meaning and can be answered in a rigorous fashion by common tools.


Allows A Common Understanding Of Your Situation Beyond What You “Feel”

This is my favorite part about data analysis:  sometimes it really yields magic.  For example, consider a trauma program where everything feels fine.  It’s pretty routine, in fact, that staff feel like the quality outcomes are pretty good.  (I’ve been in that position myself.) Why do we see this so commonly?  In part, it’s because services typically perform at a level of quality that yields one defect per every thousand opportunities.  Feels pretty good, right?  I mean, that’s a whole lot of doing things right before we encounter something that didn’t go as planned.


The trouble with this lull-to-sleep level of defects is that it is totally unacceptable where people’s lives are at stake.  Consider, for example, that if we accepted the 1 defect / 1000 opportunities model (1 sigma level of performance) that we would have one plane crash each day at O’Hare airport.  Probably not ok.


Another common situation seen in trauma programs concerns timing.  For instance, whatever processes are in place may work really well from 8AM until 5PM when the hospital swells with subspecialists and other staff–but what about at night?  What about on weekends?  (In fact, trauma is sometimes called a disease of nights and weekends.) Any data taken from the process in order to demonstrate performance MUST include data from those key times.  Truly most quality improvement projects in Trauma and Acute Care Surgery must focus on both nights and weekends.


So here again we have the tension between how we feel about a process and what our data demonstrate.  The utility of the data?  It gives us a joint, non-pejorative view on our performance and spurns us toward improvement.  It makes us look ourselves squarely in the eye, as a team, and decide what we want to do to improve or it tells us we’re doing just fine.  It puts a fine point on things.


Last, good data has the power to change our minds.  Consider a program that has always felt things are “pretty good” but has data that say otherwise.  The fact that data exist gives the possibility that the program may seek to improve, and may recover from its PGS (Pretty Good Syndrome).  In other words, part of the magic of data is that it has the power, where appropriate, to change our minds about our performance.  Maybe it shows us how we perform at night–maybe it shows us something different than we thought.  It may even tell us we’re doing a good job.


At The End Of The Day, Your Gut Is Not Enough

Issues with using your “gut” or feelings alone to make decisions include such classic problems as the fundamental attribution error, post-facto bias, and plain old mis-attribution.  It was DaVinci, if I recall, who said that “The greatest deception men suffer is from their own opinions.” We have tools, now, to disabuse ourselves of opinion based on our experience only–let’s use them and show we’ve advanced beyond the Renaissance.  So now we come to one of the “battle cries” of Six Sigma:  without data, you just have an opinion.  Opinions are easy and everyone has one–now, in high stakes situations, let’s show some effort and work to make actual improvement.