How To Collect Quality Improvement Data Without Putting Down Your Wings During the Superbowl

David Kashmer (@DavidKashmer)

Of the many barriers we face while trying to improve quality in healthcare, none is perhaps more problematic than the lack of good data.  Although everyone seems to love data (I see so much written about healthcare data) it is often very tough to get.  And when we do get it, much of the data we get are junk.  It’s not easy to make meaningful improvements based on junk data.  So, what can we do to get meaningful data for healthcare quality improvement?

In this entry, I’ll share some tools, tips, & techniques for getting meaningful quality improvement data from your healthcare system.  I’ll share how to do that by telling a story about Super Bowl LI…

The Super Bowl Data Collection

About ten minutes before kickoff, I had a few questions about the Super Bowl.  I was wondering if there was a simple way to gauge the performance of each team and make some meaningful statements about that performance.

When we do quality improvement projects, it’s very important to make sure it’s as easy as possible to collect data.  I recommend collecting data directly from the process rather than retrospectively or from a data warehouse.  Why?  For one, I was taught that the more filters the data pass through the more they are cleaned up or otherwise altered.  They tend to lose fidelity and a direct representation of the system.  Whether you agree or not, my experience has definitely substantiated that teaching.

The issue with that is how do I collect data directly from the system?  Isn’t that cumbersome?  We don’t have staff to collect data (!) Like you, I’ve heard each of those barriers before–and that’s what makes the tricks and tools I’m about to share so useful.

So back to me then, sitting on my couch with a plate of wings and a Coke ready to watch the Super Bowl.  I wanted data on something that I thought would be meaningful.  Remember, this wasn’t a DMAIC project…it was just something to see if I could quickly describe the game in a meaningful way.  It would require me to collect data easily and quickly…especially if those wings were going to get eaten.

Decide Whether You’ll Collect Discrete or Continuous Data

So as the first few wings disappeared, I decided about what type of data I’d want to collect.  I would definitely collect continuous data if at all possible.  (Not discrete.)  That part of the deciding was easy.  (Wonder why?  Don’t know the difference between continuous and discrete data?  Look here.)

Ok, the next issue was these data had to be very easy for me to get.  They needed to be something that I had a reasonable belief would correlate with something important.  Hmmm…ok, scoring touchdowns.  That’s the whole point of the game after all.

Get A Clear Operational Definition Of What You’ll Collect

As wings number three and four disappeared, and the players were introduced, I decided on my data collection plan:

  1. collect how far away each offense was from scoring a touchdown when possession changed
  2. each data point would come from where ball was at start of 4th down
  3. interceptions, fumbles, or change of possession (like an interception) before 4th down would NOT recorded (I’ll get to why in a minute.)
  4. touchdowns scored were recorded as “0 yards away”
  5. a play where a field goal was attempted would be recorded as the where the ball was on the start of the down

Of course, for formal quality projects, we would collect more than just one data point.  Additionally, we specify exactly the operational definition of each endpoint.

We’d also make a sample size calculation.  Here, however, I intended to collect every touchdown and change of possession where a team kicked away on fourth down or went for it but didn’t make it.  So this wasn’t a sample of those moments.  It was going to be all of them.  Of course, they don’t happen that often.  That was a big help here, because they can also be anticipated.  That was all very important so I could eat those wings.

Items like interceptions, fumbles, and other turnovers can not be anticipated as easily.  They also would make me have to pay attention to where the ball was spotted at the beginning of every down.  It was tough enough to pay attention to the spot of the ball for the downs I was going to record.

With those rules in mind, I set out to record the field position whenever possession changed.  I thought that the position the offense wound up its possession at, over time, might correlate with who won the game.  Less overall variance in final position might mean that team had less moments where it under-performed and lost possession nearer to its own endzone.

Of course, it could also mean that the team never reached the endzone for a touchdown.  In fact, if the offense played the whole game between their opponents 45 and 50 yard line it would have little variation in field position…but also probably wouldn’t score much.  Maybe a combination of better field position (higher median field position) and low variation in field position would indicate who won the game.  I thought it might.  Let’s see if I was right.

Data Collection:  Nuts and Bolts

Next, I quickly drew a continuous data collection sheet.  It looked like this:

Figure 1: Example of a continuous data collection sheet.


Sounds fancy, but obviously it isn’t.  That’s an important tool for you when you go to collect continuous data right from your process:  the continuous data collection sheet can be very simple and very easy to use.

Really, that was about it.  I went through the game watching, like you, the Patriots fall way behind for the first two quarters.  After some Lady Gaga halftime show (with drones!) I looked at the data and noticed something interesting.

The Patriots data on distance from the endzone seemed to demonstrate less variance than the Falcons.  (I’ll show you the actual data collection sheets in a moment.) It was odd.  Yes, they were VERY far behind.  Yes there had been two costly turnovers that lead to the Falcons opening up a huge lead.  But, strangely, in terms of moving the ball and getting closer to the endzone based on their own offense, the Patriots were actually doing better than the Falcons.  Three people around me pronounced the Patriots dead and one even said we should change the channel.

If you’ve read this blog before, you know that one of the key beliefs it describes is that data is most useful when it can change our minds.  These data, at least, made me unsure if the game was over.

As you know (no spoiler alert really by now) the game was far from over and the Patriots executed one of the most impressive comebacks (if not the most impressive) in Super Bowl history.  Data collected and wings eaten without difficulty!  Check and check.

Here are the final data collection sheets:

Figure 2: Data collection sheet. Patriots offense performance.
Figure 3: Data collection sheet. Falcons offense performance.

Notice the number in parenthesis next to the distance from the endzone when possession changed?  That number is the possession number the team had.  So, 52(7) means the Falcons were 52 yards away from the Patriots endzone when they punted the ball on their seventh possession of the game.  An entry like 0(10) would mean that the team scored a touchdown (0 yards from opposing team’s endzone) on their tenth possession.

Notice that collecting data this way and stacking similar numbers on top of each other makes a histogram over time.  That’s what let me see how the variation of the Patriot’s final field position was smaller than the Falcon’s by about halfway through the game.

Anything To Learn From The Data Collection?

Recently, I put the data into Minitab to see what I could learn.  Here are those same histograms for each offense’s performance:

Figure 4: Summary of Patriots offense performance.
Figure 5: Summary of Falcons offense performance.

Notice a few items.  First, each set of data do NOT deviate from the normal distribution per the Anderson-Darling test.  (More info on what that means here.) However, a word of caution:  there are so few data points in each set that it can be difficult to tell which distribution they follow.  I even performed distribution fitting to demonstrate that testing will likely show that these data do not deviate substantially from other distributions either.  Again, it’s difficult to tell a difference because there just aren’t that many possessions for each team in a football game.  In a Lean Six Sigma project, we would normally protect against this with a good sampling plan as part of our data collection plan but, hey, I had wings to eat!  Here’s an example of checking the offense performance against other data distributions:

Figure 6: Testing Patriots offense performance to determine whether it deviates substantially from these known distributions.

Just as with the initial Anderson-Darling test, we see here that the data do not deviate from many of these other distributions either.  Bottom line:  we can’t be sure which distribution it follows.  Maybe the normal distribution, maybe not.

In any event, we are left with some important questions.  Notice the variance exhibited by the Patriots offense versus the Falcons offense:  this highlights that the Patriots in general were able to move the ball closer to the Falcons endzone by the time the possession changed (remember that turnovers aren’t included).  Does that decreased variation correlate with the outcome of every football game?  Can it be used to predict outcomes of games?  I don’t know…at least not yet.  After all, if stopping a team inside their own 10 yard line once or twice was a major factor in predicting who won a game, well, that would be very useful!  If data is collected by the league on field position, we could apply this idea to previous games (maybe at half time) and see if it predicts the winner routinely.  If it did, we could apply it to future games.

In the case of Super Bowl LI, the Patriots offense demonstrated a better median field position and less variation in overall field position compared to the Falcons.

Of course, remember this favorite quote:

All models are wrong, but some are useful.  — George E.P. Box (of Box-Cox transform fame)

Final Recommendations (How To Eat Wings AND Collect Data)

More importantly, this entry highlights a few interesting tools for data collection for your healthcare quality project.  At the end of the day, in order to continue all the things you have to do and collect good data for your project, here are my recommendations:

(1) get data right from the process, not a warehouse or after it has been cleaned.

(2) use continuous data!

(3) remember the continuous data check sheet can be very simple to set up and use

(4) when you create a data collection plan, remember the sample size calculation & operational definition!

(5) reward those who collect data…maybe with wings!


David Kashmer is a trauma surgeon and Lean Six Sigma Master Black Belt.  He writes about data-driven healthcare quality improvement for,, and  He is the author of the Amazon bestseller Volume To Value, & is especially focused on how best to measure value in Healthcare.




2 Reasons Rudolph Is The Most Famous Reindeer Of All

By:  David M. Kashmer, MD MBA (@DavidKashmer)


Yes, you know Dasher (and Dancer and Prancer and Blitzen…) but why, when you think of Santa’s reindeer, is there really just one who’s top-of-mind?  Here we explore what makes Rudolph the Red-Nosed Reindeer, and what his classic tale can tell you about your startup business.




When the Montgomery Ward department store wanted to create its own coloring book in 1939, the tale of Rudolph was born.  What is it about the character that has made Rudolph “the most famous reindeer of all”?


Rudolph is unique.  Sure, he was laughed at.  Aspersions were cast.  Yet, like it or not, Rudolph has one very obvious feature that is ingrained into what he is.


Michael Porter, and other modern thinkers in business strategy, espouse this idea of competing on uniqueness.  Look here.  There was no way Rudolph could successfully compete with other reindeer on their terms.  What reindeer could dance like Prancer, or pull that sleigh with the strength of Blitzen?  (Ok, who knows if Prancer dances and Blitzen is strong–but, I mean, come on:  Prancer and Blitzen!) The key is to find some difference that it can preserve.  Just as Rudolph has a singular, unique feature, much of modern strategic thinking for startup businesses focuses on being something special…





…and being something unique, alone, is not enough.  Rudolph’s nose generally got him laughed at until it did something incredibly useful.  Once that red proboscis had saved the day, it didn’t seem so silly to have a large, red nose.


The ability to translate uniqueness into value is an important essence of building competitive advantage.


After the holiday season, when we all get back to work, it may be worthwhile to spend some time thinking of exactly what the “big red nose” is for you or your business.


How, exactly, will you design a valuable uniqueness into your startup?  How will you find what it is that makes your team go to the front of the pack?


Until then, Happy Holidays from all of us at the blog.

How To Run Thanksgiving As A Startup


By:  David Kashmer, MBA (@DavidKashmer)


Did you know that thinking about Thanksgiving as a startup just might help us focus on what’s really valuable about this great holiday?  Consider this as you take some time to think about how you’d startup Thanksgiving in your own unique way:


Step 1: What’s The Value Here Anyway?


Before you do ANYTHING, you need to decide what Thanksgiving means to you. What’s the value proposition? Is it the food, the time with family, the football game on TV? What is it exactly? It’s from that central decision that everything else flows…and I won’t answer it for you. Only you can decide, and decide you must, before moving on with your Startup Turkey Day.



Step 2: Build the Team


Ok ok…many of us who have started a business before think that you need at LEAST two things to make for a truly great business model: a great team AND a great idea. Which is more important? Well, I don’t think that question really matters! Because, experience teaches, if you don’t have the right team then there’s no idea that’s good enough to fly. If you don’t have at least a “good enough” idea then a great team can’t make it. However, I have seen a truly awesome team take the most marginal idea and make it gobble up the competition…


So, speaking of flight (and birds), what about Thanksgiving? (Ok the bad puns end here…) What kind of team will you need to execute the value proposition that is Thanksgiving?


Did you know, for example, that there’s evidence about startup team size and eventual success? Consider, for that reason, keeping the team size to 3 or 4. More on that here.


What about family? Should they be involved in your startup turkey day? Well, in general, I’d say DON’T start a business with your family(!) You can read more about it here. Yet, in this case, well…I mean it’s Thanksgiving! Probably not a good idea to keep the family out!


…and, hey, be sure to involve the family members who have complimentary skill sets! Maybe one person takes care of scheduling the when and where, another handles the setup of the actual event (tables and chairs), while yet another handles the food. Don’t forget that one family member who makes the stuffing you like so much.



Step 3: Use Some Tools


Ok, so now the three (or four) people of the team are all together. How will you all deliver the value that is Thanksgiving? Now it’s time to use a tool to make sure you are all on the same page as to what you’ll be doing, how it will deliver value, and how you’ll do all those things to make your Thanksgiving a success. For example, one tool that works great (WAY better than those old fashioned business plans that dissolve as soon as you make first contact with the market) is called the business model canvas. You can read more about the virtues of this great startup tool here.


As your team turns to the tool, think about what makes your startup unique and therefore somehow more valuable…take a moment to decide what your special sauce, exactly, will be…even if it is just cranberry.


Beneath, I’ve included an example of a partially completed business model canvas for a fictitious local startup, the Boxed Up Bird Association (BUBA pronounced “Bubba”).


Pretend your team of 4 decided that the central value proposition of thanksgiving was about sharing the feeling of thanksgiving with as many as possible, getting together families to share the holiday spirit, and uniting them over a tasty meal…Well, in that case, maybe your canvas would look like this:


A Sample Business Model Canvas for The Boxed Up Bird Association (BUBA, pronounced “Bubba”), a 501c3


BUBA seeks to bring Thanksgiving to multiple people by facilitating get-togethers, over food, for families that are separated somehow. Maybe BUBA, like in the canvas above, would help families share Thanksgiving when they are geographically separated…a military family could share thanksgiving via telepresence (eg gotomeeting or a similar platform) while BUBA sent them each a boxed version of a Thanksgiving meal to share. Maybe BUBA would do this by arranging all of the online get-together, the meal, and even the highlights of the football game. (Don’t forget to avoid spoilers across multiple time zones!)


While these families would get to enjoy Thanksgiving, so would yours. Seems like everyone would get what they need with the feelings of Thanksgiving…



Step 4: Like Your Idea?…Now Try And Kill It!


Just as many great, much-loved turkeys meet their ends around Thanksgiving, it’s time for you to try to end your idea now…while it’s on paper and before you’ve spent any resources on it!


Hey, we all fall in love with our own ideas. Now that you and the team have explored how you will deliver your Thanksgiving value proposition, it’s a good idea to try and figure out all the ways it won’t ever work. I tend to start with the Costs and Revenue section at the bottom of the business model canvas.  Maybe you’ll start with what it will take to get “customers” to participate in your greatest-Turkey-Day-ever experience.  Do customers even exist for this sort of thing?  And how could you know or find out?


For costs, I use worst-case scenario costs. Seriously, we’re dealing with a brand new, never-been-tried scenario here. It may even be an industry you haven’t been in before. You lack info. There’s uncertainty. Plan for the worst as you figure out what costs it will take to make BUBA (or your personal Turkey Day Plan) go!


I recommend completing the canvas with the costs you will have each month and the one time startup costs you will have. It will help you figure out how much funding you will need for the startup, and figure out cash flow issues in the future. (More on that later in the entry.)


Under revenue part of the canvas…same deal. How will you collect revenue? How will you avoid cash flow problems? What margin (after tax) will you need to make it worth your time? Can you justify the price you’d have to charge to get that margin? Are you ready to work this hard? Do you recognize the seasonal component of BUBA (or your startup) in your business canvas and cost structure? Tough questions! Again, plan for the worst so you don’t come up short. Figure out how much of your service / product / awesome Turkey Day Experience you will have to sell in order to at least cover your monthly costs (rough break even calculation). And, again, remember that Turkey Day is a seasonal experience…


The idea behind all this is the recognition that your startup is an experiment. You have unknowns that you should try to quantify as you constantly (and quickly!) learn. However, despite trying to minimize the unknown, you never really know how things will play until you run the test and loose your Turkey day startup on the real world. So, plan accordingly!


(Maybe, if you’re lucky, you’ll stumble on something that takes the Turkey Day experience and extends it throughout the rest of the year so that you can have income then too…like “Holiday In A Box” or something.  Who knows.)


If your idea survives this ruthless barrage of cost-loading and planning for worst-case-revenue, then move on to the next step. If it doesn’t, or it looks bad now (a wounded turkey but one that’s not dead yet in the best Monty Python sense), re-think what you’re doing! It’s much easier to pivot / adapt your business on paper before you’ve spent any money.


Step 5: Align The Team


Guess what, if BUBA (or your personal Turkey Day startup) takes off it is going to be TOUGH to keep all the players on the team aligned. From experience, let me say that now is the time to make sure you and all the teammates will be rowing the same direction at each stage of the company’s growth. When Aunt Maud’s awesome Thanksgiving whipped potatoes are a huge success 5 years in the future, that is NOT the time to sort out which team member contributed to the Turkey Day’s startup success. Do it now!


How do you keep everyone aligned? There are some really great tools for that such as dynamic ownership equity. Check them out and use them now! Look here. After all, one of the most common reason startups fail is mis-alignment of team members. Use the tools and avoid this terrible issue that seems to magnify as the startup goes on.


Step 6: Funding Turkey Day


Wow…all this because you decided to think about what Thanksgiving meant to you, and then you realized that the feeling of family and togetherness should be shared with everyone. You really signed yourself (and your team) up for something here, didn’t you?


Well, now, if you’ve made it this far it’s time to find funding for your startup Turkey day. How much funding do you need? Of course, it depends on the idea and your business canvas…


A classic (and very functional) recommendation is that you should fund your new business with 4-5 month’s worth of “runway”. That means, pretend your business won’t have ANY revenue for 4-5 months and yet will incur all of its costs. After 4-5 months, if your fledgling startup (sorry about another bird reference—promise about no puns violated) isn’t at least breaking even then re-evaluate the startup. Is time to pivot / adapt the business, or is it time to call this experiment unsuccessful and to move on?


Only you can decide.


Conclusion: However You Do It—Happy Thanksgiving!


If nothing else, putting this much thought into Thanksgiving allows for deep learning about what you really value in the day. Is it time with family? Is it giving thanks for what we have and sharing that feeling with others? Maybe it’s the food and football! Whatever it is for you and yours, have a Happy Thanksgiving from me and your friends at the fictitious Thanksgiving startup non-profit…The Boxed Up Bird Association (“Bubba”)!






Let’s Celebrate July the 2nd!?


By:  David Kashmer, MD MBA (@DavidKashmer)


Adams Thought We’d Celebrate July 2nd…

Did you know that at least one of America’s progenitors was confident that we would be celebrating July the 2nd as Independence Day?  Well, I’m no historian, but…


John Adams’ famous letter of July 3, 1776, in which he wrote to his wife Abigail what his thoughts were about celebrating the Fourth of July is found on various web sites but is usually incorrectly quoted. Following is the exact text from his letter with his original spellings:

The Second Day of July 1776, will be the most memorable Epocha, in the History of America. I am apt to believe that it will be celebrated, by succeeding Generations, as the great anniversary Festival. It ought to be commemorated, as the Day of Deliverance by solemn Acts of Devotion to God Almighty. It ought to be solemnized with Pomp and Parade, with Shews, Games, Sports, Guns, Bells, Bonfires and Illuminations from one End of this Continent to the other from this Time forward forever more. You will think me transported with Enthusiasm but I am not. I am well aware of the Toil and Blood and Treasure, that it will cost Us to maintain this Declaration, and support and defend these States. Yet through all the Gloom I can see the Rays of ravishing Light and Glory. I can see that the End is more than worth all the Means. And that Posterity will tryumph in that Days Transaction, even altho We should rue it, which I trust in God We shall not. (The Book of Abigail and John: Selected Letters of the Adams Family, 1762-1784, Harvard University Press, 1975, 142).

Writing that letter was an act of celebration.

Above excerpt is from You can find similar info all over the web on Google.  I first learned of John Adams’ thoughts on July 2 when reading his letter to Abigail as cited above.


The Players Involved Were People

I really enjoy historical instances that demonstrate how the people involved were human beings.  Have you heard of the one where, when crossing the Delaware, one of the participants (owing to his size) could have capsized the boat when stepping onboard?  George Washington warned him (Henry Knox, a man of considerable girth) not to get on too fast or he’ll “swamp [overturn] the boat.” (Washington’s actual words were much more colorful as the story goes.)


Instances, like these, demonstrate just how messy and human History can be.  So to all of the celebrators out there making history, and Mr. Adams too:


Happy Fourth of July.



Santa Should Have Done A Lean Startup




For Mr. Kringle, starting up a global manufacturing and distribution network in 2013 would be challenging.  Looking back on his success, a nice instance of where success is not defined by income by the way, it is easy to imagine that Santa’s service emerged fully formed and streamlined.  (If by streamlined we mean hand-delivering all toys from house to house the world over while relying on flying reindeer who are, no doubt, apt to get sick and / or just not fly.) Lean startups leverage a sophisticated understanding of how to start a company in order to get things done.  Nowadays, no doubt, Santa would startup Lean.


Although Santa’s team has deep roots in history, the lean startup does not.  The lean startup is a relatively recent phenomenon that conjures thoughts on a certain attitude, philosophy and tool set for how to start a business both effectively and quickly. Three of the most important tools of the lean start up include the business model canvas, the minimum viable product and an iterative approach to the business model along with number 4, customer development.  These modern tools would have helped Santa quickly determine whether people liked his service & products.  He would have learned quickly what to focus on even if his market changed.


Santa may have used the business model canvas and the concept of the minimum viable product.  We spoke before about the business model canvas and we have also described the concept of the minimum viable product (which is a product offer that has the least amount of features a customer will purchase). Perhaps, for example, children would have accepted receiving gifts over two, three, or seven days instead of all in one night.  This may or may not have made life easier for Santa.  Again, do children need Santa to deliver everything himself?  Who knows now, yet at the beginning Santa could have learned what people would accept and what he could create most easily.


Let’s focus on the idea of the minimum viable product (MVP) more closely.  The MVP is a useful concept because it allows you to iterate the product quickly, it is usually more straightforward to build or create as it has less features, and it helps you test the market in a rapid fashion.  Santa could have learned a great deal from his Lean experiment.


Before Santa built toys to spec and delivered them all in one night via reindeer (and often chimney), what child could have even imagined such a thing?  There is a classic line from Henry Ford that is if he had “asked their customers what they wanted they would have said a faster horse.”  This indicates that, sometimes, the market doesn’t know exactly what it wants until it sees it.  Santa could have developed the demand for his service and perhaps spread across the world more quickly.


It is just that customers in the market have points of pain, issues, or other things they need satisfied.  They will tell you soon enough if your product  helps ease their pain.  One of the functions of our team of angel investors is to give some guidance on some other unique tools that can really lean the process for you to get your business. We have a series of several steps that are changed depending on your unique business idea and model.  We have both established and new businesses benefit from these steps, and we have mapped out certain tools that can also eliminate completely sources of overhead that can be barrier to your startup.  Who knows if our experience could have helped Santa–after all he did very well on his own if we measure by market share alone.


In the end, Santa may have been able to utilize modern tools of the startup to great effect.  Using a business canvas and MVP, he may have been able to learn that there is no need to push his reindeer so throughout one night once per year.  Perhaps he would have used another service or independent contractors to help make deliveries.  However, as historians will tell us, it is often unfair to use modern knowledge to criticize the ancients–after all they did very well with what they had.  So it is with Santa:  even though he may have a different model if he had used the Lean startup tools, he’s done very well for us all.

Happy Holidays.


To read more about Lean startup tools:  on the main home page, click the plus sign and select from the archives.  If you have a moment and interest, go back and review the first blog entry in the series that highlights the business model canvas as an effective way to create and iterate your business model.