Healthcare Quality Podcast: 4 Types of Bad Metrics In Healthcare

By:  David Kashmer MD MBA FACS (@DavidKashmer) & Vivienne Neale (@SupposeIAm)

LinkedIn Profile here.


Listen to the podcast here.


DDD, bringing you the metrics behind the data. Here is your host, Vivienne Neale.

Hi, and welcome to DDD, which is Data Driven Decision Radio, episode six. My name is Vivienne Neale and I am delighted to be back with you. For those who have asked, my background is in education, training, broadcasting and social media. Of course, I am also a sometime patient curious to know what decisions are being taken in my name that might just affect me and of course thousands of people just like me. So, once again I am joined by David Kashmer, Chair of Surgery at Signature Healthcare. David is an expert in statistical process control, including Lean and Six Sigma. He has a special interest in new tools to improve healthcare, like gamification. David also edits and writes for a blog called So, hi David and welcome back.


Vivienne, thank you so much for the warm welcome and it’s great to be here again with you and the listeners.


Right, well I don’t know about you, but I’ve actually found something in the news this week that I thought you and the listeners might find quite interesting.


Vivienne, it would be great to hear. You always have interesting news stories for us and I’m interested to hear another.


DDD News


I have gone slightly left field here and I’ve decided to look at a story that actually has nothing to do with health care, but has a considerable amount to do with big data. So, in the UK we have an underground, or would I be better to call it a metro system?


[Laughter]. I think we would understand you with either word, Vivienne, but both are welcome. The underground or metro system in London is a great mode of transport, really amazing.


It is and it gets better and better. However, we are about to introduce all night trains and there is a little bit of an issue with the staff who are going to run them. There have been a series of strikes, including one in February 2004. You might say to me, “Well, excuse me, what is a strike on the London underground got to do with health or anything else?” Well, it was interesting because the University of Oxford and Cambridge did a joint study where they explored the data that is being collated by travellers when they use a swipe card, which you load money onto and then you can literally run through the underground all day and then at the end of the day it will calculate how much you’ve spent.


It’s the Oyster Card.


It is the Oyster Card, yes. 10 out of 10 [laughter]. So, anyway, they found out that a sizable fraction of commuters were able to find better routes to work and ironically actually produced a net economic benefit. The reason they found this out was that they examined 20 days’ worth of anonymised Oyster Card data that actually contained more than 200 million data points. They were able to see how individual tube journeys changed during the strike and what happened… this particular strike only resulted in a partial closure of the tube network and so not every commuter was affected by the strike. So, you could actually compare directly what was going on. So, the data enabled researchers to see whether people chose to go back to their normal commute once the strike was over, or if they found a more efficient route and decided to switch. So, therefore you can start seeing that actually regular commuters affected by the strike, either because certain stations were closed or because travel times were considerably different, about 1 in 20 decided to stick with their new route once the strike was over. So, I started thinking that they also did… while the proportion of individuals who ended up changing their routes may sound small, researchers found that the strike actually ended up producing a net economic benefit. By performing a cost benefit analysis of the amount of time saved by those who changed their daily commute, the researchers found that the amount of time saved in the longer term actually outweighed the time lost by commuters during the strike. You wouldn’t expect that kind of knowledge to be thrown up really and it’s only through big data, and we are talking massive data, 200 million data points that throw this up. They also found out, which I also found interesting that the London tube map, which is iconic, may have been a reason why many commuters didn’t find their optimal journey before the strike. In many parts of London the actual distances between trains are distorted on the iconic map, by digitising the tube map and comparing it to the actual distances between stations, the researchers found that those commuters living in or travelling to parts of London where distortion is greatest were more likely to have learned from the strike and found a more efficient route. So, what I was thinking is how is big data throwing up very different, surprising insights into health practice in the US?


Vivienne, big data, just as you say today and as we’ve said previously on the podcast, is powerful especially in that it generates these surprising conclusions and the ability to process these large data sets is part of what has empowered us to see these counter intuitive or surprising conclusions. In the United States, we see that in many different places. For example, on a somewhat smaller scale in what is sometimes called “small data” or “little data”, as an allusion to big data, we have the ability to use data sets from the hospital to do predictive modeling for things like when our hospital will be so full we’ll have to divert patients. When I say small data or little data, these are still large data sets. The bottom line is that the computing power we have now coupled with statistical sophistication lets us see things that we’ve never seen before and just as with this news story, the classic is necessity is the mother of invention, we can see how changes in our department of surgery or in our hospital… sometimes small tweaks, like for example, opening up more intensive care unit beds can have large impact and even predictable impact on the rest of our system. That’s one use of fairly large data and larger data sets that I have used before in part of our surgical system. We have made a predictive model for when our hospital would have to divert patients so that we would know ahead of time and be able to offset it. Things like ICU bed capacity made a world of difference for all sorts of end points through our hospital and it would not have been as easy to tease out, or we wouldn’t have been as focused with our interventions without the use of larger data sets and sophisticated statistical modelling techniques. So, I think the lesson learned from analogous situations like the London tube strike, or the most recent London tube strike, is that this power that we have now can bring us to sometimes counterintuitive conclusions, or sometimes just surprising conclusions that we wouldn’t have guessed otherwise. I think it’s a good lesson, and as you said, it’s right there in the news today.


What I found really interesting was the fact that most people hadn’t bothered to find their optimal route until they are forced to experiment. That says something about human nature, but it’s also something we should learn from. Perhaps we shouldn’t be too frustrated that we can’t always get what we want or that others sometimes take decisions for us. That was the conclusions of one of the co-authors, Dr Tim Williams from Oxford University’s Department of Economics. He said, and the final note to the article was, if we behave anything like London commuters and experiment too little, hitting such constraints may very well be to our long term advantage. That turns everything on its head, doesn’t it? That through adversity, another take on your necessity is the mother of invention, but through adversity we do discover other things. I think with the use of big data, those gains could be huge.


Vivienne, I think it’s a very interesting message and in the quality control and improvement world we have two related techniques. One is called the Design of Experiments, where on a small scale we tweak certain portions of a system and through the use of statistics we can tell if the changes in the systems output are related to our initial tweaks. Again, it’s called DOE, or Design of Experiments, and we do it prospectively rather than analysing data retrospectively and using a more big data technique. So, there are lots of ways to get it… just what the professor said. The fact that sometimes tweaking and innovating around that is key. The last technique is more of a theory actually called the theory of constraints. The theory of constraints goes along with this idea of these boundaries. You need to understand what they are, and again, if you just stayed within the current frame you would never find, for example, your faster way to work. So, what an interesting article.


You mentioned retrospective data and I think that whole business of using data retrospectively has lots of problems to it, don’t you think? I think it is something that perhaps we ought to talk about today in a bit more detail than we’ve mentioned in the past.


Well Vivienne, thanks for highlighting that issue and I think today it’s worth spending some time on some of the different types of bad metrics we have seen in healthcare. I think there are four broad categories that we see recurrently in healthcare that really make it challenging to have a data driven department or hospital, when after all its very valuable to have a data driven mentality because it allows us to make all of these advancements, like the ones you talked about. So, I will share with you today some of the different types of difficult metrics I have seen and how they impact us as patients.


Right. So, would one of those be the difficulties when you can’t actually collect accurate or complete data? Does that cause a problem? Well, I suppose it does [laughter].


You have highlighted one right at the onset and that is these metrics for which we cannot collect accurate or complete data. Vivienne, it is surprisingly challenging in hospitals to collect data, accurate complete data. Often, in fact, in some western management systems data collection is frowned upon or it’s an afterthought. People are so busy that it’s an imposition. So, as we launch in here, let me just tell you that often in hospitals we will hear that we can’t collect any data or any more data or data on a particular problem. Of course, if it’s not measured, it’s really challenging to manage. So, if you can’t afford the time to collect good data, well sometimes it’s useful to stop and think, we really can’t afford not to collect good data.


So, in fact, you are talking about some staff. I don’t think it’s specifically in health. It’s in education certainly, I know that, where people are seeing it as an imposition. I suppose before you do anything, a change of perception, perspectives and attitudes is important before you do anything. Like you say, you can’t afford not to collect big data or data of any kind.


You are exactly right. Not only is it changing people’s thoughts about data collection, it is certain techniques that can make data collection much quick and accurate. For Six Sigma in particular, we have certain ways to collect prospective data that really takes a second, two seconds at most as patients come through the system. So, part of it is leading the culture and our colleagues around us to understand the importance of data collection and then there are specific techniques that can be used and specific ways to set up data collection for both discreet and continuous data that get us the data we need a lot faster and I’m happy to share those with the listeners. If they want to email us through our website or get in touch, we can talk about some specifics for how to set up data collection to get at good data for difficult managerial problems.


We are DDD. Data Driven Radio.


Well, I’m going to throw you a curve ball. We have been talking in this country, just today in fact, about robotics and how robots are becoming more useful to us in the workplace, and I assume in healthcare and everything else, where there will be certain manual tasks or simple tasks, or not quite so simple tasks that will be done by robots. Now, that would be quite interesting if those robots automatically collected data as well, wouldn’t it?


It would be and you probably know that robot comes from a Czechoslovakian word for servant. In fact, if I remember right, it was coined by Isaac Asimov, but I may be wrong about that part. [I was wrong here.  It was Josef Capek.] The fact is, yes, it would be very useful, especially with some of these rapid data collection techniques, to have a robot or a similar automaton that does some pre-described straightforward task and also collects us some data. Yeah, it’s an interesting opportunity and it segways nicely into another type of bad data, or bad data metric. Those are metrics that complicate operations and create excessive overhead. Those are some of the worst ones, Vivienne. The ones where you think you have something important to measure and just the doing of it is so arduous and it makes things so difficult and it takes staff time. Those are metrics that should be designed out and probably aren’t as useful.


Have you got any examples of those?


Sure. Often in healthcare, the reason that staff recoil at data collection is when you hear data collection it can mean one page filled with 10-12 checkboxes that float amidst the sea of checkboxes, vital signs, prescribed forms that we have to fill out. Something even as arduous as one more page can make a big difference for our colleagues when we are at the front line taking care of a patient, or our nursing colleagues who are checking a patient into the post anaesthesia recovery unit. It can be challenging if it’s even one more form. So, yes, I can think of probably five or six stories where management wanted some data collected and the doing of it was just really difficult.


So, in fact the development of specialised software which is ongoing will probably take a lot of the pain away, enabling healthcare workers, surgeons, teachers, whoever, to actually get on with the business that they feel that they are trained for. I saw in the news today actually that there is software being pioneered by some law firms which take out the really basic jobs like checking whether a contract is appropriate and legal. Which are really tedious jobs that you would give a junior to do that can done really quickly by software alone.


Well, electronic health records and similar tools hold great potential to allow us to retrospectively pull data. For the Six Sigma and quality projects, we really like to collect prospective data and often the end points that have the most meaning in our systems are not ones that are typically thought of to be including in an EHR. Again, electronic health records can be very useful, but the Six Sigma teaching is to collect data prospectively in statistically valid sample sizes directly from the line or the process. So, I think you are exactly right. Electronic health records hold great potential, and I would add that just as valuable, if not more, is prospective data as we do it.


Have you got another example of bad data for us?


Vivienne, one of the other big problems we see routinely are metrics that are exceedingly complex and ones that don’t tell a story. There is a value to having a metric that feels how things are going. You may have seen ones created, like the happiness index for countries or some similar metric that almost is a statistic with a humanised element built in. Metrics that are exceedingly complex, that are difficult to explain, that don’t have an intuitive feel, those are tough. On the blog, we have an entry where we briefly describe a metric for operating room readiness. Are you ready to hear, Vivienne, what the operating room readiness was for our department of surgery for last month? Do you want to hear what it was?


Yeah, hit me with it.


Okay, it was…pumpkin. Now, if you are recoiling or confused, well “pumpkin” is a confusing answer or metric or statistic for how well the operating room is doing in terms of readiness. Yet, all the time throughout healthcare and surgery, we have metrics like pumpkin for operating room readiness that really don’t tell us much, or that answer a question in an odd way. That is really challenging. So, like using the score pumpkin for operating room readiness, sometimes in healthcare we have metrics that are complex or difficult to explain or lack a feel to them.


I am speechless here [laughter].


It’s a strange example, but what is stranger are some of the unusual metrics we see on quality dashboards all the time. That idea of just putting some metric on a dashboard… a bad metric, Vivienne, can also cause employees to just ‘make their numbers’. When staff can’t feel the metrics or you have something that really doesn’t have a lot of meaning or you can’t see how it affects patient care, it can be very hard to engage yourself in wanting to collect it, wanting to improve it and that type of challenge makes a metric less useful. So, a metric that causes an employee to just make their numbers or focus on the number rather than what it means, that’s a whole other category of bad metric in healthcare, and they are ones that we see all the time.


Right, so would you like to give us a couple of takeaways, bearing in mind that you’ve been researching and looking into these problems with data?


Yes. I would say, Vivienne, wherever possible, especially as we continue in the information age with so much data and big data that we should take the opportunity to make sure the end points we collect are tailored to have meaning for us, ones that we can feel. I think the way to avoid being hit by the train of bad metrics, and we can see the train coming, it’s important to step out of the way. Some of the metrics that would be most useful for healthcare are the ones that can help us tell a story of our patient care, ones that have a meaning to them, rather than just as we often see, dry percentages and different ones like that. So, especially as we collect more and more data in healthcare, I have this feel like [22:51], a visual display of quantitative information. We have a lot of info and it’s important to represent it in a way that holds meaning for us. So, wherever possible its useful to have humanised endpoints that don’t complicate operations or create excessive overhead and that are straightforward to explain to others, and ones that we can collect accurately and completely. I think those are the takeaways.


Well, thank you very much, David, and I’m sure this is something health teams all over the place will consider very carefully. In fact, we are very interested to hear what innovative practices are being undertaken in your health provision. So, if you’d like to appear on the show, contact us through our website. David, would you like to give us the address?


Sure. They can contact us via and there is an address linked to that page, or they can contact us at our info address, which is


So, we look forward to hearing from you. Meanwhile, if you’ve liked the show, do leave us a rating on ITunes, and you can catch us on Soundcloud too. It’s one way we can ensure the word is spread. So, we look forward to being with you next time. So, from David and from me, its bye for now.

We are DDD, Data Driven Radio. Catch us on Soundcloud and iTunes.

The Use Of Social Media In Surgery–Podcast

By:  David M. Kashmer (@DavidKashmer) & Vivienne Neale (@SupposeIAm)


Click here to listen to the podcast.


DDD, bringing you the metrics behind the data. Here is your host, Vivienne Neale.

Hi, and welcome to DDD, which is Data Driven Decision Radio, episode five. My name is Vivienne Neale and I am delighted to be back with you. For those who have asked, my background is in education, training, broadcasting and social media. Of course, I am also a sometime patient curious to know what decisions are being taken in my name that might just affect me and thousands of people just like me. So, once again I am joined by David Kashmer, Chair of Surgery at Signature Healthcare. David is an expert in statistical process control, including Lean and Six Sigma. He has a specialist interest in new tools to improve healthcare, like gamification. David also edits and writes for a blog called So, hi David and welcome back.


Vivienne it’s great to be with you again today and I hope the weather in England is just as nice as it is here on this side of the pond.


Well, I have to say, yes it is. It’s a beautiful sunny afternoon, so we are very grateful for that, but the evenings are now getting just a wee bit cool and certainly we are in autumn, or should I say fall?


Well, it’s great to be back and what do you have to share with us from the news today?


DDD News…

Well, actually just a quick thing. Some of you might be aware of our export, which is Jamie Oliver who is a chef, who a few years ago, probably four or five now, was campaigning for better school meals and he did that in the UK and also in the US. He has just made another documentary where he has been showing the impact of the amount of sugar we are all eating in the UK. I assume that probably in the US its worse than where we are here, but he did a fairly shock jock type of documentary where he said although we are not recommended to have more than six or seven teaspoons full of sugar a day, most of us are eating between 30-50 with all the unknown sugars, or hidden sugars, that we have in various processed foods. So, he came up with a rather startling statistic that around 130 limbs a week are amputated in surgery by people with type 2 diabetes, which was a really, really shocking thing, and he explored what that means for our national health service. So, that’s what we’ve been looking at here. What about you?


Well, Vivienne, having been through the United States school system I can easily support improved meals and better food.




I also have to agree that it is not surprising that we would consume so much sugar beyond what’s recommended in various ways that we can’t even appreciate. So, I have an easy time believing that that’s our current rate of sugar consumption and it’s easy to get behind the idea that nutrition is a key thing and likely needs improvement in our diet as well. So, I appreciate the news, and I have a piece of news from the last week.




This is an update from last week’s show, and a bit of a correction. We were discussing new prosthetic limbs with 3D printing and you brought up an interesting question about a product called Ninja Flex. After reading the article briefly from it looked as if Ninja Flex was the neoprene sleeve used to fit a prosthetic or some variation of a similar sleeve. Well, it turns out that with further research Ninja Flex is one of the new 3D printing filaments on the market. The filaments are the different type of substrate that come to us on a spool that our printers use to build whatever it is we are looking to make. So, to the creators of Ninja Flex and to clarify for our listeners, Ninja Flex is an innovative product that allows for a flexible construction. So, whatever you build has some flex and give to it, unlike many of the other 3D printable filaments, like ABS and PLA that are very firm and rigid by comparison. So, to the creators and users of Ninja Flex, I am glad we have some clarity now and it seems like a great opportunity for prosthetics in the future.


Right. Let’s hope we don’t need too many. Of course, if anyone is interested in Jamie’s new ideas, he’s got a hashtag which is #JamiesSugarRush, and if you check on Twitter you’ll be able to see how much conversation is being generated by his particular documentary. He finally made the decision that like Mexico, who have put a tax on fizzy drinks, he is also going to impose a tax within all of his restaurant empire to start funding research and so on and so forth.


How interesting. Very interesting.




We are DDD. Data Driven Radio.


So, anyway, that was one aspect of the news. I’ve also been thinking about the number of data points being generated every day and I know that you are very much at the forefront of this research. So, I’m just thinking, well if you’ve got billions of data points being generated, surely some healthcare providers may well actually feel overwhelmed by all that’s being generated. I guess you could end up wondering just what to do with big data, and I think that’s quite understandable, don’t you?


Oh, very much so, Vivienne. With the incredible amount of data points that are out there, it is easy to contract what we often term ‘analysis paralysis’. Whether we have the inability to draw meaningful conclusions from the data or really just freeze in the face of it all. That’s a well-known phenomenon, as you described.


Yeah. So, I suppose the whole idea that data can be difficult to manage, but of course we’ve got that dilemma that is absolutely impossible to ignore, not that we should of course. What happens if you do feel your department’s information isn’t being represented or you are wondering where to go next? Do you have any experience of situations like this?


Vivienne, in line with what we spoke about briefly last week, I do. The challenges are many, just as you say, owing to the amount of data that’s out there. What I’ve found useful, along with many others in my field, is to make sure you have a clear vision for strategy first. Where you want things to go, the type of care you want to deliver and where your focus is. Having that clarity allows you to know what type of data you are looking for and the questions that you can pose in the face of these huge data sets. So, I think with more data than ever I have seen data represented poorly, I have seen data that asks questions that really don’t have a lot of meaning, like the classic how many angels can dance on the heads of a pin? Interesting question, but probably not very applicable to what we do every day. So, it really refocuses us on strategy. When we see data that asks questions that aren’t necessary of gives answers that are either Archaean or really just not useful in any way. So, yes, I have seen it, I have lived it and avoiding it starts with having a strategy for your department and team.


I would say that it’s not even a strategy, but not wanting to split hairs here, but like a philosophy. I think that the whole department has to believe in what they are trying to do and that data isn’t literally, and this is not a joke, an add-on, but is actually part and parcel of everything that’s being done and the conversations and being made and the changes that you want to see are being driven by this.


Well, you’ve said it… I think you stated it much more succinctly than I did. You’re right. It’s how to make data an everyday part of improvement for your system and how to have a strategy to deal with it and the questions it can ask. I think you’ve stated it really nicely. The team has to understand that effective use of data is part of what we do on a daily basis. So, really well said.


Well, thank you, but I’m going to throw a curve ball at you now. As I am not a mathematician and I’m not a data analyst or data scientist, I actually find that my background in literature studies has put me in a really good place to start asking the stories beneath the stories generated by data. I find that I am creating a narrative which is really quite different from people who are used to analysing data in that conventional, almost mathematical way. So, I think that actually the quality of the insights that are being delivered, you have to really think about that, and what sort of insights have been delivered by your department when using big data, and do you ever bring in anybody that’s not medical to help you out? Just a question.


We do, especially in those sections that use data according to a standard required by whatever governing body. What I mean by that is, in the United States the American College of Surgeons plays a strong role in trauma care and it mandates that a certain registry be kept with certain types of data. It is pretty rare in healthcare Vivienne that we have people who know how to build models out of data and ask questions in certain ways, and we’ve talked about that in other episodes. The challenges of extracting a meaning from your data. So, yes, I have used other Six Sigma black belts for projects and other people who have deep knowledge of what questions you can ask data and its limitations. I really like what you said about using your literature background to extract meaning from data, it’s a well-known technique to use a story to highlight data to make it more human. So, for example, if we have data that says we are not being as responsive as we are to the emergency department, because we tracked all our response times and we saw the distribution of them, and there was a really large variation and that creates defects where we don’t show up in a timely fashion for our patients, however we may define timely. Then, we tell a story. We say, “Well, this patient coming into the system, Miss Smith, is likely to experience this based on our data”, where if we have a Miss Smith who did experience a certain finding, we use that to highlight our data. So, I really like your take on humanising the data by telling stories that line up with it. I’ll share with you that doing the reverse, using a story and saying, “Well, this is how it goes”, and driving change with one story is actually not always as effective. The reason why is there’s such a distribution of times, if we don’t know that distribution, Vivienne, we don’t know our performance rigorously. We may pick a story where we did unusually well and we may extrapolate from that that, boy, we do a great job, but in fact that one person, that story that we picked does not indicate our true performance. I’ve seen that very frequently with process improvement. So, I really like your idea of looking at the data and humanising it with a story rather than going right for the story not having a sense of the systems performance. That hasn’t been as effective in my experience.


In fact, there are different types of data, aren’t there? Like structured, unstructured, semi-structured, and internal and external data. I quite like… I don’t know how much you make use of, and maybe you don’t, of external data, which is that which is created or generated outside of your department that you don’t own or currently have access to. Some external data is free to access and some is not. Do you ever factor that into any of your analysis or not?


We do. We use multiple data sources that are external to our department, including what is called NSQIP, which is another body of data from the American College of Surgeons, it is pay to play and it benchmarks you versus other centres in some anonymous fashion. The centres tend to be anonymised that you look against, but Vivienne, I will share with you that there are challenges to benchmarking. Meaning the definition of what may be called a wound infection or how other centres come by the ability to tag something as a wound infection. It may be different than what you call a wound infection. I use would infections, but in fact every end point that you’re looking at may have a slightly different operational definition or the data may be cleaned better or worse by one centre than it is at another. So, I’ll share with you that the process improvement take on it, at least the way I was educated is to rigorously improve your system with data and before you benchmark, make sure you have an operational definition of what you’re looking at, which matches up with eventually how you are going to benchmark. The bottom line is, don’t benchmark it first. You probably have a long way to go, and once you’ve gone down that path to improvement, then look to those external benchmarks and external sources of data. It’s a different take on it than some others, but it’s been very valuable.


If you are saying that actually some people are not naming things correctly, should there be… or consistently rather than correctly, do you think there should be some state or national take on this that everybody does things in a similar way, or is that inappropriate?


Well, there often is a national or a CDC guideline or a definition mandated by the College of Surgeons. Let me explain a little further. For example, pretend I work at a centre and this is not the centre I currently work at. I want to be clear that this is a fairy tale, but pretend I work at a centre that seems to have a very increased risk of wound infections among wounds that are clean. It has… boy, it just seems like too many wound infections in clean wounds. Well, believe it or not, in the United States, one of the most common reasons or difficulties we have with labelling things as wound infections in clean wounds is that different raters, different people do not get the same classification of wound when they look at it. In other words, when they use the system of wound classes 1, 2, 3 and 4, the typical way wounds are classified to extrapolate risk of wound infection, they don’t call the same things class 1. One place may call the wound a class 1 and one may say it’s a class 2. That completely changes the expected outcome or the probability of a wound infection. In other words, Vivienne, sometimes certain measurement systems are limited. The wound classification system, 1, 2, 3 and 4, has an interrater reliability which is not perfect, meaning it’s not always reliable that two people using the same tool come to the same conclusion. Now, why is that? it’s a little beyond our scope, but I’ll share that it can be because if you have a surgeon reopen a wound, some centres would say, “Well, that wound is now a class X”, and another centre may say, “Well, that’s a class Y”, and it’s because different descriptions of the wound classification system don’t make it really clear what that would be. Now, my bottom line here Vivienne is, it may not be that another centre is counting things wrong or poorly. It’s not always that clear. It just can be that your centre counts things differently or somehow improperly. That’s why, again, I stress you have to make sure your centre is as improved as it can get by looking internally. Fix what you can fix and then go outside and benchmark against other centres. I’ve found when you do that it really gets you a lot further. You’ve built the infrastructure for how to improve, you have the data collection on what it means and you have a rigorous way to do better and achieve the lowest defect rate you can, and then you benchmark. Usually, when you wind up doing that, you find out you’re doing pretty well.


So, that moves me onto the notion of the social media use in surgery. So, if you were actually tweeting what you were doing and maybe it was something to do with wound infection, for example, does that mean that people will end up starting to ask questions about the nomenclature, to use that word, or the naming of particular… whatever you were talking about, about the type of wound, classification X or what have you. So, for example, if you were tweeting then perhaps that would actually open up a dialogue.


Fascinating question, and I am all for us, as patients, knowing what we’re getting into as much as we can. It’s a tough balance though between friction, blocking physicians from doing what they need to do, and respecting the years of education they have in this. Yet balancing that with the fact that we have a right to know about our bodies and what’s happening to us. So, it’s interesting. When I look at this like other professions, it’s rare that, for example, I would see a lawyer tweet about how they’re looking at ipsa res loquitur or a legal doctrine as they’re considering my case. There is a boundary between where it becomes obstructive versus useful. Now, they have to get done what they get done as lawyers, similarly physicians and surgeons have to get done what they have to get done. Yet I think in very select circumstances tweeting and the use of social media can be very valuable. So, I don’t have an easy answer on that one, but I think in general, conversation about what’s happening to our bodies as patients is a good thing.  One option is to automate tweets ahead of time so that they are put out at regular intervals.


It would be interesting to see what other people would feel about that. I mean, there would be possibly issues of privacy as well, wouldn’t there?


Absolutely. When you do tweet something, like live tweeting from a surgical procedure or something similar, you have to, in the United States, respect the HIPAA laws, or health information portability and accountability act. You need to be very careful with how you do it regarding protected health information.


Yeah, but it’s interesting because I know sometime, about three years ago, you had written about the fact that one in four physicians was using social media daily, and I assume that has gone up considerably since that time. So, these and other data indicate that twitter is one of the least used platforms by physicians, and at that time, three years ago, it was 7% of total use. Do you think that’s gone up much since then?


Experientially it seems to have. I’ve seen more physicians use twitter on a daily basis and even health systems allow tweeting from the OR in select circumstances. So, I do think that it’s becoming more useful and although I don’t have the hard data, I would estimate that that 7% has increased. There is even data regarding now the factors that influence the adoption and meaningful use of social media by physicians. So, people really have looked to tease out what is making us use it more or less and using it with meaning. There are these external factors such as how useful we think it is and how easy it is to use and then other drivers for our behaviour. In fact, there is an entire journal now dedicated to looking at things like this. So, really fascinating.


More specific data on social media use in Surgery here.


Yes. So, thinking about it, given the prevalence of social media usage, where do surgeons fit into the fold, do they?


They do. Despite that 7% usage that we talked about, tweeting from the OR happens every day and during cases, both laparoscopic and open procedures, surgeons do use twitter to highlight, educate and advertise for their programmes and how well they’re doing. Rex Healthcare, for example, in Raleigh, North Carolina, did join the growing roster of hospitals that have experimented with live tweeting. This is also being done in academic medical centres and beyond.


So, what do you personally see as the major benefits of this development?


Well, currently surgeons blog and tweet routinely, as we said, and they do that about the challenges in modern day surgery. They also share useful professional facts and opinions. Some of the business influencers in our field can add a lot to our practice and our approach. They can tweet out different things coming with the International Classification of Diseases version 10, ICD10, and other similar things. So, it’s this constant stream of ideas and an exchange of ideas across the county. So, in any event, tweeting, blogging and use of social media is now common, which represents a huge shift in my understanding of use of social media. Very different than when I trained, at which time we avoided things like that at all cost.


Yeah. I think in many ways it’s quite interesting, even in teaching. Teachers were encouraged not to be part of social media at all, but in fact without being part of this new phenomenon you can’t possibly understand it. I think it’s essential that we do make use of it, but each profession appropriates social media to fit their particular needs.


Well said. As you’ve mentioned before when we’ve talked offline, social media content is not all upside. Things that take just a moment to tweet can last a lifetime and have serious implications for professionals. Also, professional advice given out online can take on a life of its own. So, there are several strategies to mitigate this, including attempting to make it more than one click for professionals who want to ‘tweet angry’ or ‘blog angry’ or say something otherwise inflammatory. There is a real focus on putting useful content out there that staff in similar circumstances around the country can focus on. So, as you say, social media is not all upside, it can be very useful, but if used improperly, highly inflammatory and almost like an online permanent record of what we’ve done and it’s something we really need to be careful with.


So, you are saying that there are still many potholes you need to negotiate.


Well said.


Well, that’s that. We’ve sorted it. It would be really interesting to talk to physicians and surgeons that make innovative use of social media at times of surgery or pre or post-surgical experiences. It would be lovely to add yet another voice to this DDD podcast.


Well, Vivienne, I think in the next coming weeks we’ll have some colleagues on who have tweeted from the OR or have seen social media used to great effect from the hospital or operative side. So, watch for that coming in the future. I think we’ll have several of our colleagues on here soon. I would also add that one of the useful things to avoid those common mistakes with social media are guidelines written both here, by the American Medical Association, and abroad, about responsible use of social media. So, as I think it would be said on your side of the world, do have a look in on those useful guidelines for physicians and other professionals as our listeners begin to use social media more and more.


Well, thank you very much, David. Certainly I think we’ll have a lot to consider over the next seven days.


I think so.


Okay. Thank you. I hope you’ve enjoyed today’s episode and if you want to keep up to date with David Kashmer’s approach to quality and statistical process control, business model innovation and critical practice, do join us for the next programme. In fact, we are very interested to hear what innovative practices are being undertaken in your health provision. If you’d like to appear on the show, contact us through our website. We are looking forward to hearing from you. Meanwhile, if you’ve liked the show, do leave us a rating on ITunes. It’s one way we can ensure the word is spread. We look forward to being with you next time. Bye for now.


We are DDD, Data Driven Radio. Catch us on Soundcloud and iTunes.

The Healthcare Quality Podcast: Big Data in Healthcare

By:  Vivienne Kneale (@supposeIam) & David Kashmer (@DavidKashmer)


Listen to the podcast here.


Hi, and welcome to DDD, which is Data Driven Decision Radio, episode four, if you didn’t know. My name is Vivienne Neale and I’m delighted to be back with you. For those who have asked, my background is in education, training, broadcasting and social media. Of course, I am also a sometime patient, curious to know what decisions are being taken in my name that might just affect me and thousands of people just like me. So, once again I am joined by David Kashmer, Chair of Surgery at Signature Healthcare. David is an expert in statistical process control, including Lean and Six Sigma. He has a special interest in new tools to improve healthcare, like gamification. David also edits and writes for a blog called So, hi David and welcome back.


Vivienne, again it’s great to be here with you and all the listeners today.


Yes. We are all looking forward to seeing what you have for us, but before you start, I have been scrabbling about in the news and I’ve come across something I think you’ll find quite interesting. It was an article about a designer who has managed to create a 3D printed elbow and upper arm prosthetic. This actually has hand actuation and it was made for a boy without an elbow. I know that you have a real interest in 3D printing. So, what do you think about that?


Vivienne, this fascinating phenomenon has been seen throughout social media. I am connected via Twitter to many of the 3D printing hubs and we have seen interesting cases like this over the last 4-5 years along with the rise of 3D printing. The particular one you reference comes to us from and they run through a nice example of manufacture and design of a useful novel prosthetic for a child, just as you mentioned, and one that is able to grip and flex. It was about three to four years ago that I first saw an example of a 3D printed prosthetic for a child and this one really shows the nice advances that have come between now and then, but Vivienne, again what we are seeing is this rise of personalised medicine where a sharable stereolithographic, or STL file, can be posted on the web, transmitted easily across the internet and can be used to size, design, redesign and tweak a prosthetic limb for a child or an adult. It’s really a fascinating time that we’ve come to.


I agree. It’s like science fiction, but its science fact and its science now. Do you know very much about what Ninja Flex is?


Not the Ninja Flex product in particular, but my understanding from the article is that there are several types of implement that will relieve pressure from the plastic prosthetic against the skin. My understanding from a brief review of the article is that one of these products is a Ninja Flex inner liner. It allows a comfortable fit for the prosthetic while relieving pressure. If you’ve ever seen the show Dolphin Tale where there is a prosthetic limb manufactured for a dolphin. That tail requires pressure relief against the skin, and similarly, but distinct, for humans, we require some type of pressure relief to both fit the prosthetic properly and make sure we don’t have pressure sores develop at the sight of the limb interface. In this case, my understanding is that Ninja Flex is one of those products, but pardon me if I misspeak, I am not familiar with that particular one.


Well, it’s certainly exciting and I’m looking forward to a full body transplant. I want to go somewhere and have a look at all the bodies on offer on hangers and say, “yeah, it’s okay. I’ll have that one”.


I think there may have been several science fiction books written in a similar vein.


There has been, and there is a new film out as well. So, yeah, I am all for it. So, anyway, talking about things that are changing the way we might possibly live our lives, I have been exploring a little bit more about data this week. We both know that data points, absolutely billions of them, are being generated every day. I think, and you may be in this position, some healthcare providers may well feel totally overwhelmed by what’s being generated. You can actually end up wondering just what to do with big data. For some people, even knowing what big data is, is a hurdle. So, let’s start right at the very beginning. I mean, big data, as far as I can see, is at the heart of the smart revolution and the basic idea behind it is that everything we do is increasingly leaving a digital trace. I quite like the phrase ‘digital exhaust’ myself and this trace can be used by us or others to analyse what is happening and making decision making much more smart. The driving forces in this brave new world are access to ever increasing volumes of data and also our ever increasing technological capability to actually mine that data for maybe commercial or organisational insights. So, you can actually end up thinking, well, I’ve collected all of this stuff, what am I going to do with it? We actually know that data in itself can be difficult to manage, but that’s not an excuse not to do something about it. It’s impossible to ignore. So, what happens if you feel your department’s information isn’t being represented or you’re wondering where to go next? Do you have any experience of that kind of situation David?


Vivienne, I do. Not just in my professional life, but in every day of our lives. Let’s review. Most of the data that exists has been generated in the last ten or even five years. What I mean by that is, information scientists teach us that the data we’ve generated lately, relatively lately in human history, vastly outstrips all of the information that we had produced here before. What is interesting about that is we live in a very different world, some say that our minds were originally designed for. It was evolved to be hunters and gatherers eventually and now we live in a sea of information, which is very different and much more fast paced, some say, than what we are designed for. Now, to step back from that philosophically into more of, I guess, a concrete example, I’ll tell you that in our department of surgery one of the things we wrestle with is how to extract meaning from the data we have. As you said, big data implies the use of these massive data sets that exist to come to meaningful conclusions with the power to change our mind and make us act differently, even sometimes in counter intuitive ways. Two examples, one from a surgical department and one not. Let’s talk about the one that’s not first. About two or three years ago, a very famous pop news article came out about how Target was able to identify, the store Target, was able to identify women who were pregnant via their web browsing and associated web browser history before the women themselves knew they were pregnant. They would receive targeted advertisements and other data that was related to their pregnancy before they knew they were pregnant. It was just based upon their web search history and the big data that Target had used, extracted, culled and made a predictive model of that could tell us which customers were pregnant. Well, it was so good that again it would identify women before they knew themselves that they were pregnant and such is the value of big data. To come to these seemingly impossible, but very valid conclusions from enormous data sets. Now, let me tell you just in the hospital setting, we are now evolving away from the very consistent historically utilised process of process improvement that we always had. We had a quality improvement process that focused really on specific people. Us as surgeons, individuals, who went wrong, how did they go astray strategy. We are learning now with more complex data, statistically robust tools and the valid use of data this different lesson. The lesson of systems and how to evolve even predictive models for what we are about in our surgical department. For example, we put together a model based on our data over a long stretch of time that would predict when our hospital would have to divert patients, when we would be unable to render the care we really focus on for sick and injured patients and that process is called diversion. We built a model with a large data set that would tell us ahead of time when we would be going on diversion, and then we leveraged that to patch up our system so that we would know how to avoid going on diversion and how to take care of patients without having to turn anyone away. So, those are two example, Vivienne, of how we leverage big data both in healthcare and how I’ve seen it used in industry.


I can see that as being exactly the point in the UK with the National Health Service. We have some extraordinary periods of overload and I think that data will obviously help tremendously in predicting that actually almost down to the wire to make sure that the whole process works smoothly. Certainly, I think the health profession in general has been able to reveal new insights and opportunities and even excavate unknown or recurring problems. I think that will help both in efficiencies and also economic efficiencies as well, which is just as important. When you overspend, you don’t have the money to do what you have to do.


Vivienne, that’s just one of the ways, as you said, one of the important ways that big data can really serve us in healthcare. I think that there are several insights we can get from big data and then there are some insights that we can’t even imagine. what is so fascinating about the big data process is we can consider it more of a way in which we have become sophisticated enough with data to process big data along certain lines, large data sets, to draw from them meaningful conclusions and its more of a way to focus on what insights… rather how to get insights from data instead of the specific nature of what they are going to be, but just as you said, they fall in line with resource allocation, whether its diversion or some other aspect of quality fairly routinely.


Yeah. in fact, Bernard Marr, I don’t know if you are aware of his book, it’s called Big Data, discussed starting with a smart model and what he suggests is you start with strategy, you measure metrics and data, then you apply analytics, report the results and then you transform the business of caring. Is that how it works in your organization or are you doing something different?


No, it’s fascinating that although I have not used the smart model, many models point to the same pattern. For example, the Six Sigma model is called DMAIC, which is where you define what you are looking at, you measure it rigorously according to statistically valid sampling patterns, you analyse that data and then you improve, make changes. Then, an additional aspect that the smart model does not include, you control the project over time, meaning once you’ve made changes you have a plan to look in on it again to see how you’re doing. So, the DMAIC model incorporates some things the smart model does not and yet the smart model starts us off with, I think, a key element, which is strategy. The DMAIC model does not include really whether you should be doing a particular quality project, it just starts you off with definitions, making sure you have the correct or a reliable clear definition that can be used consistently. What I like about the smart model is the focus on strategy upfront.


Yeah. Well, it’s quite interesting. I like the controlling the project afterwards. So, we need a new one that combines both, I think, because it’s so easy, isn’t it, to actually have a huge exploration of something and then brush off the dust and say, “excellent. That’s done”, and it has to be ongoing.


I agree completely and I would just say that what’s done in Six Sigma, as far as the strategy portion goes, is not again the project pattern or the DMAIC pattern that you’ll use to improve a process to find, measure, analyse, improve, control. That’s once you get into a project, but Six Sigma practitioners, especially what are called master black belts, that’s where the strategy aspect comes in. There are other tools that we use outside the DMAIC project to say, “Hey, look. In our set of possible projects, which ones should we be doing? Which ones will have the most bang for the buck?” We use tools such as FMEA or failure mode effects analysis, FMEA, to decide which projects have the most bang for the buck, where we should go first, and that’s the strategy component that compliments the DMAIC pathway. So, it does get there, just in a different way.


I don’t want to put you on the spot, but I will [laughter]. Can you tell me what observations and insights have really given you the most bang for your buck in your experience? Is there something that really stands out for you?


There is. There are several projects, which across centres, give bang for the buck. What is useful about the FMEA tool is it steers you towards projects that maybe you wouldn’t have thought of, for example, it focuses on the ability to detect a problem. Meaning if you have a problem that is very challenging to figure out because you don’t have a good mode of surveillance or you don’t have a good way to catch it, that’s a more important problem. Also, the incidence of the problem, how often it happens, the severity of the problem. When it does happen, how bad is it? These and other factors are used by the FMEA tool to come up with a composite number to prioritise projects. Across centres, you see similar projects come up. Some are disaster planning at certain centres. What would happen if the building collapsed? Is that going to happen often? No, but it’s so severe and it’s so immediate that often a FMEA will find that. So, for surgery departments there are similar projects. That’s how it usually goes.


Well, certainly thinking about the profound explosion that happened in China that would have been an example where a disaster model would have to have been put into practice immediately.


Yeah, absolutely.


So, do you think therefore that what we are saying is data makes healthcare or practice more predictable, if that’s every possible?


Well, I do think it is possible, I would say. I think one of the typical replies to quality interventions is, “well, every patient is so different”. I think that’s true. I do think patient to patient there are opportunities to personalise medicine, and we should, and that’s coming both with the personalised dosages and 3D printed pills that we talked about last week, to genomic medicine. There are really so many ways to personalise healthcare. That said, we can consider each system, and not just individual patients passing through it, but each system of patients, each population of patients who come through the ED, the patients who… we can consider them as a population. There will be a bell curve, not always a normal distribution, but some curve associated with, for example, how long patients stay in the emergency department, and yes, from both personal experience and quality education we can do things to improve that populations, for example, time in the emergency department. We can decrease the variation, the width of the curve with certain interventions to improve our care. I think it can be done, I’ve done it, it’s what I focus on every day. It is very different than how we are typically educated in healthcare and that’s what makes big data valuable, but often more challenging to implement in different systems.


Of course, that does bring up some moral and philosophical questions, in terms of if you keep so much data on your patients or a group of patients, you could possibly ultimately deny treatment or you could say, “well look, look at the stats. Look at how you’ve looked after yourself. Do you really think that I want to use my time, my skill, my money, my department’s money in giving you another stint when you haven’t given up smoking or you’ve eaten too much cholesterol or what have you?” I mean, I know this is trivializing it in one sense, but there has got to be… I’m sure in the wrong hands we might end up giving data freely and it’s then used, not against us, but not to help us.


Vivienne, there are two large ideas that you reference in your comment. I think it’s important to talk about them. One is data security. You indirectly say we give up so much data, and we do. In the United States, we’ve had in the veteran’s association even one laptop being stolen that wasn’t secure, exposed the data of many, many patients. That is a challenge in the age of big data. How do we keep our data secure against cyber-attacks? How does my personal health information stay secure? There are lots of answers to that. I think it’s going to be a little beyond the scope of what we talk about today, but it’s no less important. I think its key. Then the second related thought you had as well, we may have all of these aggregated… all of this data, but the challenge is then what if it is used for a purpose that’s, let’s just say nefarious or maybe not what we would want it to be, or maybe to deny treatment to an individual? I would say, yeah, that is a real possibility. I have seen large data sets and ones that aren’t ‘big data’, just different studies used to say we should or should not treat populations. So, I think that risk exists whether we talk about big data or just staff who are driven to say there are certain things we should not treat. That’s a tough balance. I don’t look to big data to make individual treatment decisions and we teach, at least when I teach data and data use, I focus on the fact that there is a real difficulty in applying population level data, bigger data to an individual patient, to any one individual. It’s very tough to do for a lot of different reasons. You have to watch it. So, where there is a down side, like you said, where maybe somebody will use this not to treat, there are incredible upsides. Maybe we will use this to treat you better, which is my focus. Maybe we will use this to get you through the emergency department faster because we know that actually makes you do better in the hospital stay, for one example. I am not saying that’s true across hospitals. I am just using this as an example. ED time and length of stay at some places may correlate to outcome, maybe at some it doesn’t, but the issue I reference is I would say, and I’ve heard exactly what you bring up before, maybe they will use the data for the wrong thing. Yes, but I would say maybe they’ll use the data to treat me and my family much better than we could otherwise get. That opportunity, that side of things is what I tend to focus on, and I think your concern is well said.


Yeah. Well, I think it’s really interesting. We could take a whole podcast to talk about the pros and cons in a wider field and I’ve got so many ideas buzzing in my head. I will keep you right on the straight and narrow [laughter]. So, data does show important trends and what can improve quality of care, but that actually is often an individual experience and I suppose data becomes more valuable to you when it’s used to further customise and personalise a client’s experience. Certainly in the marketing context, customer experience is all. So, the question for me now, my last question is, how far does personalisation actually go without actually being unattractively intrusive in the medical profession, or should we ask, does that actually matter when more hospital procedures are intrusive anyway? What do you think?


I would say, for me personally, I believe in personalised medicine as a window on better healthcare. I think as we become more advanced there is a real opportunity, not just for satisfaction in my experience of care when I’m a patient, but also to get a better outcome because we are using medications that will work for me, a prosthetic that will fit for me, or whatever my needs are. So, I think it’s very useful, but I would also say, while we manage things down that road it’s important to manage and have the ability to interpret and gather larger data sets that say, “this is our performance for the group of patients in the emergency department as a whole, or referrals to our outpatients surgery practice as a whole”. When we manage things that way also and use our data set and slice it up in different ways to get… those two complimentary paths give a much better experience. We can ask ourselves, okay, how well are we doing when it comes to getting a patient into our practice and using the tools, for example, of personalised medicine for that patient. You can use big data to ask questions like that. So, I really think there are two different ways at getting at excellent care, neither one of which is alone enough. Those are my thoughts on it.


Thank you very much. Well, certainly you’ve given us much to think about this week and thank you very much David. So, I hope you’ve enjoyed todays episode, and if you want to keep up to date with David Kashmer’s approach to quality and statistical process control, business model innovation and critical practice, do join us for the next programme. In fact, we are very interested to hear what innovative practices are being undertaken in your health provision. If you would like to appear on the show, contact us through our website. We are looking forward to hearing from you. Meanwhile, if you have enjoyed the show, do leave us a rating on iTunes. It is one way we can ensure the word is spread. We look forward to being with you next time. So, bye for now.