How To Measure The Success Of Your Gamification Project

By:  David Kashmer, MD MBA FACS (@DavidKashmer)


A colleague asked:  “How will you know if this gamification project is successful?” and it got me thinking…how do you track success in a re-designed environment that uses some techniques typically seen in gaming?  How do you track, measure, and improve employee engagement or, better yet, that elusive endpoint called culture?


Asking For Endpoints


In case you haven’t seen case made regarding the “why bother with gamification”, look here.  If you believe the sobering statistic that 70% of US workers are actively trying to hurt their company, then you know that something must be done.  Even if the issue is simply emotional detachment from work (rather than actively attempting to hurt the company), and missed opportunities for your company, then it truly is worth trying some tools to help improve the situation.  A recent Deloitte report (2015) highlights that issues of employee engagement and culture have become “the number one challenge in the world [of business]”.


In the quest to do something about it on the healthcare side I’ve helped deploy gamified systems before.  (More on the experience here.) Recently, while setting off on a deployment, a colleague who was in favor of the new system asked:  “But how will we know if it works?”


What’s Been Done Before


The game endpoints, to tell whether the system has accomplished your organization’s goals, should be specific to your company’s needs.  Do you need staff to complete the yearly compliance work?  Then there’s an endpoint.  Do staff need to treat each other differently?  That may be tough to measure…


After all, these things center around that notorious intangible called “culture”.  And how, in short, do you measure that?


In a previous post, I discussed attempts to measure job satisfaction using a standardized questionnaire that has previously been used & validated in healthcare called (simply) the Job Satisfaction Survey (JSS) pre and post game deployment.  (For more on that, look here or here.)  Lots of famous business journals and authors write about how to measure culture.  What my colleague had asked seemed to have no simple solution:  do we measure something as simple as percent compliance with yearly training, as complex as some global measure of culture, or both?


Nuts & Bolts Of Endpoints


Clearly we are out of the realm of easy or straightforward science & testing here!  Consider, for example, a seemingly straightforward move like administering the JSS before gamification and then after the system winds up.  Let’s pretend, at the end of the deployment, there has been statistically significant improvement in many of the scores on each question from the JSS.  Great…except, well, a lot of things in your organization likely changed over the time the game system was deployed.  How do you know whether the system was really the driver of the improvement?


Now let’s take the more simple endpoint:  compliance with yearly corporate training such as fire safety training.  How do we know that, after the deployment, that any increase in fire safety training compliance was due to the system and not just passage of time with more participants completing the training?


The bottom line, here, is that there may be no perfect endpoint for the game system.  Even endpoints that seem straightforward, such as participants returning to a certain place at a certain time, or reviewing certain materials, is just as prone to criticism as endpoints of typical work across the sciences.


Another important consideration is timing.  Consider, for example, an endpoint that your organization truly values such as employee churn.  Perhaps churn has reached a steady state over the last year prior to the new system you’re deploying.  After the system (which directly impacts participants considered in the churn metric) was deployed, the rate of employees leaving the organization dropped sharply.  This may have meaning in your individual system.  So, another important consideration in these metrics is timing:  choose something on which you can reasonably expect an impact from the new game system, which is already measured / important to the organization, and which has achieved a steady state.




Just as challenging as measuring culture in your organization is the measurement of endpoints to determine how successful your gamified system is.  I recommend a combination of endpoints that compare post-game performance to important measures that have achieved stability over time prior to game system deployment and which you can reasonably expect a change related to the new system.


This is no easy task!  Gamified systems, often designed to impact culture and organizational behavior, can be challenging to quantify owing to all the vaugeries of measuring culture in general.  Consider the Job Satisfaction Survey, in addition to more specific endpoints rooted in quantifiable behaviors, to get a sense of the performance of your gamified system prior to update, revision, improvement, and release of any version 2.0 you have planned.


Questions, comments, or thoughts of endpoints for gamified systems in healthcare or in general?  Wonder how to deploy a gamified system to promote engagement and certain actions in your organization?  Email me or comment beneath.


How To Know When You’re Tampering With A Healthcare System

By:  David Kashmer MD MBA MBB (@DavidKashmer)

Lean Six Sigma Master Black Belt


Quick, what’s worse:  to work in a healthcare system that pretends everything is ok when things obviously are not ok, or to work in a healthcare system that looks to correct problems that may not even exist?  Both are difficult scenarios, and each makes meaningful quality improvement difficult for different reasons.  In this entry, let’s explore some tools to help recognize when a problem actually exists and how to guard against each of the extremes mentioned above.  Where is the balance between tampering with a system and under-controlling for problems?  Would you know whether you were in either situation?


If you’re seen a process improvement system that seems to be based on someone’s gut feeling, one that is based on who complains the loudest, or one that is based more in politics than in actually improving things, read on…


It Starts With Data


One of the toughest elements of deciding whether an issue is real and needs correction is good data.  Unfortunately, many times we stumble at this initial step in guarding against tampering with good systems or under-recognizing bad ones.  We’ll talk more about this in the last section (A Tangible Example) but, for know, realize that there are barriers to being able to make an intelligent decision about whether the issue you’re looking to improve is real or imagined.


Using data from your system, as we’ll describe later, is much better than using just your gut feeling about whether you’re tampering with a system.



You Need Some Knowledge


It takes some extra training and knowledge, often, to understand the issue of tampering versus under-controlling.  You may get the training you in a statistics course or lean six sigma coursework–that’s where I got it.  Wherever you get the training, it may go something like this:


Statistical testing helps us guard against important errors.  Like one where we think there is a difference in system performance after an intervention when there is no difference.  It also helps us when we think there is NO difference in system performance after we’ve made changes yet something has changed.  These errors have names.


A type 1 error is the probability of thinking a difference exists when there isn’t one (tampering) and a type 2 error is the probability of thinking there is no difference when in face on exists.  Type 1 errors, also known as alpha errors, are prevented by using statistical testing and YOU making an important choice.  More on that beneath.


Use Some Tools


The tools to help you avoid tampering with a system that is ok include statistical testing…but which test?  Previously on the blog, I shared the most useful file that I use routinely to decide which statistical test to use for which data.  You can see it here.


great tool for understanding the problem with tampering with systems is here.  This video (thanks Villanova) drove home for me the issues that occur (and how we make things much worse) when we tamper.  If you’ve never heard of the quincunx (cool word) check out the video!


Tampering often happens when a healthcare system that is well-meaning and wants to do better, to grow or achieve greatness, embarks on making changes ad infinitum without meaningful improvement.  Heart in right place for sure and a very difficult problem to avoid in healthcare systems that really care.  It’s a much better problem, I’d say, to have than under-controlling (head-in-sand ostrich syndrome) that is sometimes seen in systems.


A Tangible Example


Pretend your process improvement system is very focused on an issue with time.  Time for what?  Let’s say it’s the time patients stay in the emergency department.  You and the team decided to get some real data, and it wasn’t easy.  There were all sorts of problems with deciding what to measure, and how, but (finally) you and the team decided on a clear operational definition:  time in the ED would be measured from the time the patient is first logged in triage until the time the patient physically arrived at their disposition whether that was the OR, floor bed, ICU, or something similar.  No using the time the orders to transfer went in for you!  You go by actual arrival time.  You & the group decided the time in ED would be measured based on patient’s triage until they were wherever they are going.  The team likes the definition and you proceed.


You measured the data for patients for a month because, after all, the meetings with the higher-ups were once a month or so and they wanted data and the monthly meetings were the routine.  So, ok, sure…you measured the data for a month and had some cases where patients were in the ED for more than six hours and those felt pretty bad.  You discussed the cases at the meeting-of-the-month and some eminent physicians and well-regarded staff weighed in.  You made changes based on what people thought and felt about those bad cases.


A few more months went by and finally the changes were achieved.  You still collected the data and reported it.  There were, you think, fewer patients in the ED over six hours but you aren’t too sure.  I mean, after all, some patients were still in the ED too long.  Eventually, other more important issues seemed to come up, and the monthly meetings didn’t focus too much on the patient time in ED.  Did the changes work?  Should you make more changes because, after all, you think things may be a little better?  Hmmm…


Welcome, my friend, to the problem:  how do you know whether things are better or worse?  Should you do more?  Do less?  Wait and see?  Is the time patients spend in the ED really better?  Yeeesh, it’s a lot of questions.  Let’s go back to the beginning and see how to tell…


Back In Time…


You and a group are looking at how long patients are in the ED.  After getting together, you decide to measure time in ED in a certain way:  the clock starts when the patient arrives at triage and ends when the patient arrives at his/her destination.


Because you’ve learned about DMAIC and data collection plans, you decide to calculate how big a sample you will eventually need to detect whether you have improved time in the ED for patients by at least 10 minutes overall.  This sample size helps guide you that if you see a 5 minute decrease in your time in ED after you make changes to the system, well, it may just be fantasy.  The sample size math you do calculates the size of sample that will allow you to detect ten minutes as the smallest meaningful change and no smaller.  How do you do that calculation?  Look here.


Ok so now you know how many patients you will need to look at after you make changes.  By the way, how long will that take?  It may not coincide with your monthly meetings.  It may take longer than a month to get enough data.  That would tell you not to make any more changes to the system until you’ve given any changes you already made enough time to show how they work.  This is part of how sample size protects use from too many changes too fast.  We need to collect enough data to tell whether any changes we’ve made already work!


So let’s say you and the project team make some changes and now know the sample size you need to collect to see how you’re doing.  Well, after those data are collected, now what?  Now it’s time to pick a test!


You can use the tools listed here, but be careful!  The type of tool you pick (and what it means) may take some extra training.  Not sure which tool to use?  Email me or use the comment section below and I’ll help.


So you pick the proper tool and compare your data pre and post changes, and you notice that your new median patient time in ED shows a decrease by 25 minutes!  The test statistic you receive, when you run a statistical test regarding the central tendency of your data (maybe a nice Mood’s median test) shows a p value < 0.05.  Does that mean you’re doing better regarding the median time in the ED?  Yup…especially because you chose ahead of time that you would accept (at most) a 10% chance of a type 1 error (tampering).


That’s what the p value and alpha risk (tampering) is all about:  it’s about you saying ahead of checking the data that there is some risk of thinking there’s a difference when there is no difference.  You further say that you will accept, let’s say, a 10% risk of making that wrong conclusion.  You could be wrong in one of two ways:  thinking the new median is significantly higher than the old one or significantly lower and so to be fair you split the 10% risk between two tails (too high and too low) of that distribution.  And viola!  A two tailed p value is created of 5% probability:  if the p from your test is < 0.05 (5%) then you say well the difference I see is likely to be legitimate, after all there’s a less than 10% probability that it is due to chance alone because it’s less than 5% in this tail and so even if I accept the whole other tail (5%) then I’m still less than 10% alpha risk I set at the beginning…


So the bottom line is you set the alpha risk (tampering risk / thinking there’s a difference when there isn’t one) waaayyyy back when as you set up what you’re doing with patient time in the ED.


Take Home


If you slogged through that, you may start to see the headlines on using statistics from the Six Sigma process to prevent tampering:


(1) calculate a sample size required to sort out the minimum amount data you need to be able to detect a certain size change in the system.  The calculation will tell you, given your choice about the smallest change you want to be able to detect, how many data points you need.  It will make you wait and get that data after you make changes to a system rather than constantly adjusting it or tampering with the system owing to pressure of having the latest meeting come up, etc. etc.


(2) perform statistical testing to determine whether you’re really seeing something different or whether it’s just a fantasy.  This will help disabuse you of your gut telling you you’re doing better when you aren’t (and thus missing opportunities) or charging at windmills with lance drawn as you misidentify when things have actually improved and continue to charge on.


Call To Action


Now that you’ve heard about some strategies to guard against layering change after change on a poor, unsuspecting process, imagine how you can use tools to avoid making a lot of extra work that (as the quincrux teaches) can actually hurt your system performance.  Want more info on how to get it done?  Get in touch!


Have you seen any examples of tampering with a system or failing to appreciate issues?  Let me know about your experiences with or thoughts on tampering and under-controlling beneath!






How To Avoid Mistakes With Your First Job Hunt As A Surgeon

By:  David Kashmer MD MBA (@DavidKashmer)


Finding your first job as a doctor is a major life decision. The choices you make now are going to have impacts, both in the short and long run, on the rest of your life. You’re at a very busy time in your life—you’re wrapping up a residency or fellowship and you’re getting ready for your boards, and on top of that, you’re about to start job hunting. Busy as you are, put some serious thought into what you want your first job to be. The more you know what you want from that job, the more efficient your hunt will be and the more likely you are to end up with the job that’s right for you.


Timing Your Job Hunt


Chances are that as you approach the end of your training, your inbox is starting to fill up with mail from recruiters and matching agencies. Some recruiters work for employment companies—they’re medical headhunters. Others work in-house for a specific hospital or group of hospitals.


It’s fun and even flattering to suddenly be getting email from people who say they want you, but don’t jump on an opportunity just because it has arrived in front of you. Save the offers that seem interesting but you don’t need to rush into interviews. Most jobs stay open for a fairly long time while the search committee looks around. Unless a job seems so perfect that you want to try to grab it right away, take your time. Don’t dawdle, however. You’ll be done in July and you’ll want to have your job lined up well before then.


If you’re finishing your residency or fellowship, you’ll start job hunting in the last three months to four months. Sometimes, if you’re a fellow in an academic Mecca, the hospital will try to retain you. You’re already there, you’re credentialed, you know the system, you’re ready to go on day one. When a physician goes to a new hospital and has to learn their coding and billing system, there’s a cost associated with that. The hospital won’t be reimbursed all it could be as that physician learns the system—it could be six months before you start bringing in significant income. That delay usually translates into several hundred thousand dollars.


Academic Centers & Salaries


However, benchmark salaries for fellows who stay on are usually much lower than what they could get in a different practice venue. At academic centers, people work very hard and their take-home revenue isn’t as high as what they’d make in another system. But because academic centers also have a lot of positives, many fellows do stay on. They’re familiar with the environment, it’s comfortable, they’re already living there, the kids are in school, their partner has a good job in the area, and so on. If you have a particular research interest that the center is supporting, that’s a good reason to stay.


If you decide to look beyond your current hospital, give yourself plenty of time to develop meaningful options before July comes around. I recommend getting started (seriously) by April or May at the latest. After all, remember that you may need four to six months of lead-time in order to get a license (and be able to work) at this new job. Do a little math: if it takes six months…


Interested in more advice about your first job hunt?  Look here for more from The Hidden Curriculum:  What They Don’t Teach You At Medical School.

Here’s How To Avoid Disaster With Your Contract

By:  David Kashmer, MD MBA (@DavidKashmer, LinkedIn profile here.)


You’ve narrowed down the choices, you’ve gone on the interviews, and now you’re looking very seriously at offers from your top choices. It’s time to do something for which you have absolutely no training: negotiate your employment contract. If you’re like a lot of young doctors, you’ll just sign whatever contract the hospital administration puts in front of you. Maybe that will work out okay—and maybe it will turn into a disaster. With the information in this section, you can tilt the odds strongly against that horrible disaster.



When you see the salary on your employment contract, you might be tempted to just say yes and take the money. That’s the approach a lot of young doctors take, to their cost. That salary number should be just one part of the starting point for your negotiation.

Before you meet with the hospital administration to discuss your contract, think about your negotiating position. A good starting point is the BATNA. This is an acronym for Best Alternative to a Negotiated Agreement. The principles were developed by the Harvard Negotiation Project back in the 1970s. In 1981, they became the basis for a wildly popular book by Roger Fisher, William Ury, and Bruce Patton called Getting to Yes: Negotiating Agreement Without Giving In. I recommend reading it when you’re done with the content here.

BATNA nicely summarizes your ability to influence the outcome of a negotiation. You develop the alternatives to the deal in front of you based on what’s most important to you. Your strength in negotiation is directly related to your BATNA. The better the quality of your executable options, and the more you have, the better you can influence the negotiation.

Having a good BATNA makes you more apt to talk about alternatives with the other party to the negotiation.  And if you’re more willing talk about alternatives with the people in front of you, you’re more willing to push the structure of the deal and how it needs to look. As physicians, because we don’t know about business stuff, we tend to see negotiations with the hospital as an adversarial “us versus them” situation. I take, they give—that’s called positional negotiating.

You can dig your heels in and say “I need this,” but in reality, a better negotiating path is to understand what the interests are of the other side. That’s very different than the positional negotiating described above.

Their real interests may be different from what they’re articulating in the first contract they park in front of you. If what they really want is someone to come in and take on a large administrative component, yet they’re reimbursing based on clinical work and straight RVUs, you probably want to influence that so that they get what they really want rather than their standard contract. That can be challenging, but can be more worthwhile in that both you and the other side may be much more poised for success with an agreement that represents what you each actually want.

You need to educate them a bit even as you’re learning from them what their interests are. You’re trying to satisfy the interests behind what they initially ask for. It’s a different way of looking at it than “I win, you lose,” which I’ve found is more typical doctor’s way of looking at it.

Remember, on the other side of the negotiation, when it’s all done, you’re going to be working there. You need to make sure the relationship is reasonable and that relationship starts as you negotiate with the hospital or whatever team you’re joining. This is one good reason for going on a lot of interviews. The more alternatives you have, the better your position to compare the current offer. Too many interviews give you diminishing returns, but you want at least three or four high-quality alternatives to get a sense of where you want to be. It’s not always “more is better.” It’s important to develop meaningful alternatives—ones that you can actually execute if you need to.

The interview process itself is time-consuming because you’ve got to prepare for it and then go do it. The hospital pays for your transportation and hotel and sets it all up for you. You don’t usually have any significant out-of-pocket expenses, but your time has value. And sometimes you’re up against a deadline—you need to get some cash flow going. Sometimes the closer deadline is on the other end. The position needs to be filled before their current surgeon goes on maternity leave, for instance, or before the end of the budget year.

In negotiating, you need to have as much information as you can. You want to know more about them than they know about you. At the interview, you’ll probably be asked about your timeline. It seems like an innocuous question and usually comes with “When are you looking to make a move?” But in reality, giving up your timeline allows the other team in the negotiation to have a little more control. If you know the hospital wants to fill the job in two weeks, you have information about their timeline. You can leverage that because they have a deadline. They may be willing to come around a little bit faster than they otherwise would, so they may be willing to negotiate some other points to get you to sign on the dotted line and get the job filled. In general, my advice is to make it seem like you have all the time in the world and to use that to get a sense of what their timeline is.

In reality, young physicians are usually finishing their residencies or fellowships in July. Everybody knows that, so every interviewer knows your timeline. You could say to them, “Well, I have plenty of time. If I don’t find something, I’m going to take some time or work across the country as a locum surgeon, so I don’t need a position until August or September,” but in reality, most young people need a job by July and the hospitals know it. That’s why we have this cynical saying which I mentioned earlier: “In your first job, you’ll probably get your brain stolen.” You’ll be under-reimbursed because you need a job now. You’re more likely to take any serious offer without really negotiating.



As part of preparing for the contract negotiation, it’s helpful to prepare a list of five or six points that are really important to you. One of those points, however, should probably be a pawn—something you’re willing to sacrifice as part of making the deal. You’d like to have it, but you’d be willing to give it up.

As you give it up, you can use it to negotiate the points that are more important to you. For example, you might say, “Well, if I can’t have 20 weeks of vacation, then I need to have a different call schedule.” Twenty weeks of vacation is obviously a lot and you didn’t really expect them to agree to it, anyway, so it’s an easy sacrifice. This technique is called “log rolling” because you take one point and roll it into the other.  It also takes advantage of the reciprocity effect…


The above excerpt is from The Hidden Curriculum:  What They Don’t Teach You In Medical School.  For more information about contract negotiating techniques for physicians (page 53) look here.