Gamification Applied To A Surgical Residency: Caught In The Act Of Doing Something Right

 

gamesjpg

 

You may have heard the term “gamification” (pronounced game-ification) previously.  Gamification is the process of taking certain elements from the world of computer and board games and applying these toward motivational and customer retention strategies for different groups.  Game dynamics may also be applied to other important functions for different companies. Importantly, gamification is a hot topic and is even being taught in some business schools.  It is currently thought that gamification will resonate with the millennial generation (“millennials”) and subsequent generations to a greater degree than, for example, Generation X and the Baby Boom generation.

 

There are multiple important strategies in gamification that we could discuss in this blog post.  Here, we focus on several important game dynamics as they were applied to a general surgery residency in 2012.  Our group used game dynamics for our section of trauma, emergency surgery, and surgical critical care to assess their impact on resident motivation and perception of quality of learning.  Here, we will discuss the dynamics we used and different outcomes.  Interestingly, we also utilized game dynamics for our team of surgical attendings.  We agreed to participate in a similar strategy so as to demonstrate our support for this new approach.

 

The set up included the creation of a consensus group of behaviors by the trauma and emergency surgeons that the team wished to reinforce in residents.  Similarly, the resident staff created consensus behaviors they wished to see demonstrated by surgical attendings.  Many behaviors were already present to varying degrees, and the consensus behaviors were not a list of all new behaviors–rather, they were ones that each group wished to reinforce or make more common.  Each group assigned certain point values to the behaviors. The point assignment was arbitrary and was contingent on several factors including the relative scarcity of the event as well as the importance of the event to our trauma and emergency surgery section as a whole.

 

We then set up an email address and surgeons were able to email from their smartphone each time they caught the resident surgeon doing something correct.  This is a very new concept in residency education:  “catching someone in the act” of doing something correctly.  Interestingly we really focused on catching the resident in the act of doing something good.

 

Next, residents were given a letter so as anonymize them. Each resident knew his or her letter only. The letters were drawn as part of a leaderboard which was displayed in the trauma and emergency surgery conference room.  Therefore, at each morning report, residents could see their progress and point accumulation relative to the point accumulation of their anonymized colleagues.  Certain threshold levels of points were set and were displayed on the leaderboard.  That is, there were certain thresholds of points at which events took place. Some of these events include obtaining a new skill, such as the ability to clear a cervical spine.  Residents would be educated in cervical spine clearance and the appropriate template cervical spine clearance note.  They were then empowered to clear a cervical spine with the supervision of the trauma surgical attending.  Other events would occur at different levels of achievement including a letter of support to the residency program director for the resident’s file and other important elements.  Unbeknownst to the residents, the overall points leader at year’s end was given a special congratulations and a year end gift at the resident’s award dinner.  This was the only time at which the resident point total was revealed, and each resident except the overall points leader was kept anonymous.

 

A survey was given to the residents prior to the institution of this motivational pathway.  This was a validated survey which used a visual analogue scale so as to determine job satisfaction.  This is called the Job Satisfaction Survey (JSS) and has been validated among emergency department physicians.  All resident years participated in the system.  All residents received the job satisfaction survey prior to and then after the year long process.

 

A statistically significant improvement was noted in a proportion of residents (two tailed p < 0.01 by Chi-squared test) who perceived the quality of their education to be excellent.  These data were reported at a trauma and emergency surgery conference at Atlantic City, NJ.

 

This is one nice case study for how gamification is possible in surgical residency.  This program leverages multiple dynamics including comparison for each individual to a peer group and a focus on positive reinforcements for appropriate behavior.  Experientially, the trauma surgeons involved found this to be highly effective in improving resident behavior and reinforcing positive behavior on the service.  Interestingly, some of the residents originally felt that the gamification may belittle resident eduction or somehow cheapen it.  Instead, by the end of the process, these residents were reporting positive results and were receptive to the process. Again, this type of education is a far cry from typical resident education. The terminology of ‘gamification’ was felt to decrease buy in as some residents felt, as mentioned, initial feedback from a minority of residents included an idea that turning their residency into a game was not appropriate.  However, once residents saw that this was merely an assessment tool that focused on positive behavior and reinforced it while maintaining anonymity, they became much more receptive etc.  There was no ability to remove points from any participant in the system at any time, and, again, focus was placed on what the residents did appropriately according to the defined behaviors.

 

Experientially, from this, our team learned that gamification is achievable in the inpatient medical education world.  We also learned that this innovative process was a true boon for the human resources portion of our section.  It changed the dynamic and interaction between the surgeons and the residents for the better.  Also, experientially, we learned that  performance seemed to greatly improve.  There was a constancy of expectation of the residents by the trauma and emergency surgeon and vice versa.

 

As mentioned, the trauma surgeons also participated in the system.  Trauma surgeons were also anonymized and had a leaderboard. Each trauma surgeon was given a letter and each trauma surgeon only knew his or her letter.  Point totals were accumulated as emails were sent from the residents to a third party who was not one of the surgeons who attended.  These were then reflected on the points on the leaderboard.  Many of the dynamics have names in the gamification world.  For example, the system described above incorporates some of the most basic game dynamics including what are called PBL’s.  PBL stands for points, badges and leaderboard.  Points are self explanatory as is the leaderboard.  These leverage positive peer pressure and reinforcement so as to increase performance.  The badges are those things that are achieved to signify an improvement in level. Although we did not give true badges to be put on the physicians coat etc, stickers and other cues maybe options for other programs.

 

We did use another dynamic, that of “leveling up”, to allow this gamification process to recognize good performance and to increase point accumulation by allowing residents to obtain new skills independent of their year level. Although senior residents had certain skills grandfathered in based on year level, such as PGY-5’s ability to clear c-spines with supervision of an attending surgeon, younger residents were able to attain these and other skills via achieving a certain point total that demonstrated competency in the performance of similar tasks.  This process makes for a competency-based method of advancement and attainment of new skills.

 

Surgeons and residents were greatly satisfied with this innovative system and it is one you may apply in your educational or motivational process.  Consider applying the process to your team of surgeons, residents, or Advanced Practitioners.

 

Questions or comments?  As always we invite your thoughts.

Handy Tools For Startups & Investors

 

In this entry we focus on some of our favorite resources for potential startup founders and investors.  These are resources focused on better understanding the nature of startups in terms of mechanics, fundraising, valuation and other important keys. We will highlight several of our favorite resources on these important topics.

 

 

1 – Kauffman.org

 

Some of the most useful resources for startups and potential founders is the Kauffman Foundation.  Visit kauffman.org.  Most education from the Kaffuman Foundation is free and readily available online for download.  The Kauffman Foundation is an organization focused on those key facts and further resources for startup entrepreneurs.  We found it particularly useful for a broad range of topics. Interestingly, Kauffman Labs focuses on hands-on, laboratory type interactions for potential startup founders and investors to better understand the mechanics of starting a new venture.

 

2 – Social Media

 

Social media is one of the more interesting resources for potential angel investors and potential startup founders.  In particular, Twitter has multiple users who tweet daily on different important topics in investing, startups, and other useful information.  There are also multiple CEO’s who maintain Twitter accounts.  On twitter, for example. you can find Y-combinator, Kickstarter, 500 startups, Start-ups.co, and multiple other interesting resources such as entire VC firms like Sequoia and others.  Social media like Twitter has a plethora of daily updates that give free information, often with links to evidence, or blog entries that discuss certain useful topics for startups.  Angel investment teams like Grizzly and Gibbon LLC can also be found on Twitter and, although unlikely to result in a deal, you can follow these different investment teams on that platform.

 

3 – The Founder’s Dilemmas

 

Another useful resource for startups is The Founder’s Dilemmas by Noam Wasserman.  This text highlights many of the issues involved with startups.  Old favorite subjects that we also cover on this blog include dynamic ownership equity, alignment, scaling, team composition, and whether to startup with family.  We can’t stress highly enough just how useful this text is for potential startups.  It really functions as a roadmap and can prevent you from having to learn lessons by brute force methodology or by going through them yourself.  Take a look at The Founder’s Dilemmas if you have a moment.

 

4 – The Lean Startup

 

Another useful text for startups is The Lean Startup by Eric Ries.  This is one of the fundamental texts that develops and introduces the concepts of applying lean methodology to business startups.  Other old favorites on this blog, such as the minimum viable product, the business model canvas and host of others are introduced by The Lean Startup.

 

5 – Novoed.com

 

One of the other useful sources of knowledge for investors and startups is Novoed.com.  novoed.com runs classes such as Clint Korver’s Venture Capital 101.  VC101 which recently completed an online class was conducted with a team from the Kaffuman Foundation and Clint’s useful course gives insight into the multiple functions of VC, the mechanics, and the investment decisions involved with venture capital. Clint gave excellent case studies from Ulu Ventures, which is the VC fund he co-manages.  Novoed.com has an excellent team-based approach to learning the intricacies of VC etc.

 

6 – Coursera.org

 

Coursera.org is another online platform for acquiring the tools necessary to startup effectively.  Entrepreneurship 101, and other similar courses have excellent opportunities to join a team that runs through the startup mechanics online.  These course, often put on by Stanford University Professors, are free to join and make you part of a team that is often located throughout the country or even the world.  The creation of a business model canvas is a focus as are other topics such as optimal team size, creation of a low fidelity prototype and other useful decision making strategies.

 

We can’t speak highly enough of this selection of tools for making your next startup or investment decision of a higher quality.  So, today we have reviewed several useful tools for better understanding startups, investment opportunities and the nature of entrepreneurship.  We highly recommend the online learning platforms of Novoed and Coursera.  Further, we recommend those selected texts including The Founder’s Dilemmas and The Lean Startup.  We think these will equip you well so you can avoid having to relearn some of the same lessons we have experienced in our time.  Here is wishing you higher quality investment deals and smoother, more effective startups.  We feel that using the tools above will make it easier for you to achieve your goals in entrepreneurship.

Shark Tank: The Tank Can Only Do So Much

 sharkjpg

 

Shark Tank is a very entertaining and popular show.  Each week, more than four million viewers tune in to see great ideas, strong pitches and newly minted potential millionaires.  Just as entertaining as the deals that go well are the ones that go poorly.  Here we explore the nature of deals on Shark Tank, the ups and downs of valuation, and the challenges in exposing venture capital type investment decisions to a broad audience.  After all, in the show’s need to reach a popular audience, it can’t show the more technical side of venture capital (VC) decision making.

 

First, let me share that I really enjoy Shark Tank.  I enjoy each of the venture capitalists on it.  Exposing a broad audience to the thought processes involved via a show about entrepreneurs, startups, and venture capitalists / investors is wonderful.  Yet, there are some challenges with exposing the process to a broad audience and that’s our focus here.

 

One of the givens, here, is that drama of the tank needed to be amped up to make it exciting for America.  Thus the dramatic music and monikers the VC team attach themselves including, for example, “Mr Wonderful”. I do not lament this at all.  It makes for excellent TV, drama, and adds to the appeal of the show.  Kevin O’Leary’s “Mr. Wonderful” handle is particularly funny and shows clear wit:  my bet is that, if Kevin is interviewed years from now about the show, he would be the first to explain this was sort of a tongue-and-cheek name with which he dubbed himself to add a little spice to the show.  It makes things a little over the top in a positive way.  Indeed, the juxtaposition of the title “Mr. Wonderful” against some deals which are really not so wonderful is always entertaining.  Of course, there are those moments where the deals Mr. O’Leary makes with business owners truly do make him their Mr. Wonderful–and, after all, the business owners wouldn’t be there if they didn’t need a Mr. Wonderful for something.

 

Besides the entertaining names and amp-ed up drama, there are some challenges in the show that we should explore as they relate to business model innovation, valuation and liquidation preferences.  Liquidation preferences you say? These are never even mentioned on the show.  This is one of the challenges in bringing Shark Tank to a broader audience. Each time Mr. O’Leary (ok, yes, I enjoy his disciplined investor role) asks:  “But how will I get my money back?”, he is boiling down a key part of portion of VC deals to a focus on how money is returned to investors.  In VC deals, this issue is much more complex than what can be covered in the brief, digestible portions required by the Shark Tank format.

 

Real-life VC deals typically include a term sheet, where the entrepreneur team is presented with the terms of the deal with which the venture capitalists will invest.  (Of course, many real life VC deals involve much more, such as a series of meetings rather than a less than five minute interaction on which the sharks base their investment decisions.) One of the most poorly understood elements of the term sheet is liquidation preferences.  It would be very difficult to put liquidation preferences on TV and make these sexy yet they are key in most deals.  Unfortunately, liquidation preferences are some of the most important terms on the venture capital sheet and can often lead to misalignment of the investor / entrepreneur team on down the line.  Let me explain.

 

A typical liquidation preference sheet will include terms such as a 3X, 1X, participation / no participation and no cap / cap feature.  Let’s take each one of these and explain what they can cause later in terms of investor-entrepreneur alignment.  First, when we invest in a new company we join the ‘stack.’  The stack is the order in which teams are paid back for their investment.  The most recent investors are at the top of the stack of cards and are paid back first.  Then, at the bottom of the stack, are the early stage  / series A / seed funding venture capitalists.  An exit event, most commonly selling the company to a buyer, is the time at which the buyout will occur and the investors have the ability to be repaid with their winnings.  The term sheet specifies on what terms and in what manner the investor will make back their money (and then some we hope–otherwise why assume the risk of the investment).  First, let’s focus on the ‘3X’ or ‘1X’ situations.  The investor will obtain either three times, thus the “3X” or one time (“1X”), or somewhere inbetween, amount for their initial investment.  So, if I give $5000 to a startup and there are no other investors later, and I have a 3X term I will expect to be returned three times my initial amount or $15000 during an exit event.  However, there may also be a cap.  A cap means I can receive no more than a certain amount. So, although this wouldn’t make sense for almost any deal, if I have a $10000 cap, and I have a 3X initial investment on $5000 I could only be repaid $10000 total at the end of the day.  Such deals are not done routinely.  We only use this to highlight the nature of a cap.  The cap says:  “Investors you can only obtain this much money in return total for your initial investment.”

 

More interesting is the concept of participation. That means that an investor will receive their 3X or 1X, sometimes called multiple, of investment return and then will go on to participate in the buyout of the shares (of which they own some) at the price stipulated by the buyer company.  So, in the example given above, the investor would receive $15000 plus the value of their stock as they participate in the stock buyout up to whatever cap exists if a cap is in the terms sheet.  No participation means they would receive nothing for that stock.  Keep in mind the investors are paid before the startup team in all scenarios. So, the startup team and the investor can easily become misaligned.

 

If you do the calculations for various sale prices for companies with various liquidation preferences in the term sheet, sometimes the amount the investor makes on the deal will not vary over a certain number of sale prices.  What I mean is that if a company sells between $1 to $5 million the venture capitalist may have the exact same return on the investment depending on their liquidation preferences per the term sheet.  Ut-oh.  Misalignment!  Where the startup team may want to hold out for a 5 million dollar exit (remember the startup team is paid last so they likely wish to hold out for more), the VC group may be content to sell at a much lower price.  After all, the VC makes the same amount at a 2-5 million dollar sale price depending on the liquidation preferences.

 

Do some math with the 3X, 1X, participation and cap versus no cap situation and you will see this to be true.  Again, this may cause misalignment when it comes time to sell a company.  We can’t highlight this enough:  the investors may wish to sell at a much lower price than the startup team because they will obtain the same amount in return regardless of the sale price. The investor team may want to hold out for more reimbursement for their company and yet the investors may wish to sell.  Clearly, pardon our repetition here, liquidation preferences are a source of misalignment.

 

So, on Shark Tank, liquidation preferences are never mentioned.  We can’t blame them!  How could that sort of explanation come across on TV, when many startup business owners are not acquainted with the liquidation preference issues?  In the tank, companies are often valued by saying 100% of the company’s ownership shares are equivalent to their gross sales receipts.  The shark will take that amount of ownership equity for an investment equal to the share price as determined by that calculation, and sometimes the shark also wants a certain amount of royalty for each object sold in perpetuity.  This is very challenging.   The shark becomes a part owner of the company and sometimes even takes majority interest in the company.  Again, usually the liquidation terms will be clear on the initial term sheet.  However, on Shark Tank, liquidation preferences are never mentioned.  Do the Sharks, some of whom are very experienced venture capitalists, expect to simply be reimbursed as if they were a routine owner when an exit strategy is executed? Or do they expect the company to continue ad infinitum and they can take profit as any owner would? It is very difficult to tell and again liquidation preferences would not play well on TV.  Again, in fact, liquidation preferences are often poorly understood by the entrepreneurs who accept investment deals. This is one of the more challenging aspects of venture capital and entrepreneur interaction.  It’s no surprise that some Shark Tank deals despite a handshake on TV, break down in the due diligence process or later once the show is over and the lights are off.  There is a lot more to a deal than how things look on the tank on TV, and that’s ok.

 

Let’s return to valuation:  one of the other challenging aspects of Shark Tank is the valuation.  Valuations are done very strictly and are often based on gross revenue.  Sharks often incredulously ask owners of the company how they can value a company so highly when it has done so little in sales.  For now please recognize that this is a very conservative, pro-Shark method of valuation.  It does not make the Sharks wrong or “evil”.  In fact, they are simply trying to ensure that they are being conservative investors in this respect.  They only have a brief window on which to evaluate a company for significant investments.  If it were you, wouldn’t you adopt a valuation method that is conservative in your favor?

 

However, I would like you to recognize and perhaps search online for methods of valuation for different companies. I think you will find that very enlightening and different than the seemingly straightforward method of valuation that the Sharks use on TV.  Again, I really enjoy Shark Tank and I think it is excellent because it exposes America to certain ideas about investing, building a business, and innovation.  It shows personal stories of Sharks and future millionaires who have worked hard and brought a company very far.  It really highlights the American dream.  Certain limitations are imposed by the fact that it must make for good, watchable, TV.  And I respect that.  It’s more valuable to have an entertaining show that demonstrates the basics of investing and business building rather than have a perfect show that focuses on liquidation preferences, term sheets and valuation models.

 

A lot of what the Sharks talk about is interesting to me for other reasons.  Remember, it is very challenging for a Shark to hear a cold pitch and determine whether to invest or not.  In this respect I don’t blame them, and as a matter of fact compliment them, on using conservative valuation methodology.  This is because they have not built personal relationships with these entrepreneurs.  Recall they have to decide in a relatively short amount of time, if the TV show runs as advertised, whether they will or will not invest in a given situation.  This makes valuation very difficult and makes the deal fraught with much more risk than you would typically have up front in a deal.  As was recently explained in Venture Capital 101 on Coursera by the Ulu Ventures team:  the bulk of cold calls deals, or deals that are not introduced to venture capitalists by friends or others in the VC community, typically go bad and are low-quality investments.  This is why the venture capital community typically gives a “no” 99% of the time to deals which do not come to them from friends or contacts.  Although we don’t know all the background on Shark Tank and we don’t know how entrepreneurs are vetted before they are on TV, if Sharks truly have such a brief time to vet the deal and decide whether to invest we can understand and appreciate why they are so conservative in the valuations.  My bet is that entrepreneurs and teams are often selected based on their story, the quality of the potential deal, their looks, and other things that may make them more TV friendly with only the investment opportunity as one parameter.

 

In the end, Shark Tank is an excellent TV show that exposes America to the investment community, some of the ways in which investment deals are done, and highlights many of the tenants of the American dream.  This useful, excellent show has certain constraints owing to the fact that it functions popular TV show.  Again, the dramatic music, the over the top pitches, and sometimes even the Sharks’ responses (I don’t recall as many surprised whooooaaa’s from the Sharks in the first season), are meant to heighten drama and interest in this excellent show.  Rather than feeling this detracts from the show, I feel it entices a broad audience to understand and be exposed to the rudiments of investing.  The limitations of Shark Tank include the valuation methodology and the absence of discussion of liquidation preferences to name a few, yet we can all enjoy this excellent TV show and that brings real value to us as an audience.

 

Don’t Just Decide With Your Gut: A Decision Tree Is A Great Tool

 

I have a personal interest in decision making that started from an MBA course on decision analysis.  It made me fascinated with the idea of advanced techniques like decision trees with conditional probability versus making decisions with our intuition alone.  One of the more interesting things that I have run across that I want to share with you includes the idea that the human mind is a programmed coincidence machine.  That is, because of how we have evolved, we are set up to notice unusual cases.  For example, when lightening strikes a log we imagine fire.  However, commonly, there is not fire when lightening strikes.  In short, rather than notice the everyday, mundane, central tendency of a set of occurrences, we recognize the exception to the rule. This is very normal and it is just how, some say, we are built. There are entire books written about intuition versus more explicit decision making.  Decision making tools can help us move beyond how we evolved and how we simply react.  Here we explore one tool of explicit decision making and its far-reaching effects.

 

Rigorous decision making has several advantages, including the fact that it is more easily taught, can make our assumptions in decision making very clear, and can force us to be reproducible in our decision making.  It also allows us to demonstrate how we arrived at a decision very clearly.  One of the most useful tools I have found to achieve these ends is the decision tree.

 

The decision tree utilizes conditional probability to demonstrate the expected payoff for each possible occurrence in a scenario.  Above, I have included a sample from one of our team’s projects.  The question is should we invest or not invest in a given startup.  You can see, as we work from left to right, we have several different scenarios outlined. Each scenario has an expected payout and a probability of that expected payout occurring.  We are able to multiply the expected payout from each branch, times the probability of it occurring plus at each node the similar risk adjusted probability from the other branch.  This is called rolling back the decision tree. Eventually, we will obtain an expected payoff from each branch.  Here we weigh the expected payoff from investing in a company versus the expected payoff for a more passive income type investment such as investing in mutual funds or a buy and hold strategy in equity based securities.  The specifics of what the decision is are not as important as the fact that we can frame it this way and do the best we can to find high quality probabilities for each event occurring.  The returns from the stock market are fairly well known, and, by the way, don’t bank on the returns we saw in 2013 continuing.  In any event, we can plug in high-quality probabilities to the equity back securities branch (invest in the stock market branch) fairly directly.

 

Calculating the expected payout from the investment in the new company strategy is more challenging.  We are using a home run, base hit, strike out type mentality and nickname each of our branches in that way.  Over time, we have come to describe a home run return as one with a 9 x MOI (Multiple Of Investment) return.  We are very conservative in assigning our probabilities such that we can make the highest quality decisions.  To be conservative, we place this as a low-probability event on the decision tree.

 

Another useful property to emerge from this is that we are able to examine our assumptions and the probabilities we use in terms of how they affect the model. This is called a sensitivity analysis.  We can vary the probabilities and determine if changing the probabilities changes our ultimate answer.  Interestingly, sometimes in very complex decision trees, changing our underlying probabilities does not effect the outcome owing to the magnitude of the expected payout, or other probabilities, etc.  This is always enlightening and shows us that no matter whether we agree on a certain probability our end decision does not change.  This has always been personally fascinating to me and has lead our angel team to what we feel, overall, to be higher quality investment decisions.  I invite you to read more about conditional probability along with decision trees, and the progenitor of many of these techniques:  John Nash (who received the Noble in part for his work on probabilities in Economics).  Consider watching the hollywood movie version of Nash’s life, A Beautiful Mind, if you have some time.

 

I think you will find the decision tree very useful as has our team.  A recent venture capital course, Venture Capital 101, given on Coursera.org highlighted how Ulu ventures and other venture capitalists use similar techniques in their vetting of high quality deals and of decision making in an invest or don’t invest type scenario.  Clearly this type of work would not play well on TV and is not readily performed for investment related TV shows like Shark Tank.  It would be very challenging to explain this in a way that is palatable to a broad audience.  However, conditional probability diagrams and decision trees such as this are very useful to obtain high quality decisions in investment arenas, healthcare, and many other endeavors.  Decision making tools can help us move beyond our old mammalian brain and make higher quality decisions.

 

Speaking of Shark Tank, our next blog entry will include some thoughts on Shark Tank and its positives for innovators as well as entrepreneurs.