By: David Kashmer, MD MBA FACS (@DavidKashmer)
A colleague asked: “How will you know if this gamification project is successful?” and it got me thinking…how do you track success in a re-designed environment that uses some techniques typically seen in gaming? How do you track, measure, and improve employee engagement or, better yet, that elusive endpoint called culture?
Asking For Endpoints
In case you haven’t seen case made regarding the “why bother with gamification”, look here. If you believe the sobering statistic that 70% of US workers are actively trying to hurt their company, then you know that something must be done. Even if the issue is simply emotional detachment from work (rather than actively attempting to hurt the company), and missed opportunities for your company, then it truly is worth trying some tools to help improve the situation. A recent Deloitte report (2015) highlights that issues of employee engagement and culture have become “the number one challenge in the world [of business]”.
In the quest to do something about it on the healthcare side I’ve helped deploy gamified systems before. (More on the experience here.) Recently, while setting off on a deployment, a colleague who was in favor of the new system asked: “But how will we know if it works?”
What’s Been Done Before
The game endpoints, to tell whether the system has accomplished your organization’s goals, should be specific to your company’s needs. Do you need staff to complete the yearly compliance work? Then there’s an endpoint. Do staff need to treat each other differently? That may be tough to measure…
After all, these things center around that notorious intangible called “culture”. And how, in short, do you measure that?
In a previous post, I discussed attempts to measure job satisfaction using a standardized questionnaire that has previously been used & validated in healthcare called (simply) the Job Satisfaction Survey (JSS) pre and post game deployment. (For more on that, look here or here.) Lots of famous business journals and authors write about how to measure culture. What my colleague had asked seemed to have no simple solution: do we measure something as simple as percent compliance with yearly training, as complex as some global measure of culture, or both?
Nuts & Bolts Of Endpoints
Clearly we are out of the realm of easy or straightforward science & testing here! Consider, for example, a seemingly straightforward move like administering the JSS before gamification and then after the system winds up. Let’s pretend, at the end of the deployment, there has been statistically significant improvement in many of the scores on each question from the JSS. Great…except, well, a lot of things in your organization likely changed over the time the game system was deployed. How do you know whether the system was really the driver of the improvement?
Now let’s take the more simple endpoint: compliance with yearly corporate training such as fire safety training. How do we know that, after the deployment, that any increase in fire safety training compliance was due to the system and not just passage of time with more participants completing the training?
The bottom line, here, is that there may be no perfect endpoint for the game system. Even endpoints that seem straightforward, such as participants returning to a certain place at a certain time, or reviewing certain materials, is just as prone to criticism as endpoints of typical work across the sciences.
Another important consideration is timing. Consider, for example, an endpoint that your organization truly values such as employee churn. Perhaps churn has reached a steady state over the last year prior to the new system you’re deploying. After the system (which directly impacts participants considered in the churn metric) was deployed, the rate of employees leaving the organization dropped sharply. This may have meaning in your individual system. So, another important consideration in these metrics is timing: choose something on which you can reasonably expect an impact from the new game system, which is already measured / important to the organization, and which has achieved a steady state.
Just as challenging as measuring culture in your organization is the measurement of endpoints to determine how successful your gamified system is. I recommend a combination of endpoints that compare post-game performance to important measures that have achieved stability over time prior to game system deployment and which you can reasonably expect a change related to the new system.
This is no easy task! Gamified systems, often designed to impact culture and organizational behavior, can be challenging to quantify owing to all the vaugeries of measuring culture in general. Consider the Job Satisfaction Survey, in addition to more specific endpoints rooted in quantifiable behaviors, to get a sense of the performance of your gamified system prior to update, revision, improvement, and release of any version 2.0 you have planned.
Questions, comments, or thoughts of endpoints for gamified systems in healthcare or in general? Wonder how to deploy a gamified system to promote engagement and certain actions in your organization? Email me or comment beneath.