Not everything that can be counted counts.
Not everything that counts can be counted.
— William Bruce Cameron
As the quote highlights, measuring success can be a tricky business. Not all metrics that can be collected are meaningful and some relevant aspects of telehealth success are difficult to measure accurately.
In a conversation with the CEO of a mental health agency, we discussed his need for good measures for the C-suite to ascertain the health and success of their telehealth services. His goals? To gain more insight into this strategic priority and knowing whether, when, and where to allocate more resources to create an acceptable performance – or “success”.
As many of the Telehealth Tuesday articles have discussed, there are a multitude of components at play to make a telehealth service work: workflows, policies, provider engagements, technology, and reimbursement just to name a few. When telehealth is defined as “delivering care at a distance”, it cannot be reduced to the mere video chat of a clinician with a patient (or client, in the mental health world.
Rather “telehealth” is the totality of the workflows and the sites, the technologies and the policies, the scheduling and the billing.
But how do you measure success?
Input, Output, Quality, and Outcome Measures
We’ll get back to measuring success in a minute, but let’s first briefly cover the basics of measuring success.
Ultimately we are talking about measuring the performance of a telehealth service. There are a number of types of performance measures, some of which are useful and some are essentially meaningless (i.e., they don’t count – per the quote above).
Here are the four most relevant types of measuring telehealth performance:
Input Measures – resources (e.g., time, staff, money) spent on delivering telehealth
Output Measures – quantifiable results, e.g., the number of visits or patients seen
Quality Measures – characteristics of the results, e.g. patient or physician satisfaction
Outcome Measures – impact and value generated, e.g., improved health, reduced ED visits
The first two — Input & Output Measures — are relatively easy to come by and are what is most often measured. How many providers were trained? How many visits did we have? How many hours did staff spend on seeing patients? How much did we spend on telehealth licensing fees? How much revenue did we generate?
In one example of a telehealth outcome measure, one of the longest-running telehealth programs in the US continues to measure the number of miles saved by patients (and providers) that did not have to drive to their appointment. While quite an impressive metric (I think they are cumulatively 80% their way from Earth to Mars), it bears by itself very little value. What can be derived from it, though, is driving time saved (and the resultant personal or communal productivity gain), gallons of gasoline saved, statistical car accidents avoided, etc. Which are the more meaningful outcome measures.
The other two types of performance measures — Quality & Outcome Measures — are, however, much more relevant to assessing the success of a telehealth service, though they require a little bit more effort to collect, analyze, and interpret.
The most useful examples of Telehealth Quality measures include the satisfaction of patients, of providers and of the staff (and we’ll dive deeper into these measures in a future article). The outcome measures can be about continuity of care or potentially avoidable utilization, such as an ER visit or an admission. Patient engagement is another outcome measure, as is the percentage of telemedicine visits that were reimbursed.
How do you measure success?
In the most basic definition in the context of today’s discussion, I simply define success as meeting or exceeding a pre-defined goal. Without targets, no success (or failure) can be declared.
Is 27 visits good? A Revenue of $3,853? 3.8 televisits per provider per day? A Net Promoter Score of 47?
We don’t know.
We can only measure success when we set ourselves a goal and then see if we can achieve it.
But even setting goals is fraught with challenges. How do you know what a good goal is? How do you objectively and meaningfully set a goal (which is a topic for another article).
Action: The Ultimate Value of Measuring Performance
When you are investing staff’s and leadership’s time into measuring anything, there is only one valid business reason to justify the effort that goes into the collection, analysis, and reporting of metrics: Clarity of Action. Measuring needs to drive actions.
As we work with client to establish telehealth performance dashboards, we advise all of them to identify for each metric the four following aspects:
Ownership: who is the executive owner, who is the operational owner, and who is the data steward of that metric?
Definition: how is the metric collected and analyzed? how often? how should it be reported?
Expectations: what are the targets for acceptable performance? for success?
Actions: what actions will be taken if the metric does not meet the target? what if it exceeds the target? what if it hits the target?
Especially the last aspect of pre-defining the action to be taken is what distinguishes a great performance management program from an average or mediocre one: If there is no clear pre-defined action, the organization risks wasting valuable time and resources on the collection and analysis of the data, resulting in data fatigue. Any insights that could be gained from the data are lost, since without clear action, not much attention is paid to the data.
Also, once you ask the question of “what actions will we take”, so-called “vanity metrics” quickly disappear, because they often don’t drive any meaningful action.
To use the example from above, a saving of “1 Million miles driven” is definitely a neat round number, but how would we know that it shouldn’t be 2 Million miles? Or that some visits should have occurred in person and led to negative health outcomes?
And where should we set our target and our resultant actions? 20,000 patient miles saved a month? Why not 18,000 or 24,000? And what actions will we take if we don’t meet the target – recruit patients who live farther away? Or, worse, ask those who live closer to come in, so they don’t count against our quota (i.e., maximizing miles saved per televisit)?
At the outset, the targets and actions may not be clear (e.g. when you start to get a year-round feel for the number of video visits for each specialty and clinic). But over time, it should become clearer what expected and great performance should look like.
Key Success Metrics for Telehealth
Now that we have laid the groundwork for creating a Telehealth Success Dashboard, I will focus an upcoming article on developing a framework of metrics to include the most valuable and meaningful input, output, quality, and outcome measures.
Along with the pre-defined targets and actions to accompany each metric, that is how each and every healthcare leadership team should manage their telehealth services to enable the delivery of extraordinary care.
To receive articles like these in your Inbox every week, you can subscribe to Christian’s Telehealth Tuesday Newsletter.
Christian Milaster and his team optimize Telehealth Services for health systems and physician practices. Christian is the Founder and President of Ingenium Digital Health Advisors where he and his expert consortium partner with healthcare leaders to enable the delivery of extraordinary care.
Contact Christian by phone or text at 657-464-3648, via email, or video chat.
Christian Milaster optimizes Telehealth Services for health systems and physician practices as the interim Telehealth Program Director. He serves as a Digital Health & Telehealth Advisor to startups and established Health IT firms.
Christian is a Master Builder of Digital Health and Telehealth Programs and the Founder and President of Ingenium Digital Health Advisors, a boutique consultancy focused on enabling the effective delivery of extraordinary care through workflow optimization and the judicious use of technology.
Born, raised, and educated as an Engineer in Germany, Christian started his career at IBM Global Services before joining the Mayo Clinic in Minnesota, where he worked for 12 years in various roles before launching Ingenium in 2012.