How to Measure Telehealth Success

Not everything that can be counted counts.
Not everything that counts can be counted.
— William Bruce Cameron

As the quote highlights, measuring success can be a tricky business. Not all metrics that can be collected are meaningful and some relevant aspects of telehealth success are difficult to measure accurately.

In a conversation with the CEO of a mental health agency, we discussed his need for good measures for the C-suite to ascertain the health and success of their telehealth services. His goals? To gain more insight into this strategic priority and knowing whether, when, and where to allocate more resources to create an acceptable performance – or “success”.

As many of the Telehealth Tuesday articles have discussed, there are a multitude of components at play to make a telehealth service work: workflows, policies, provider engagements, technology, and reimbursement just to name a few. When telehealth is defined as “delivering care at a distance”, it cannot be reduced to the mere video chat of a clinician with a patient (or client, in the mental health world.

Rather “telehealth” is the totality of the workflows and the sites, the technologies and the policies, the scheduling and the billing.

But how do you measure success?

Input, Output, Quality, and Outcome Measures

We’ll get back to measuring success in a minute, but let’s first briefly cover the basics of measuring success.

Ultimately we are talking about measuring the performance of a telehealth service. There are a number of types of performance measures, some of which are useful and some are essentially meaningless (i.e., they don’t count – per the quote above).

Here are the four most relevant types of measuring telehealth performance:

• Input Measures – resources (e.g., time, staff, money) spent on delivering telehealth
• Output Measures – quantifiable results, e.g., the number of visits or patients seen
• Quality Measures – characteristics of the results, e.g. patient or physician satisfaction
• Outcome Measures – impact and value generated, e.g., improved health, reduced ED visits

The first two — Input & Output Measures — are relatively easy to come by and are what is most often measured. How many providers were trained? How many visits did we have? How many hours did staff spend on seeing patients? How much did we spend on telehealth licensing fees? How much revenue did we generate?

In one example of a telehealth outcome measure, one of the longest-running telehealth programs in the US continues to measure the number of miles saved by patients (and providers) that did not have to drive to their appointment. While quite an impressive metric (I think they are cumulatively 80% their way from Earth to Mars), it bears by itself very little value. What can be derived from it, though, is driving time saved (and the resultant personal or communal productivity gain), gallons of gasoline saved, statistical car accidents avoided, etc. Which are the more meaningful outcome measures.

The other two types of performance measures — Quality & Outcome Measures — are, however, much more relevant to assessing the success of a telehealth service, though they require a little bit more effort to collect, analyze, and interpret.

The most useful examples of Telehealth Quality measures include the satisfaction of patients, of providers and of the staff (and we’ll dive deeper into these measures in a future article). The outcome measures can be about continuity of care or potentially avoidable utilization, such as an ER visit or an admission. Patient engagement is another outcome measure, as is the percentage of telemedicine visits that were reimbursed.

How do you measure success?

In the most basic definition in the context of today’s discussion, I simply define success as meeting or exceeding a pre-defined goal. Without targets, no success (or failure) can be declared.

Is 27 visits good? A Revenue of $3,853? 3.8 televisits per provider per day? A Net Promoter Score of 47?

We don’t know.

We can only measure success when we set ourselves a goal and then see if we can achieve it.

But even setting goals is fraught with challenges. How do you know what a good goal is? How do you objectively and meaningfully set a goal (which is a topic for another article).

Action: The Ultimate Value of Measuring Performance

When you are investing staff’s and leadership’s time into measuring anything, there is only one valid business reason to justify the effort that goes into the collection, analysis, and reporting of metrics: Clarity of Action. Measuring needs to drive actions.

As we work with client to establish telehealth performance dashboards, we advise all of them to identify for each metric the four following aspects:

• Ownership: who is the executive owner, who is the operational owner, and who is the data steward of that metric?
• Definition: how is the metric collected and analyzed? how often? how should it be reported?
• Expectations: what are the targets for acceptable performance? for success?
• Actions: what actions will be taken if the metric does not meet the target? what if it exceeds the target? what if it hits the target?

Especially the last aspect of pre-defining the action to be taken is what distinguishes a great performance management program from an average or mediocre one: If there is no clear pre-defined action, the organization risks wasting valuable time and resources on the collection and analysis of the data, resulting in data fatigue. Any insights that could be gained from the data are lost, since without clear action, not much attention is paid to the data.

Also, once you ask the question of “what actions will we take”, so-called “vanity metrics” quickly disappear, because they often don’t drive any meaningful action.

To use the example from above, a saving of “1 Million miles driven” is definitely a neat round number, but how would we know that it shouldn’t be 2 Million miles? Or that some visits should have occurred in person and led to negative health outcomes?

And where should we set our target and our resultant actions? 20,000 patient miles saved a month? Why not 18,000 or 24,000? And what actions will we take if we don’t meet the target – recruit patients who live farther away? Or, worse, ask those who live closer to come in, so they don’t count against our quota (i.e., maximizing miles saved per televisit)?

At the outset, the targets and actions may not be clear (e.g. when you start to get a year-round feel for the number of video visits for each specialty and clinic). But over time, it should become clearer what expected and great performance should look like.

Key Success Metrics for Telehealth

Now that we have laid the groundwork for creating a Telehealth Success Dashboard, I will focus an upcoming article on developing a framework of metrics to include the most valuable and meaningful input, output, quality, and outcome measures.

Along with the pre-defined targets and actions to accompany each metric, that is how each and every healthcare leadership team should manage their telehealth services to enable the delivery of extraordinary care.