Latest News

Benchmarking: (How) Can It Be Used to Improve Healthcare Performance?

By Kevin Schulman, MD, MBA, Bivarus cofounder

In my experience, the best managers crave feedback in order to evaluate their strategy and tactics. They link operational choices to customer response in order to assess performance. Top managers know that feedback can help refine products and services to improve their position in the marketplace or within their given industry.

In the technology industry, using data to help make strategic decisions has long been an accepted practice. For example, Google is famous for its A/B testing to drive the development of its products and profitability.

In health care, data-driven strategies are slowly gaining traction in the marketplace. We’re entering a world where feedback is increasingly possible, both with regards to clinical performance and patient experience, with a platform like Bivarus. In just a few short years, we’ve evolved from a world of manual chart review and paper-based patient surveys to one of electronic performance data and real-time patient feedback.

While our knowledge of the importance of data has changed and the data itself has become more accessible, our understanding of how to use data to improve performance is still lagging significantly.

One of the most common health care applications of data is the idea of benchmarking performance compared to peers. The concept gained industry visibility when Xerox adopted benchmarking of its performance on the way to winning the Baldrige Award, which recognizes US business, health care, education and nonprofit organizations for performance excellence, in 1989. Xerox referred to the practice as “competitive benchmarking,” defining the set of competitors and standards the company wanted to meet.

Historically, benchmarking was a means of setting internal standards, not of evaluating relative performance. Competitive benchmarking was not the only analysis technique adopted by Xerox; the company adopted quality improvement as a core principle and implemented statistical process control methodology along with the competitive benchmarking effort.

In my estimation, applications of benchmarking in health care fall far short of this comprehensive approach to quality management. We often hear of managers touting their performance percentile relative to some benchmark data set of “peers.” Benchmarking very rarely incorporates the competitive approach established by Xerox to elevate overall performance; instead, it focuses on relative rankings.

Benchmarking without performance management has a serious flaw: It tells us where we are, but not what to do or where to go.

Take baseball, for example. My son is a serious sports fan, and often he stares at statistics in the sports pages. Inevitably, he looks at data like this:

Benchmarking - Sports Statistics

As a Mets fan, there are a lot of data in this snapshot: how many wins the team has relative to others, how they have done recently and how many games they are behind the leader. However, there are critical gaps in these data. Given only this limited data set, what would he tell the manager about how to improve?

Does he need more pitching, better hitters or better defense? Is there a problem with their strategy of base stealing and advancing the runners? Is there a problem with the players, the manager, the trainers or none of the above? We know with confidence that they are in third place and 17 games behind. Other than a few definitive bits of data, we have no actionable insights on how the team can improve their performance and win more games.

Benchmarking can easily rank five professional sports teams, but it is more challenging to rank large numbers of providers, whether hospitals or individual physicians. Our research found considerable overlap between hospitals: 91 percent of a large sample of hospitals had HCAHPS scores between 8 and 9. It’s not clear if there is any meaningful clinical distinction between hospitals in this high-performing group, yet we still seem compelled to rank their relative performance. This type of comparative benchmarking has none of the attributes of Xerox’s competitive benchmarking approach to setting absolute performance standards.

What about ranking physicians? Using one performance tool, we estimated that a 0.1 difference in a 10-point score like CG-CAHPS would drop a physician 100,000 places in a ranking of all US physicians. Unfortunately, benchmarking simply based on scores is not the same as distinguishing clinical performance between providers.

So what perspective should health care organizations take on benchmarking? We need to consider the original concept established by Xerox: using the concept of benchmarking to assess our performance compared to standards set by us as managers, or, even better, by our customers. We need to focus on the idea of a comprehensive approach to quality management and using actionable insights to drive continuous quality improvement and performance within our health care organizations. Let’s get in the game of improving performance, not just comparing performance.