Leapfrog Releases Hospital Safety Grades - New York's Best Is Not Good Enough

It has been a tough few weeks for New York City. Masa, the most expensive sushi restaurant in the city, was found to have a serious health code violation for adulterated fish by the FDA. Jean-Georges, one of the city’s oldest and best French restaurants lost one of their three Michelin stars. And oh yes, the three top healthcare systems in the city where given a “C” rating for patient safety by the Leapfrog group. This last fact probably impacts more New Yorkers than the first two, but then again, with the vagaries of hospital ratings and rankings, maybe not.

I have written about hospital ratings previously, here and here. The hospitals are rated and ranked by a number of sources, the Leapfrog group, Consumers Report, HealthGrades, the Centers for Medicare and Medicaid (CMS) and of course US News and World Report that has turn rating of all sorts of institutions a cottage industry. For the most part, the information used by these authorities come from CMS with some additional measures that each group adds into the ‘secret sauce’ of their score. The secret sauce includes the opinion of physicians or the extent to which hospitals have installed computerized physician ordering systems, or even whether specialist run the Intensive Care Units. Even so, the majority of measures are taken directly from CMS, so I would expect that the rankings, even by different agencies, would be reasonably similar. Not so, as it turns out. Consistency among the various raters is hard to find, in part, because of the secret sauce additions, but more because each authority weighs the factors differently. As I have said before, some do not believe the patient deaths should carry as much weight as the opinion of the referring physicians. So I spent a bit of time looking at a group of hospitals, their raters, and their ranks and here is what I found

With respect to the raters:

• The Leapfrog Group – arguably the oldest organization to rate hospitals. It formed in 1998 by business leaders trying to control their costs by identifying best hospitals.
• US News and World Report – a news outlet that encountered cash-flow problems in the 80’s but with a change in management “flourished and expanded the Best Colleges franchise and launched Best Graduate Schools and Best Hospitals.” When I Googled, history US News and World Report, the first entry was Best Graduate History Programs – enough said.
• HealthGrades – “the leading online resource for comprehensive information about physicians and hospitals” Their words, not mine.
• Consumer Reports (CR)– “Consumer Reports is an independent, nonprofit organization that works side by side with consumers to create a fairer, safer, and healthier world. …Unconstrained by advertising or other commercial influences” Again, their words
• Joint Commission - An independent, not-for-profit organization, The Joint Commission accredits and certifies nearly 21,000 healthcare organizations.”
• CMS – The government agency that administers our Medicare and Medicaid programs, the collector of the bulk of the information and the most significant purchaser of healthcare accounting for 37% of all healthcare expenditures.

With respect to the rankings:

I took the top twenty hospitals as rated by US News and World Report, that is the top 0.5% of hospitals and compared the rankings by the other groups. Because some ratings are reported on five-point scales (CMS’s stars and Leapfrog’s letter grades), and others used a 100 point scale (Consumer Report and HealthGrades), I converted the later to a 5 point scale to make them roughly comparable. The Joint Commission certified all the hospitals, so they really did not rank any of them in a useful way for patients. Now because the top twenty hospitals represent such a small, small fraction of the total I expected the rankings to be similar across all the raters, but here is what I found. Five is good; one is bad in this graph.

You can see as you look across any row that the rankings and their ranges were not so similar. I took it a step further and compared each of the ratings to one another. If hospitals are preparing for the test, who are the tough graders. 

Here is how it breaks down. Consumer Report is the toughest grader, followed by CMS, HealthGrades and then Leapfrog. If you want to put numbers on the differences, CMS ratings were 14% higher than CR, HealthGrades 26% higher and Leapfrog, the organization that just gave all three major New York City health systems a C rating, they rank those top 20 hospitals 28% higher than Consumer Report. That is a classification level or more. Bottom line, there is no consistency amongst these ratings. 

What is a patient supposed to do? Who do you believe?

Understanding how successful a hospital is regarding preventable health problems for patients is essential. There is no one metric or even a current collection of measures that adequately express a hospital’s safety. There are a number of organizations trying to communicate this type of information to patients, why are they not sitting down as a group and developing standards and weighting of metrics. It would seem to me that this is a necessary first step. Perhaps physicians should get over their reluctance in addressing the errors of our health systems and assume a more thoughtful and active role rather than leave this up to organizations that are more involved with media than health.