Each year, US News and World Report publishes its list of the top 50 hospitals in various specialties (example here). Now, an article has been published suggesting that one aspect of the methodology used by the magazine is flawed.
"The Role of Reputation in U.S. News & World Report’s Rankings of the Top 50 American Hospitals," by Ashwini R. Sehgal, MD is in the current edition of the Annals of Internal Medicine. (You can find an abstract here, and you can obtain a single copy for review from Dr. Sehgal by sending an email to axs81 [at] cwru [dot] edu.)
Dr. Sehgal finds that the portion of the U.S. News ranking based on reputation is problematic because reputation does not correlate with established indicators of quality:
The relative standings of the top 50 hospitals largely reflect the subjective reputations of those hospitals. Moreover, little relationship exists between subjective reputation and objective measures of hospital quality among the top 50 hospitals.
More detail is provided in the article:
The predominant role of reputation is caused by an extremely high variation in reputation score compared with objective quality measures among the 50 top-ranked hospitals in each specialty. As a result, reputation score contributes disproportionately to variation in total U.S. News score and therefore to the relative standings of the top 50 hospitals.
Because reputation score is determined by asking approximately 250 specialists to identify the 5 best hospitals in their specialty, only nationally recognized hospitals are likely to be named frequently. High rankings also may enhance reputation, which in turn sustains or enhances rankings in subsequent years.
Given the importance attributed to the U.S. News ranking, this article is bound to raise concerns. I know that the folks at the magazine have worked hard over the years to make their rankings as objective as possible, and it will be interesting to see their response to Dr. Sehgal's critique.
Paul, this is analogous to the BEST DOCTORS issues of urban magazines, where quality is inferred (but never measured) in what in reality is a popularity contest. Let's be honest and compare infection rates, readmission rates, and other hospital/doctor-associated injury rates.
ReplyDeleteMeanwhile, hospitals raise a banner of the US News ranking over the door, but few inside actually know the data of their hospital's true performance. Does every member of the team know last month's infection numbers? What ball do they have their eye on?
From http://www.fiercehealthcare.com/story/Hospital-rankings-by-U.S.-News-and-World-Report-rely-on-reputation-rather-than-quality/2010-04-20
ReplyDeleteAvery Comorow, health rankings editor for U.S. News, disagrees.
"He doesn't look at reputation the same way we do," Comorow told the Cleveland Plain Dealer. "What we ask doctors to do is provide us with the names of hospitals [for patients] who need the absolute highest level of care for a complex condition or a complicated procedure....The bottom line is that we see reputation as a legitimate form of peer review for patients."
Where to begin? Anonymous wrote: "Let's be honest and compare infection rates, readmission rates, and other hospital/doctor-associated injury rates."
ReplyDeleteYes, we should do that. Except that infection rates, reliable or otherwise, are unavailable for the vast majority of hospitals. Readmission rates certainly are worth study and concern, but largely meaningless until the irreducible core rate for each hospital can be determined; otherwise good hospitals with high readmit rates because of an especially problematic patient population (elderly, noncompliant, many comorbidities...) will be penalized. I think Paul would agree that data on HACs is still immature. There still isn't agreement on the "right" mix of PSIs, for example, or on good definitions of condition-specific complications.
What is interesting about all of these elements is that they are related to process as well as to outcomes. A hospital that strives to minimize harm to patients will have a culture and procedures in place that result in fewer infections, falls, complications following surgery, etc.
The reputational survey is the proxy, in the Best Hospitals methodology, for this process component. The purpose of our rankings is not to identify institutions that do the best on routine care (although the author of the Annals study creates the impression that it it, by misstating the survey query sent to physicians). We're looking for top referral centers--hospitals that handle the most challenging patients and perform the most difficult procedures. We believe that over time, hospitals entrusted with such patients develop a standing with specialists as places that do things the right way most of the time. And so we see the reputational survey, as I said in the quote you posted above, as a form of peer review.
Considering that, how surprising is it that the high reputational scores for a relative handful of hospitals, all of which are regional and often national and international referral centers, drive them to the top of the rankings?
Of course reputation is subjective, and we are taking steps to reduce its impact. But it's not as squishy as might be thought. The scores, based on the most recent three years of surveys, are remarkably stable over time. Nor are they volatile year to year. The study author wrote that "[b]ecause reputation can deteriorate quickly from a single negative event, even highly rated hospitals should be concerned about rankings based primarily on reputation." It is evident from the 20-year history of the rankings that physicians know very well how vulnerable even the best centers are to tragedy resulting from human error. There is no pattern of plummeting scores following unwarranted patient deaths at ranked hospitals.
If reputation is the primary driver, moreover, how is it that in all 12 of the specialty rankings that use objective data, many hospitals with reputational scores of zero, plus others with scores of 1 percent or below, are ranked among the top 50--above other hospitals, ranked and unranked, that register on the reputational seismometer? In orthopedics, for example, there are 12 hospitals with no reputational score and 8 others with scores below 1.0. Their other data put them there.
I am not versed in statistics and will not comment on the methodological validity of the analysis in Annals, other than to say it raises questions better handled by the contractor that creates the Best Hospitals rankings for U.S. News. My intent here is to put reputation in context in our rankings methodology.
Thank you so much, Avery, for taking the time to comment here.
ReplyDeleteMr. (?Dr.) Comorow;
ReplyDeleteI admit I have not bothered to research the answer to my questions, but I wonder if, concerning reputation, you are asking cardiologists about heart hospitals, orthopedists about spine hospitals, etc. I assume so. Also, are your surveys geographically localized at all? For instance, I practiced in metropolitan Washington and, as a practicing pathologist, was better able to make an accurate assessment of tertiary pathologists there with whom I had direct experience, than with, say, pathologists in Boston. I fear a "national" reputation is more likely to be based on hearsay.
nonlocal MD
You can choose to publish this comment or not, but I will ask anyhow (requires some imagination and role playing). Though it may read as if I'm trying to raise a specter of COI, my intention is to allow Dr. Sehgal (and you) to answer obvious objectives that might allow others to dismiss the claim of the article.
ReplyDeleteSo with that:
If Dr. Sehgal was at Clevelend Clinics and not at Case Western, would he have the same perspective and furthermore would he have raised it to the level of publication? and is the publication an editorial or a peer reviewed article (Dr. Sehgal is an Associate Editor of the journal)?
To take this one step further, in your neck of the woods, would you have posted reference to the article on your blog if roles were reversed and you were at say the Brigham instead of BI?
As a methodology/quant guy working for an academic medical center with an acute institutional interest in the US News rankings, I've looked hard at the (ever-evolving) methodology for a number of years.
ReplyDeleteDr. Sehgal's findings aren't surprising, but it IS nice to see them published in a peer-reviewed journal and given some visibility and credibility.
To my mind, the largest problem with the use of the Reputational Score as a proxy for the Process leg of the Donabedian quality model isn't that it's subjective, or even that the (regionally stratified) sample is relatively small ... the real problem is that the design choice to allow each survey respondent to submit only five (5) hospitals' names really truncates the diversity of the response set.
And as a rather direct result, the top 10-12 hospitals (?where you'd wanted to do your residency/fellowship 5 to 20 years ago, if you're a practicing board-certified specialist now?) get 90 to 95% of the total votes in most specialties. Sometimes the Top-Few account for 99% of the vote.
And therefore the hospitals that ought to be ranked from 15 to 100 out of approx 2000 hospitals nationally .... let's call that excellent to very good ... all score a 0/100 on this third of the overall rankings.
Can you win a medal in the Triathlon if you're disqualified from one of the three events? That's exactly what happens here.
In some years, in some specialties, a hospital that ranked LOWER THAN THE NATIONAL MEDIAN PERFORMANCE ON THE "HARD OUTCOMES" has merited a top-20 national ranking, because their above-zero Reputation Score vaulted them past hundreds of other hospitals that were WAY better on the empirical Outcomes.
Mr. Comarow points out that in Ortho, a number of hospitals rank in the Top-50 despite having no score for reputation. True. But how many hospitals that get ANY score for Reputation end up in the invisible portion of the rankings, i.e., below the Top 50? That appears to be very rare.
I must mention that, just as Mr. Comarow has quickly stepped forward to add to the dialogue, the folks involved in the annual US News ranking process have always been open and candid about their methodology; they've always been very aware of the weaknesses, and sincerely interested in improving the approach.
(The Structure leg contributes little separation if you look at just the top-50 finishers; they mostly max out on the scoring for equipment and certifications, etc., or come close to it.)
Now it must be said that nationally available objective numbers for "Process" ... by specialty .. are hard to come by ... and so using some measure of Reputation is ... arguably ... as good as data as can be gotten.
But it occurs to me that the reliance on the 3-part Donabedian model could be worth a new look.
That framework was born in a different era. We now have available many many more credible, peer-reviewed, rigorously applied measures of both quality and process (core measures, PSIs, etc) than were available in Donabedian's day.
So perhaps it's time to ease off significantly on the weighting of the Process/Reputation leg, and put more weight on the "hard" quants for Outcomes and process-defects ... because the measurement science has advanced enough to do so with reasonable confidence.
Still have a long way to go on methodology and risk adjustment, of course. If it was easy, we'd already be done.
Douglas Dame
Shands HealthCare
Gainesville FL
Dr. Anonymous,
ReplyDelete1) I'm not a physician.
2) Your assumption is correct--we survey boarded specialists in their respective specialties. The surveys are randomized, regionalized, and weighted to prevent any region from tilting the scales.
3) I am sure that some part of some responses is based on hearsay, but as noted above, there seems to be a consensus about the hospitals that survey respondents consider best in class.
All this still leaves in my mind the question of whether rankings are objectively related to patient outcome - or inversely related to patient harm. This powerful venue could take the lead and add the weight of transparency of adverse events as a marker of quality. I hope that the patient safety movement will push for the kind of quality data that is obscured to the public by reputational surveys of hospital care.
ReplyDeleteWithin your call to arms to U.S. News to take the lead on incorporating measures of harm to patients lies one of our ongoing dilemmas: to lead, based on reporting that may be spotty and data that is less than robust?
ReplyDeleteSuppose we take the lead. Hospitals that diligently record and report JC-defined adverse events and other safety-related incidents and errors tend to be those in the forefront of the quality/safety movement. They genuinely want to do better, and they agree with leaders like Don Berwick, Peter Pronovost, and Robert Wachter that without good data, you don't know where you (and others) stand.
Hospitals that only pay lip service to quality and safety, on the other hand, reporting sketchily with questionable accuracy, may end up ranking higher and looking better than the true safety leaders.
Yet if we wait until everybody agrees that hospitals are doing a pretty good job of tabulating and reporting, we lose the ability to do as you are justifiably calling for us to do--send a message to hospitals that if they don't improve their patient quality and safety performance, there will be consequences in the rankings.
The most recent rankings do build in a set of PSIs, which we hope sends that message. The weight is small but meaningful, and as the discipline improves, we will enlarge the range of the metrics and boost their weight.
In the pediatric rankings (America's Best Children's Hospitals, we do use quality measures--rates of ICU BSI and of hospitalwide infections, for example. That data comes directly from a lengthy survey we send to the children's facilities. It is largely unvetted by us, but the pediatric hospital universe is small and it is not difficult to pick up data outliers that make us go to the phone and ask questions.
Mr. Comarow;
ReplyDeleteDifficult as it may be, I think you have to take a stab at it, since this is how we want to evaluate all medical care in the future. It sounds like you have already made a start. Gaming the system will exist as long as the rankings exist (witness colleges).
I admit I am a bit of a militant, but you could always say that any hospital which does not collect your desired metrics, or does so poorly, will not be considered for ranking. Then listen to the whining. (:
nonlocal MD
I think we could have an honest debate about the value of reputation as measured by the US News Survey. I believe it is worth a little, but just a little.
ReplyDeleteIt seems reputation surveys will mostly benefit large well known hospitals. And I wonder how many who answer the survey really have significant knowledge about the quality of care at the hospitals they are rating.
But what I really find bothersome is that I believe US News misrepresents the list of America's Best Hospitals as being based on hard data and objective measures of quality. The magazine report emphasizes the top 10 hospitals and the Sehgal study shows quite clearly that the top 10 are determined almost entirely by reputation score. The impact of the objective measures on the rankings is minimal, and its limited influence is almost entirely felt in hospitals ranking below the top 10.
A reading of the material in the actual US News publication and the methodology suggests to readers that the rankings are based on "hard data", and a multitude of quality measures. It is suggested that reputation accounts for about 33% of the score. Perhaps in terms of point count, this is technically true, but it is misleading because of the very high variance of reputation score. Reputation score seems to actually account for around 90%+ of the top 10---the hospitals US News focuses on in its publicity and printed magazine.
It is not clear why US News does not want to acknowledge that its listing of America's Best Hospitals is essentially a reputational survey. It seems the public is being misled by portraying the top 10 as anything more than reputation.
For more comments, see www.geripal.org
Final post absent a clear and pressing need; Paul has to be thinking, "Enough already!"
ReplyDeleteWe do not emphasize or focus on the top 10 hospitals in the magazine or online. As for press releases and other publicity efforts (talk about inside baseball!), do we really want to go for a swim in the frothy waters of public relations? Dr. Covinsky's employer, UCSF, diligently keeps me and other health reporters up on the wonderful advances there...as do many other hospitals, journals, and professional and trade groups, including the occasional list or ranking.
If you look only at the top of the specialty rankings, it is obviously true that reputation is the major factor. As noted above, a consensus concerning the capabilities of certain medical centers is unmistakable. Below the top-ranked few, however, reputation plays a lesser role and the influence of the objective measures is hardly limited.
It's a system dynamics reinforcing loop, it seems:
ReplyDeleteReputation --> Published Rankings --> Increased Reputation --> Higher likelihood of being published rankings
Wasn't there a hospital that was given top marks for a non-existent labor & delivery unit?? Was that Cleveland Clinic? I don't remember the source of that story.
Mr. Comarow,
ReplyDeleteI am disturbed by this comment you made:
"We believe that over time, hospitals entrusted with such patients develop a standing with specialists as places that do things the right way most of the time."
The implicit assumption made here is that the top referral centers will automatically work to improve quality or patient safety to preserve or improve their reputation.
But evidence is to the contrary. Among the US News top 10 hospitals, only 2 made it on the Leapfrog list of top hospitals based on quality and safety...and the Leapfrog list has been around for 6 years now.
Even more discouraging is the low priority that hospital CEOs give to patient safety and quality (http://runningahospital.blogspot.com/2010/03/still-missing-boat.html)
Yes, attracting the superstar specialists with great reputations will probably improve the quality that a hospital delivers. But it's necessary but not sufficient.
We now live in an era where care is so complex that doctors, who often see themselves as "heroic lone healers" will not result in the quality of care that patients need and deserve. (see Tom Lee's recent article in the Harvard Business Review: http://hbr.org/2010/04/turning-doctors-into-leaders/ar/1)
What's needed are systems in place that improve care quality and patient safety. Right now, "top" hospitals can get by with hiring well-regarded specialists and purchasing high-tech treatments without addressing basic systems, such as Computer Provider Order Entry, which have been proven to improve patient safety and quality.
Aaghh! The plot thickens. I am sure anon 5:27 is aware of the increasingly public criticism of CPOE systems and discussion of how to monitor HIT-related errors which harm patients. (I, for one, think they should be regarded as a medical device.)
ReplyDeleteTo the larger point, however, it seems U.S. News could address all these criticisms by simply eliminating reputation as an indicator. After all,if it forms such a small component of the rankings and generates so much criticism, how valuable is it? It would also assist the push toward more objective measurements.
nonlocal MD
This is what I get for not reading Paul's blog for a few days. I had the opportunity to do a fairly in-depth review of hospital quality indicators for JAMA last year.
ReplyDeleteMany of the challenges in this field are reflected in the comments in this blog's discussion. It's a complex question about how well science can currently measure which hospital is "best," let alone whether the U.S. News rankings reflect this.
Since the full text now seems to be available without a subscription, rather than trying to recapitulate the article here, I'll just say that for those who are interested in seeing a relatively brief summary of the high-quality research around whether the U.S. News rankings reflect non-reputational quality, it's in the "Overall Evaluations of Hospital Quality" section on p. 2356.
Full text:
http://jama.ama-assn.org/cgi/reprint/302/21/2353.pdf
Abstract:
http://jama.ama-assn.org/cgi/content/abstract/302/21/2353
- mh
This is the Lickers List: Those who have enough advantage to obtain more advantage and pummel anyone who interferes with their career, even if they are completely incompetent, too psychiatrically ill to practice, or responsible for multiple bad events. It take a village to manage the malignant media, political Boards of Registration and malpractice system. In the words of a top administrator for malpractice insurance covering physicians of Massachusetts: "It's a game!". More power equals more "game".
ReplyDeleteBad events are more often than not not reported, especially by other physicians who stand to benefit in all sorts of ways. Psychopathic physicians rise to the top.
These lists: "Best Doctor", "Top Doctor", "Best Hospitals" and "Four Star Medical School" should be disbanded immediately. They only serve to put patients and students at increased risk by feeding them into a situation that has no concern for real safety or accountability.
Better late than never…Editor Camarow of USNews knows a few of my suggestions for this “study”, but much like this thread he is dismissive even after acknowledging he has no expertise in statistics. Let’s start there…take a class and join the informed discussion.
ReplyDeleteI also don’t think it is not fair that RTI, Inc (those that did the study to your specifications) should take the blame. I expect that they agree with Dr. Sehgal et al.
I also don’t think Paul believes that “this is enough already” after all, he runs one of those organizations that you misrepresent in this “study”. This is serious business for us; it may be about selling magazines for you. Let’s not confuse the two missions.
To suggest that “There is no pattern of plummeting scores following unwarranted patient deaths at ranked hospitals.” Is to state that there is no causal relationship among the measures. So even with an expected nominal change in year over year reputation due to the use of three years of rolling data, apparently the survey respondents are daft to reality of outcomes, and the point of the thread is now evident from your own words.
After 20 years of trying harder, your objections to doing more are no worse than doing less. If you can’t stand the heat in the kitchen, get out! There is plenty of data available to determine whether this passes the sniff test, and it doesn’t. Read your own in box and perhaps USNews should assign an Ombudsman to this issue because you fail to comprehend the magnitude of bias, the misguided (although perhaps well intentioned) description of what this “study” communicates to readers.
The name is misleading, the methodology is harder to find than the “Marketing Opportunities” and even as a non layperson, this idea that you are trying to identify the hospitals to attend if you have the most serious and complex conditions, is not clear. When did stage 1 breast cancer and knee replacements become so routine that they don’t deserve the best? Who are you to decide that members of COTH are the first input requirement to the study group? What double blinded clinical trial evidence do you have that suggests that the other 4,000 US hospitals fail the inclusion test? The test is so narrow, to be dismissive of most US health care.
Since you may be too tired of reading, I’ll stop here. USNews; hold a focus group, invite some smart people like the ones who post similar notes to the USNews Web site and wake up to the reality that this isn’t about selling magazine for the rest of us, it is about transparency, consumer engagement and improving outcomes. You contribute nothing toward that end.
On one side we have the reputation of the hospitals as noted in the article and on the other hand their "e"putation or their reputation specifically on the web and electronic media. This eputation is becoming more and more important as people are doig more searches online. The troubling fact we each (hospitals, other businesses and even individuals) have more control over their reputation than their eputation. Once you develop an eputation it can go viral without any control.
ReplyDelete