Tuesday, October 17, 2006

A Statistically Valid Sample?

Every year, US News and World Report lists the top hospitals in the country. Everyone in the health care field knows that the methodology used in this survey has no statistical or scientific validity, but everyone wants to be at the top of the list!

Here's a story on efforts by the University of Pittsburgh Medical Center to try to score well in this annual survey. We wish them well, especially because their former Chief of Medicine is now Chief of Medicine here at BIDMC. So if they get highly ranked, based on his many years of fine work there, we will share in their pleasure.


Anonymous said...

If everyone in the health care industry knows that the US News & World Report hospital rankings are based on questionable methodology, why does everyone continue to participate? Why don't you do what some colleges and universities have done and refuse to fill out the forms to be listed?

Anonymous said...

Because everyone hopes to get to the top of the list . . . .

Patient Dave said...

This is a grumpy rant, and I don't know if you'll want to post it. No problem.

I'll second the sentiment about the questionable methodology of US News ratings, and I'm not talking about health care. The area where I know a little is college rankings. I think it's downright irresponsible (and unprofessional) for them to publish (and trumpet) ratings like that without full disclosure of the methods used and their limitations.

A great example is the college "competitiveness" rating, which evaluates (or used to evaluate) how many incoming "offer-ees" accept your offer, instead of turning you down. On the surface that makes sense: if 1000 kids are accepted by both School X and School Y, and 60% of them choose School X, it appears that X is more competitively desirable.

But it's axiomatic in business that "You get what you measure" - regardless of what you THOUGHT you were measuring. Colleges learned to "game the system" - the whole high-pressure early admission system reportedly arose out of a desire to manipulate this metric by getting 100% acceptance of their offers. (To get high rankings, you try to avoid giving out offers that aren't accepted. So you offer early admission that must be accepted before other schools send out their offers.)

Does that in fact measure school quality? No; it measures a strategy that manipulates the metric.

Another example is the reported percentage of who gets financial aid. Numerous colleges turned themselves into the equivalent of Mattress Discounters: they inflated their list prices and now give aid to everyone. Completely bogus and no indication of school desirability.

And that's the thing with ratings that are published without disclosure of how they can be "gamed."

This is not minor detail, especially in medical areas. Any student of statistics knows that things must be measured carefully, or you end up with false conclusions.

What are the chances US News would open up its methods for peer review, hm?

End of grumpy rant.

Anonymous said...

Doesn't seem grumpy to me . . .