The knives, er scalpels, are out in force, attacking the methodology used by ProPublica in setting forth conclusions about America's surgeons. Here's a piece from Justin McLachlan, essentially accusing ProPublica of journalistic malpractice. Here's another one by Benjamin Davies, calling the work "clickbait."
Let's see, absent access to the secret NSQIP data, which admittedly does not exist for of all surgeons, what are we left with?
What are we left with if I am a referring doctor who wants to send a patient to the best possible surgeon?
What are we left with if I as a patient want to check out the performance record of several surgeons?
What statistically valid methodology is available to me?
I'll tell you what methods are currently in use:
Anecdote. Bias. Personal friendships between doctors. An unsupported feeling that "Dr. Smith does a really good job."
Justin and Benjamin and others, what exactly are you proposing should supplant the entirely subjective and unreliable sources of information currently available? How soon will your idea (if you have one) become feasible and be put into the public domain? Or do you simply propose that we should have less information about picking a surgeon than we do about virtually any other consumer choice we make?
Well, maybe it doesn't matter. Maybe all surgeons are equally competent and have similar records of success. Maybe the personal stakes to me as a patient are too insignificant to matter.
Right.
Let's see, absent access to the secret NSQIP data, which admittedly does not exist for of all surgeons, what are we left with?
What are we left with if I am a referring doctor who wants to send a patient to the best possible surgeon?
What are we left with if I as a patient want to check out the performance record of several surgeons?
What statistically valid methodology is available to me?
I'll tell you what methods are currently in use:
Anecdote. Bias. Personal friendships between doctors. An unsupported feeling that "Dr. Smith does a really good job."
Justin and Benjamin and others, what exactly are you proposing should supplant the entirely subjective and unreliable sources of information currently available? How soon will your idea (if you have one) become feasible and be put into the public domain? Or do you simply propose that we should have less information about picking a surgeon than we do about virtually any other consumer choice we make?
Well, maybe it doesn't matter. Maybe all surgeons are equally competent and have similar records of success. Maybe the personal stakes to me as a patient are too insignificant to matter.
Right.
36 comments:
You can't really judge the effectiveness of a surgeon without looking at the entire team. For example, in a total joint operation, if the team that normally works together has members who are swapped out in the middle, or there happens to be a number of staff who walk in and out (not optimal, obviously, when implanting a joint), this can affect time under anesthesia or introduce contamination. Likewise, the nursing team that accepts the patient for post-op care can fail to infuse antibiotics within the specified time ranges, neglect to administer anticoagulants, and so on. It's truly a team effect, and to suggest that surgeons are solely responsible for any and all outcomes attributes an incredible amount of power to them they truly do not possess. (PS I'm a nurse)
Think about what you are saying:
There is no way to quantify the ability of the surgeon to perform a surgery well or without causing harm to the patient. If that's the case, how on earth do referring doctors decide where to send their patients? How do you expect patients to make reasonable choices?
You are suggesting that this is such a craft industry that every case is simply a one-off. If so, this industry will find itself in deeper and deeper trouble, as the public and therefore the government will demand more accountability and interfere in ways that the professional will find highly troubling.
In surgery, the captain of the ship is the surgeon, and he or she will be the one that will be held accountable. The smart surgeons will make sure that their teams and the hospital support services are in place to help get the best possible results.
As anesthesiologists, we have an unusual and unique perspective on how surgeons behave toward others, use their judgment, observe their technical skills, and see the level of concern they have for the patient’s welfare. This kind of networking is valuable in good quality hospitals, though I couldn’t suggest how to quantitate the information. Obviously, its reliability is dependent on the quality of the anesthesiologist who is doing the observing so it is not free of bias, and it is not a systematic way to make referrals. I have, when asked, referred many patients to good surgeons based on these considerations.
Yes, you and I know that, and we use that network.
It doesn't solve the general problem, as you suggest, but it usually allows us insiders to get good service!
Mr. Levy,
Couldn't the insiders collect the information and create a source to share it with the public?
No,that's not practical.
Sorry but if they can't be bothered to share data like that, then we get to rank them in whatever way possible. We need more data and if they can't trust us with it, why should we trust them? It works both ways. Way too much secrecy in the profession.
With that being said they still are the very best source. As long as some people benefit from it, something at this point is better than nothing.
One other interesting data point that I would like to know about all doctors is how many doctors and nurses they have as patients and is that number considered average, high or low for their specialty. This could presumably give patients some insight into how the insiders choose doctors for their own care.
Those insiders who are privy to this information do share it, but with family and friends, which is great for them. That's what I meant by thank God somebody benefits from it. Meanwhile the rest of us can continue using the old fashioned way " word of mouth". Which is not a reliable source to find the best of the best surgeons, who we all want and why not say we deserve. You know why they don't share and it's a secret because if we all had the same info. nobody would use the not so good or inexperienced surgeons. Oh I get it, it would burn out the great ones and only discourage all the others- all for the "greater good " comes to mind again... Yep, that's it!!!
I think the reality is that newly minted and less experienced doctors are likely to build most of their early experience base treating lower income people and the uninsured.
If you look at ratings, if there were credible ratings that most people accepted, they would probably look something like bond ratings in the financial industry which range from AAA down to C. Anything between AAA and BBB would be considered the equivalent of investment grade which probably means the doctor’s skills are good enough for most people under most circumstances. Ratings of BB or lower might be the equivalent of junk bonds which most people would prefer to avoid if they have a choice.
I’m still troubled by potential problems with any risk adjustment mechanism though because it may not fully capture the outcome risks associated with operating on an elderly patient with multiple co-morbidities. If a surgeon’s score could be hurt by operating on these high risk patients, no matter how great his skills are, it would create an incentive to avoid those patients if possible.
As a practicing vascular surgeon for over 20 years I welcome any effort to help the poor naive patient in their efforts to obtain the best care available. From my experience the medical staff is very reluctant to deal with an MD who clearly is providing an inferior level of care, based on community standards. Hospital administration has openly refused to get involved with what they consider a medical staff issue, even when presented with glaring episodes of MD incompetence. The trustees, of course, are 2 steps removed and largely ignorant of MD issues (perhaps a legal remedy exists, as you have suggested). So what to do with an MD who provides a substandard level of care. As I see it, three possibilities:
1. Self discipline: This would require an MD to track his results, compare to accepted standards, and adjust accordingly either with further education or even further training, perhaps with a mentor already on staff. At least for surgeons, we tend to be a bit of an egotistical bunch and I don't see self discipline as realistic for most.
2. More formal monitoring: As part of my training we had weekly morbidity and mortality conferences. Attendence was mandatory. Here the resident or attending would discuss a patient and what went wrong. There could be an issue of omission (no antibiotics given and post op wound infection), comission (common bile duct injury during cholecystectomy), or shit happens (post op arrhythmia). Discussion was very open in an effort to prevent future complications. Self flagellation was expected. Maybe reinstituting formal morbidity/mortality conferences would lead to improvement and perhaps improve self discipline.
3. Public shaming: In the era of social media and now ProPublica this is perhaps the current reality. It ain't perfect, but I think it is a start. It then becomes incumbent on the poorly rated MD to deal with the negative press. He can ignore the data, but eventually this will affect his practice. He can whine that his cases are harder, patients are sicker, whatever. But, then he must do all this research to justify his whining. Not likely to happen. Or, he can see this as an opportunity for introspection and improvement. With public presentation of data, hospital administration and pethaps the trustees now have a real obligation to act.
Patients are increasingly going to look for an objective way of delineating the competency of their MD. For me it is easy. My wife needed major surgery. I know all the general surgeons in town and have a pretty good sense of who to see. I spoke with an anesthesiologist friend who sees the work every surgeon does on a regular basis. One question for him, "If it was your wife, who would you choose to do her surgery?" I know everyone will never have this kind of inside info, but I also think what ProPublica presents is at least a step in that direction.
wdsufer....I personally love and appreciate the bluntness!!! I am that naive patient so I need and want to know something about everything. I can't express enough how I wish it works out for all of us. Thank God the world has all of you guys, what would we do, if not?
Are you suggesting that bad data is better than no data? This is the inverse of the Monty Hall problem: here the host is equivocal as to whether there is a goat behind Door #3.
Anon 1:45
I think the point here is that bad data in this sort of situation inevitably provides the incentive to produce good data in future iterations. While there is no data, which has been the case for the last decades, it is easy to defend the status quo, isn't it.
Then there is always the obvious point that crying 'bad data' is about the same as crying 'our patients are sicker.'
Barry Carol,
Question, is it standards practice for lower income and uninsured patients to be the test subjects for new surgeons starting out? If that's a fact that pretty much speaks volumes. And that's exactly what's wrong. I hope nobody poor or uninsured reads this.
I've never heard of such a thing.
I do not believe that Mr. Carol has any particular education in medical or surgical care although he did make the assumption that newly minted and inexperienced doctors would somehow practice on the low income and uninsured patients and in another post assumed that board certification and longer experience was some assurance of excellence. Happily both assumptions are not correct and am surprised that Paul let those slip by. The post by Wbsurfer is the most credible. As a surgeon of some experience he knows what the realities of the problems are. While board certification is a measure of having some advanced knowledge, neither boards or long experience can promose that a surgeon will have what is called "good hands." Personal egotism too often allows poor performers to deny their lack of skill---easier to claim that they treated sicker patients or had tougher cases. Ego and arrogance can also allow some to make some claims as experts in circumstances where they are not and the board of trustees who function far from the facts of what goes on in the hospital and the administration who has a great investment in believing that his hospital is a great place allows the surgeons to find it near impossible to lay blame or criticism on one of their own.The wish to believe that we work in good hospitals, among ethical and highly skilled surgeons and giving the very best care is so strong as to make us less than keen sighted in defense of the wish. Nothing has a chance at changing until there is real transparency and the code of silence is no more.
Elegantly stated.
I love you, Mr. Retired, and you know why, thanks.. " newly minted" and " inexperienced".
Just "Retired" Mr. or Ms. (Sorry, don't know why I assumed Mr. I don't usually do that) As always your thoughts are so perfectly written.
A number of years ago, I came across a study that examined the outcomes for 54 heart surgeons in the Boston region. Presumably based on outcomes with some risk adjustment, each surgeon’s results received one of three ratings – outcomes as expected, outcomes better than expected and outcomes worse than expected. Of the 54 surgeons, 52 were rated outcomes as expected, one was rated better than expected and one worse than expected. To me, that suggests that with one exception, patients would have been in good hands with any of them.
For the significant number of surgical procedures that must be performed under emergency conditions, patients aren’t likely to have much choice in which surgeon operates on them. If I’m having a stroke, I just hope I can get to a hospital that can handle strokes and be successfully treated within 90 minutes of arrival.
For surgeons who practice mainly in safety net hospitals or in the VA system, most of their patients are likely to be low income and that will affect outcomes no matter how skillful the individual surgeon is. By contrast, a surgeon who practices in a wealthy zip code and operates mostly on upper middle class and wealthy people with few or no co-morbidities, outcomes are likely to be better than the results in the safety net and VA hospitals.
Finally, my sense is that the quality of training in the U.S. suggests that most surgeons are quite good at what they do at least once they have a reasonable number of procedures under their belt. As in much of the rest of medicine, though, there are probably a small number who aren’t very good at all and account for a disproportionate share of malpractice awards. The medical profession needs to do a better job of weeding them out.
Oh my goodness Barry Carol, you just answered a question I was trying to figure out how to ask. So- and be as honest as you guys can be. Because trust me now I'm driven to figure out the truth, so you better believe I have questions and I don't care how dumb they sound. Are inexperienced surgeons intentionally either by choice or not, placed in cities where there are more lower income and uninsured patients to practice their skills? And if so, why? Once practice makes perfect they then move to the bigger and wealthier cities. Im confused and want to know why there are affected outcomes based on lower income and VA patients regardless of surgeons skill. Excuse my French ( and I am, Cajun French) What the hell does that mean?
Retired from medicine, in fact. I would still like to address the difficulty of following Mr. Barry's assumptions regarding questionable studies, the quality of institutions and surgeons based on zip code and the idea that a U.S. trained surgeon with experience must be "quite good." For example, my sister who is a brilliant professional woman with an Ivy League education was a terrible driver 50 years ago. After fifty years of experience, she is still a terrible driver. Practice does not always make perfect., and studies are only as good as the information and method of research used. A high risk patient might be the fellow who lives on the street, dining from a. Dumpster and abusing drugs or a highly compensated exec who is stressed out, drinking too much and eating too many business lunches. Both will arrive at the ER at BWH in Boston in a bad neighborhood through the same door. Some comparisons with business fit in medicine, but most do not when it comes to performance. Health care and especially surgery occurs behind somewhat closed doors and observed by only a few. We must trust in the integrity of practitioners and in our ability to judge excellence with not nearly enough information. It is certainly not malicious on the part of most surgeons, but no one wants to think that they are working in a substandard hospital or doing a poor job, so poor outcomes may be explained away in ways which do not shed a bad light on them. Institutional and collegial loyalty along with some defensiveness conspires to allow even the best of practitioners to judge in a less than impartial manner. The assumption that malpractice finding will limit the problem surgeons is also erroneous. The threshold for malpractice findings against surgeons is no protection for patients despite all the furor over it. Being thought of as a good guy by peers and as likable by patients can keep a poor surgeon busy sadly enough. As I stated before, until we have measurable parameters and transparent reporting of all and without bias, we will continue to be in the dark. This is the place for science, not baseless assumptions.
A number of the issues that I tried to comment on are discussed by an M.D. in a blog post linked to below. If it were easy to come up with credible quantitative measurements of risk-adjusted quality, we presumably would have done so a long time ago. Even Medicare’s individual risk scores which are used by CMS to determine how much to pay insurers to cover a given Medicare Advantage enrollee are still widely considered to overpay for the healthy and underpay for the sick. At any rate, here’s the blog post referred to above:
http://thehealthcareblog.com/blog/2015/07/17/after-transparency-morbidity-hunter-md-joins-cherry-picker-md/#comments
Check out this new column by Ashish Jha. Excerpt:
Disruptive innovation, a phrase coined by Clay Christensen, is usually a new product that, to experts, looks inadequate. Because it is. These innovations are not, initially, as good as what the experts use (in this case, their network of surgeons). They initially dismiss the disrupter as being of poor quality. But disruptive innovation takes hold because, for a large chunk of consumers (i.e. patients looking for surgeons), the innovation is both affordable and better than the alternative. And once it takes hold, it starts to get better. And as it does, its unintended consequences will become dwarfed by its intended consequences: making the system better. That’s what ProPublica has produced. And that’s worth celebrating.
https://blogs.sph.harvard.edu/ashish-jha/the-propublica-report-card-a-step-in-the-right-direction/
Well that totally explains why you wrote what you did, understood now. After reading that, and by the way "thanks" Really. I'm still letting it all sink in, And I'm so lost for words and don't know why. As a layperson and that naive patient that's spoke of, It feels like I just read a classified hospital document. Way past curious what my favorites and others will write. I'm going to sit back and enjoy!!!
Wait a minute, my last comment at 5:38 was to Barry Carol, I read the link he suggested, and replied back now I just read Mr. Levy's post and link, read it and I'm confused again! I now am going to rip those 2 dog gone links apart to make sense of all this. Something's not right!
It's always good form to cite back to the link to which you are referring, Carole. I have to print things in the order they arrive: Blogspot offers no flexibility in that regard.
I'll most certainly do better with that from now on Mr. Levy. Meanwhile I'm back at square one. So I'm going to re-read everything. Just when I think I have it all figured out and a clear understanding of all this information, in a split second another persons point of view and comment helps me realize I don't. You all are the teachers I'm the student, I want to learn.
Thanks for the link to Ashish Jha’s post, Paul. I never understood why there’s so much resistance among medical professionals toward initiatives that could help patients make more informed choices. The mentality seems to be that if the new initiative or method isn’t perfect, it’s no good even if, for patients, it’s a heck of a lot better than what they had previously which was very little.
No disrespect Mr. Carol, but what the (bleep) just happened?????? You made some excellent points, stand by them !! If your changing your entire opinion please be so kind as to explain it to me at least, thank you, with respect.
Carole – I stand by the limitations I raised, especially the potential for the inadequacy of the risk adjustment mechanism to create incentives for surgeons to avoid operating on the highest risk patients. At the same time, the Pro Publica effort puts patients in a better position to evaluate surgeons than they were before.
This is not an academic exercise for me. I’ve had six surgical procedures in the last 15 years and will probably have another one coming up in the near future. In the past, I’ve relied on my referring doctor to send me to a capable surgeon and so far that has worked satisfactorily for me. It would be nice, though, to have some objective data that would confirm his recommendation or raise questions about it. Maybe something like an Angie’s List for doctors would be a good idea.
I’ve also learned that it’s not just the surgeon that’s important but the experience of the whole team in the operating room. For example, I’ve read that for heart transplants, a hospital should do at least 10 per year to sustain decent skills. In the Philadelphia region, fewer than 100 procedures are done each year and about two-thirds of those are done at one hospital – Penn. The others all do fewer than 10 per year. I think that’s valuable information to know.
Finally, for a procedure that I’m likely to need soon, my referring doctor won’t send me to his hospital because he doesn’t think they do enough of that procedure each week, month and year to ensure the best outcomes. I’ll bet plenty of doctors, especially if they work for HMO’s and are expected to keep as many patients as possible within the system, wouldn’t send me to an outside hospital that does that particular procedure in much higher volume.
The bottom line is that transparency is a good thing even if there are some limitations to the data.
Nothing wrong with the idea of having surgeon ratings. However, the 'baseball card' concept needs to be critically examined. You can turn the card over and read the statistics for various procedures, but the primary data going into those numbers is very complex and should be handicapped to account for the myriad of factors, including demographics, co-morbidities etc. Surgeons who routinely take secondary or difficult cases can come out looking weaker, when in fact they might be better than those doing only easier primary cases. The data should be reviewed by specialty-specific experts so that the numbers coming out are at least somewhat useful. But this is a start.
Anectdotal information despite its limitations, is often quite good too. Don't discount the words of the O.R. managers, ICU staff, nurses on the surgical units, chief residents, etc...
Mr. Carol
Thank you sir for explaining that to me, I appreciate and respect those who stand buy their beliefs and opinions, but are still open minded to others. And now that you have shared your story and the medical position your in, you have even more of a reason to stand your ground and make important and valid points that people like myself will listen to, and God willing all the others. You are the very person this applies to right here right now, because of that reason alone I'd be interested in all your Imput. The sad reality is we have no choice right now but to except that something is better than nothing.until things change and hopefully they work out the kinks. I care and feel strongly about fairness for us all and transparency a is moral responsibility, and nothing else will do. God bless thank you again, and the link you suggested was super awesome, just like Mr. Levy's was as well.
Okay I finally read everything over again trying my best to understand this from a layperson and or a naive patients perspective. First and foremost for Surgeons not completely convinced the way the data would be collected because of human error-would be accurate, seems to me it could help some undeserving inexperienced Surgeons and hurt experienced others. For patients the idea is a step forward in the right direction. However we do have that huge issue with the data needing to be kept secret from us, and for that reason I'm wondering if something were to go wrong, would a lawyer and the courts be able to also hold the refering doctor (who referenced the data) accountable for an incident (which I do not agree with) "the source" those who collected the data- if data was proven incorrect.(I do agree with) as well as the surgical team, (questionable? And only if true) I did think something was better than nothing for the patients sake, but now I have to be honest and consider what's best for the Surgeons sake to, and most of all for the refering Doctor. "Everything good or BAD from the get go that could happen is based on the collection of that data". It's only a good idea and a start if all Surgeons sign off on it and the data was made public so patients solely make the decisions (to protect Doctors who did not create the data or sign off on it) with the understanding the information is close not an exact science and not 100 percent set in stone accurate, but it will be. I Just want Surgeons to be happy and satisfied with the data and their reputation and careers in tact, Doctors protected for refering a Surgeon based on the data, and patients given the very best care possible- because of the data. All of you guys are so smart which I'm super impressed by and offer out sincere points of view which I respect. Thanks for not biting my head off when I mis-spelled a few words only realizing after I sent it out and when I obviously didn't quite understand something. That's of course if you are even reading this !!!
Post a Comment