Gary Schwitzer, writing in JAMA Internal Medicine -- in an article that has been moved out in front of the paywall --- freely accessible for 6 months -- summarizes the work of a
team of reviewers from HealthNewsReview.org who evaluated the reporting by US news organizations on new
medical treatments, tests, products, and procedures between 2006 and 2013. The results:
After reviewing 1889 stories (approximately 43% newspaper articles, 30% wire or news services stories, 15% online pieces [including those by broadcast and magazine companies], and 12% network television stories), the reviewers graded most stories unsatisfactory on 5 of 10 review criteria: costs, benefits, harms, quality of the evidence, and comparison of the new approach with alternatives. Drugs, medical devices, and other interventions were usually portrayed positively; potential harms were minimized, and costs were ignored.
Here were the standards applied during this seven-year period:
Ten review criteria were used to assess each news story reviewed. The review criteria addressed whether the story (1) adequately discussed the costs of the intervention; (2) adequately quantified the benefits of the intervention; (3) adequately quantified the harms of the intervention; (4) evaluated the quality of the evidence; (5) widened the diagnostic boundaries of illness and promoted public awareness of these widened boundaries, which may expand the market for treatments, a practice that has been termed disease mongering; (6) quoted independent sources and identified the conflicts of interest of sources; (7) compared the new approach with existing alternatives; (8) established the availability of the intervention; (9) established whether the approach was truly novel; and (10) appeared to rely solely or largely on a news release as the source of information.
The one type of mistake that I have seen all too often is described in the article as "the tyranny of the anecdote." Gary explains:
Stories may include positive patient anecdotes but omit trial dropouts, adherence problems, patient dissatisfaction, or treatment alternatives. In one example, 4 major US news sources used the same “breakthrough” patient anecdote of one man’s reported improvement from a heart failure stem cell experiment. This example illustrates another common problem: patient anecdotes provided by public relations representatives for medical centers or industry that put an overly positive spin on an intervention.
How perfectly these stories coincide with the anecdotes used by firms and hospitals in promoting their products or services. How much more so when the reporter works in a small or medium size community whose hospital has decided to buy into a firm's technology. It becomes quite evident that the journalist is often taken in by the jingoism of the hospital and "finds" the anecdotes that support the joint marketing campaign of the suppplier and the hospital. Gary suggests that "fawning coverage of new technologies" is the result:
Journalists often do not question the proliferation of expensive technologies, such as proton beam radiation machines and robotic surgical systems. The use of new technologies can increase the costs of medical care but may not improve care.
Gary suggests:
Our findings can help journalists improve their news stories and help physicians and the public better understand the strengths and weaknesses of news media coverage of medical and health topics.
I hope so. These reporters, often younger men and women on the staff of newspapers, need to understand that their reports can have a substantial impact on the public perception of new medical devices, therapies, and hospitals. They owe it to the public to be diligent and rigorous in their coverage. Are they up to the task?
After reviewing 1889 stories (approximately 43% newspaper articles, 30% wire or news services stories, 15% online pieces [including those by broadcast and magazine companies], and 12% network television stories), the reviewers graded most stories unsatisfactory on 5 of 10 review criteria: costs, benefits, harms, quality of the evidence, and comparison of the new approach with alternatives. Drugs, medical devices, and other interventions were usually portrayed positively; potential harms were minimized, and costs were ignored.
Here were the standards applied during this seven-year period:
Ten review criteria were used to assess each news story reviewed. The review criteria addressed whether the story (1) adequately discussed the costs of the intervention; (2) adequately quantified the benefits of the intervention; (3) adequately quantified the harms of the intervention; (4) evaluated the quality of the evidence; (5) widened the diagnostic boundaries of illness and promoted public awareness of these widened boundaries, which may expand the market for treatments, a practice that has been termed disease mongering; (6) quoted independent sources and identified the conflicts of interest of sources; (7) compared the new approach with existing alternatives; (8) established the availability of the intervention; (9) established whether the approach was truly novel; and (10) appeared to rely solely or largely on a news release as the source of information.
The one type of mistake that I have seen all too often is described in the article as "the tyranny of the anecdote." Gary explains:
Stories may include positive patient anecdotes but omit trial dropouts, adherence problems, patient dissatisfaction, or treatment alternatives. In one example, 4 major US news sources used the same “breakthrough” patient anecdote of one man’s reported improvement from a heart failure stem cell experiment. This example illustrates another common problem: patient anecdotes provided by public relations representatives for medical centers or industry that put an overly positive spin on an intervention.
How perfectly these stories coincide with the anecdotes used by firms and hospitals in promoting their products or services. How much more so when the reporter works in a small or medium size community whose hospital has decided to buy into a firm's technology. It becomes quite evident that the journalist is often taken in by the jingoism of the hospital and "finds" the anecdotes that support the joint marketing campaign of the suppplier and the hospital. Gary suggests that "fawning coverage of new technologies" is the result:
Journalists often do not question the proliferation of expensive technologies, such as proton beam radiation machines and robotic surgical systems. The use of new technologies can increase the costs of medical care but may not improve care.
Gary suggests:
Our findings can help journalists improve their news stories and help physicians and the public better understand the strengths and weaknesses of news media coverage of medical and health topics.
I hope so. These reporters, often younger men and women on the staff of newspapers, need to understand that their reports can have a substantial impact on the public perception of new medical devices, therapies, and hospitals. They owe it to the public to be diligent and rigorous in their coverage. Are they up to the task?
Unfortunately his findings also point up that we the patients must now apply the same 'let the buyer beware' standard to proposed medical care that we do to buying cars, insurance or other peddled products.
ReplyDeleteOn anecdotes, it is perhaps ironic that within the medical community, surgeons in particular are criticized by their colleagues for living by the anecdote, rather than the data quoted (ad nauseum) from journals by the internists. The 'power of story' is impressive in some contexts but not always what should be relied upon.
nonlocal MD
Paul, Great post. I am forwarding this post to my pals in the angel investment community; we need to be reminded of these findings. The fact that the info presented is filtered through a biased perspective shouldn't surprise any of us, but we don't always have the time to really vet facts presented and proliferated through sources we assumed to be reliable. Imagine that.
ReplyDeleteMedia report either bad news (politics, fires, accidents, tragedies overseas, etc) or good news (medical breakthroughs, new technology, new businesses, etc). Their failure to report trial drop-outs, and other things that would make an otherwise positive report more neutral, is part of a larger problem, not a specific problem in medically related coverage.
ReplyDeleteIn descending order of credibility and willingness/ability to present a full picture of a given story are: law journals, major science magazines (Science, Nature, etc), medical journals (with a big range themselves), long form magazines (New Yorker, New Review of Books, etc), newspapers (New York Times, then a big drop-off, all the way down to the Boston Herald and New York Post), TV cable shows (MSNBC is the top in my opinion and Fox is at the bottom), network TV, local TV, and finally the National Enquirer!
All of them of course report medical stories but only a few at the top of list give a balanced presentation. The rest neither can (because they are staffed with people who don't know enough about the subject matter) nor have any motivation to do so (because the role of the reportage is to offset the 'bad' news earlier in their reporting).
My own view on the media, Paul:
ReplyDeleteI believe what they say absolutely. I have always gotten good information from the media, with one exception: when I happen to know about the subject they are describing, I inevitably find important errors. But they are the only errors I detect, so I conclude that they are nearly always correct.
I also trust the NSA absolutely, As I would have trusted the STASI, the KBG, and the NKVD. After all, why not?
See story on cancer treatment front page NYT yesterday, which illustrates how to do it I think. Yes uses anecdote, but only to illustrate patient involvement, reaction, not science. Has the kind of qualifiers I like. Makes one a bit hopeful.
ReplyDelete