Cadmore media player playing audio Victor M. Montori, MD, MSc, discusses the Users’ Guide for misleading presentations of clinical trial results.
Audio Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the audio. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Transcript
Result Count
of
Search
Segments
Result Count
of
Search
Name:
Victor M. Montori, MD, MSc, discusses the Users’ Guide for misleading presentations of clinical trial results.
Description:
Victor M. Montori, MD, MSc, discusses the Users’ Guide for misleading presentations of clinical trial results.
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/f966e467-0dea-4ab4-b35c-9425981a8fd4/thumbnails/f966e467-0dea-4ab4-b35c-9425981a8fd4.jpg?sv=2019-02-02&sr=c&sig=cclBhpFwLZEjX8b%2B5DPSxDwzFeTHGeo6EYHFseLapmY%3D&st=2024-12-22T05%3A24%3A16Z&se=2024-12-22T09%3A29%3A16Z&sp=r
Duration:
T00H31M24S
Embed URL:
https://stream.cadmore.media/player/f966e467-0dea-4ab4-b35c-9425981a8fd4
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/f966e467-0dea-4ab4-b35c-9425981a8fd4/18958792.mp3?sv=2019-02-02&sr=c&sig=bB%2Frv%2FYK84SmZLo9SAE2cI9HTN6mjfdnZMp%2F4iJf0Ik%3D&st=2024-12-22T05%3A24%3A16Z&se=2024-12-22T07%3A29%3A16Z&sp=r
Upload Date:
2022-02-28T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
[ Music ] >> Hello. I'm Gordon Guyatt. I'm a Professor of Medicine and Clinical Epidemiology at McMaster University in Canada and the editor of the JAMA Users' Guides to the Medical Literature. Today I'm going to be talking about one of our users' guides related to misleading publications with Dr. Victor Montori who is a Professor of Medicine at the Mayo Clinic.
And Victor has played a very prominent, perhaps the major role in our users' guides to misleading presentations in publications and has also made enormous contributions to the Users' Guide to the Medical Literature. Welcome, Victor. >> Thanks, Gordon. It's a pleasure to be here with you today. >> Great. So thinking of misleading presentations of research, your first major publication regarding that appeared quite a while ago in the BMJ in 2004.
When we talk about misleading presentations of evidence, what are we actually talking about? >> Yes, you say it was 2004. I think it was about ten years old when we published that. And what we're talking about is that there are clinical trials that are publishing results and these results, when we take them in, we think of them as potentially being true. And if they're not true, then we have a biased result here or we have random error associated with the result.
It's an imprecise result. But then we start seeing studies that are reporting results and when we look at their usual methods to detect risk of bias and error, they're passing with flying colors. These are phenomenally strong, methodologically well-conducted trials and yet they are presenting the results in a way that they're giving us the impression that the-- for instance, that the intervention is better than it probably is in real life.
And this is achieved often as a result of choices that the investigators make in the way the study is designed, conducted, and reported. And we notice that people who became very good at looking at bias and error were missing these potential ways in which trial results were misleading them. And so we are talking about ways in which the methods and the results are described. We're talking about the way the results are analyzed, what kind of results are highlighted, where they're highlighted, and how things appear in front of users, and how those choices affect their sense of confidence about the intervention.
>> A colloquial word for everything you just said would be "spin". Is that right? >> Yes. So "spin" is an interesting word because the reality is difficult. And so then you present it with a lot of makeup on to try to make it more appealing. I learned of this -- I'm originally from Peru and I learned of this watching the Peruvian National Team get, you know, killed in soccer matches, you know, for the World Cup. And still somehow the coach will come on TV and say, "Ah, we played well and we had the great scheme.
Our players were awesome. We lost by nine goals but we were wonderful in the pitch." And so that clearly was an example of spin. And to see investigators do this for their clinical trials I think made it quite appealing to use that word "spin" to describe to this misleading presentation of trial results. >> Is it true that that would typically be making things look better than they really are? >> I think it is, unless you're trying to make a competitor look worse. One of the examples that we found was this issue of using inappropriate comparators.
So for instance, you may have a new drug, and this was done with antipsychotics. You would have a new drug and you would want to make the case that your new drug was superior to the old drug. And so you will see studies in which the investigators chose a particularly high dose of the old drug and as a result were able to give adverse effects to a large proportion of people in the control arm who were using a very large dose of the old antipsychotic to then make the claim that their new one is superior to the old one by making the old one look worse.
>> I remember that you once gave a talk related to this with a rather dramatic title since both you and I are advocates of evidence-based medicine. And it was entitled The End of Evidence-Based Medicine. Can you tell us what led you to choose that title and give that talk? >> I suspect that it was some inherent fatalism that comes from again being Peruvian. But you came up with evidence-based medicine in 1991.
And as I recall, this was about 2007 or so. And at that point there were two issues that were making us feel that evidence-based medicine, evidence-based practice was not likely to succeed. The first one had to do with the evidence itself. And so we were aware of the potential for bias in clinical research and at that time clinical trial registration had come to bear. We had standards for reporting of clinical trials.
We had a number of things in place to-- you know, further dissemination of the things that contributed to bias. So there was some idea that bias was becoming potentially either easier to detect or less of a problem. So that was something that was keeping us excited. Precision was another issue but trials were getting, particularly for important things, were getting bigger with more events. And so there was some suggestion that perhaps the problem of random error might get smaller. Systematic reviews were exploding.
A lot of people were doing meta-analysis, so we had large evidence. But then spin was still under recognized at the time and rampant. And so people were potentially being mislead into taking up interventions that were much less efficacious or much more harmful than they had thought in people who had a very low chance of benefit from those things. And then of course there was the possibility that continues to be thankfully a minority of situations, of fraud in the conduct and dissemination of results.
So there was a problem with the evidence. And we had linked that problem of the evidence fundamentally to the increasing participation of for-profit agents in the production of that evidence, where the evidence was really being used as an arm of the marketing department of for-profit, particularly pharmaceutical and device, companies. And this ensured that using the trials in this manner either through the way they were designed or disseminated that the results will be in favor of their product more often than not.
And the spin was being used for that purpose and also for the purpose of when they show that the results were less favorable than expected that they will use substantial spin to mislead people into thinking that the results were in fact favorable when they were not. And so we thought that this corruption of the evidence by shifting the reason you conducted research, which is to improve the clarity and understanding of what works and to what extent it works for which people, that mission was corrupted and now evidence was being used to improve the approval of marketing of drugs and devices.
And we thought that that would be kind of the kiss of death for the evidence in evidence-based medicine. And then the second part was the key principle of evidence-based medicine is that the evidence never tells you what to do. And you have to consider the patient's own priorities, goals, values, preference, and particular situation, and that requires the involvement of patients and co-creation with clinicians of treatment plans, something that sometimes is referred as shared decision making. And at the time we were convinced that shared decision making in practice was a unicorn, you know it really never happened or happened very seldom.
And so as a result, with the evidence being corrupted by for-profit agents using it for marketing purposes and the lack of patient involvement in decision making means that the principles of evidence-based medicine could not be and were not being practiced in the way that it was intended and that clearly indicated the end of evidence-based medicine. After a dramatic pause in the presentation, which felt like the sudden ending of one of these European movies where boy meets girl and they seem to go about, you know, they're about to kiss and then of course the movie fades to black and then the credits roll and you never know what happened.
I would then continue on to say that the reason we can tell we have a problem with evidence is because of the tools of evidence-based medicine and critical appraisal, the tools are summarized in the Usage Guides to the Medical Literature, and the fact that we can understand the nature of the problem means that we can probably begin to work hard towards a solution. Then I concluded by inviting the audience to work towards independent forms of evidence production and to work towards increasing the prevalence of patient-centered care.
>> You've already alluded to one way that the spin happens and that is in the way the people who design trials, plan them and implement them. And you've alluded to making the competitor look bad by, in the case of the antipsychotic example, effectively overdosing the control group with the competitor's drugs. Are there any other prevalent or dramatic or particularly important ways that the misleading spinning is done that you might want to highlight?
>> In our guides, the first guide that we offer, perhaps one of the most controversial, was to invite people who read the primary reports of clinical trials to bypass the discussion and conclusion sections, both of the abstract as well as of the full paper, in favor of carefully reviewing the methods and results. This is particularly useful advice for people who know how to read the methods and results, where again the users' guides are helpful.
But the invitation to bypass a discussion came from evidence that demonstrated that even after adjusting for the size of the benefit and the potential for harms, studies that are funded by for-profit organizations tend to discuss their results in the discussion section with incredible enthusiasm that seems again to be greater than justified by the size of the benefit or the size of the harm and the relevant benefits and harms of the competitors.
Phrases like "as a result of this study, the intervention should now be the treatment of choice for this condition" will be more likely to be found in studies funded by the makers of that particular intervention regardless of the size of the benefit or harm. So given that there is evidence for that and the discussion section is in part being used to reframe the results in ways that benefit the sponsor, readers probably are protected from that form of spin by directly assessing the methods for their validity and the results for their magnitude and precision and importance independently from what the authors will write.
A similar suggestion was also made instead of reading the abstract submitted by the authors to the journal for publication that they would look at abstracts prepared independently which is often the case when people are summarizing the results of studies for journal club type publications, the ACP Journal Club comes to mind, Evidence-Based Medicine Journal comes to mind, where independent editors review the paper and produce their own abstract summarizing the result. And that may be a better abstract to review than the abstract that came out with the journal itself.
So that has to do with the training or the presentation of the results in the discussion of the paper itself. And then there were other things that readers have to pay attention to and this has to do with the size of the effect. Studies that report massive benefits oftentimes are conducted in trials that have very few events, so these are very imprecise estimates. Because they're very large, they're likely to be statistically significant and gather a lot of attention.
But one looks carefully, there are very few events driving this; that makes them very susceptible to change in magnitude and even in significance and direction with just a few more events. And this sometimes can happen when trials are stopped earlier than planned because somebody looked at the interim result and found a large difference between the arms and decided to stop the trial at that point. And you and I have reported, along with other colleagues, that this kind of practice tends to produce estimates of effect that are quite larger than what eventually ends up being the estimate when the studies are taken to completion.
So when people report on large effects, there's a lot of publicity, there's a lot of hype associated with this. You almost don't need to spin them as much as long as people don't pay attention to how few events are contributing to that estimation. And then on the opposite end, and this is a much more common situation, the problem of trials that find very small treatment effects, and so they have to really-- the spinsters have to work very hard at making these very small treatment effects appear compelling.
And so there the methods usually involve drawing attention to the effect of intervention on surrogate endpoints, so endpoints that replace the patient important outcomes and usually the effect on surrogate endpoints tends to be big. They tend to draw attention to the point estimate. Even if the confidence interval around that point estimate kisses the line of no difference or approaches it very closely, the attention is drawn to the point estimate rather than to the lower end of that confidence interval or the attention is drawn to the significant P value away from the point estimate altogether.
And oftentimes these treatments that are capable only of very small treatment effects are evaluated in trials in which there's a mix of low-risk patients and high-risk patients and then the attention is drawn to the fact that there's all this large number of low-risk patients who benefit from this very small treatment effect. And so, therefore, the effect is very large when in fact the majority of the effect was concentrated in the high-risk population. So there are a number of concerns there. And I would think that of all the ones that we've discussed so far this perhaps is the most common and therefore the most dangerous because it draws attention, it draws effort, it draws resources, particularly scarce economic resources, to the funding of interventions that are capable of making only a very, very, very small difference in a very small number of people and yet tend to be touted as significant innovations worthy of large investments.
>> Going right back to that original 2004 paper, the most striking commentary that the paper received was a primary care doctor who said, "I know if the trial has more than 10,000 patients, I don't have to worry about it because the underlying effects when it was necessary to enroll 2000 patients to see them are too small for my patients to be interested in." And that was a kind of dramatic illustration of the point you've made around small effects.
>> I'm an endocrinologist, Gordon. So there's obviously the flipside of that argument which is that many, many trials are too small to have achieved any kind of prognostic balance for randomization and sometimes they stand alone for a long, long time. So there's no way of making them contribute to a bigger body of evidence because nobody reproduces those studies. So in my field of endocrinology, we have studies that produce statistically significant results, particularly on biochemical markers, that are just too small.
And so people have to, I think, gain an appreciation for large trials or at least large evidence to produce that prognostic balance that gives us confidence that the difference in prognosis at the end of the trial is due to the intervention and not due to imbalances at baseline. And so yes, I think large trials probably are capable of producing only minor differences between, you know, only minor effects and so most people shouldn't care about them. On the flipside of that, small trials should also be for the most part ignored until they contribute to a large enough body of evidence.
And so I think we need to have an appreciation for size because it does matter. >> In particular with regard to the issue that you just mentioned, you've referred a couple of times to what are called surrogate or substitute endpoints. So one way to get away with an apparent dramatic effect in a small sample size trial, which fails to focus on what we call patient-important endpoints, would be when they focus on surrogates. And I believe you've highlighted that particularly in your area of major interest, diabetes, where there is a focus on glucose or hemoglobin A1c instead of what matters to patients which are the macrovascular and microvascular complications of diabetes.
>> Yes. And that's been I think a very challenging finding for us because when you look at the large trials of glycemic control, their ability to detect any impact of years of effort to try to achieve glycemic control on outcomes that people can experience and value, you know, like death or major cardiovascular events, amputations, renal failure requiring dialysis or transplantation or blindness, those trials cannot or have not been able to detect important differences or even detectable differences in those outcomes with the majority of the evidence being favorable in relation to outcomes that are surrogates of those when they're available.
It's interesting because we have massive industry and massive clinical intervention and massive self-management effort on the part of patients to try to achieve this glycemic control and yet what is informing all those efforts is essentially a body of evidence that although it's very large, its major contribution is on its impact on the surrogate endpoints. And so this has led our group to wonder out loud whether for type two diabetes, the aim of achieving tight glycemic control, we're talking glycemic control beyond symptom control, whether the aim of achieving glycemic control and looking for more medications to lower blood sugars is an appropriate use of resources or whether we need to start thinking about medications and interventions that effect other aspects of the syndrome of type two diabetes.
And interestingly, in the last few years we now have interventions that have minimal effect on glucose values but have major impact on cardiovascular or for instance cardiovascular and kidney endpoints and they're not mediated by blood sugar but are potentially affecting the prognosis of patients with diabetes. So the story is still evolving. >> Yes, so a major theme for clinicians is to be skeptical when all that's been shown is effects in surrogates like blood glucose or bone density or in dementia scores on memory tests and focused more on patient-important outcomes like macro and microvascular complications in diabetes, like fractures and osteoporosis, like activities of daily living in people with cognitive impairment; so, great point.
There are some new areas that have emerged since your prior writings on misleading presentations of evidence. One is large observational studies based on administrative data that people have called, perhaps in a spin itself, real-world evidence. Any thoughts about that in the context of misleading evidence? >> We need to be clear that real-world evidence probably comes in like almost any other evidence seeking to draw inferences about cause-and-effect causality.
They come in two flavors, randomized and observational. And, you know, we are recording this in the midst of the COVID pandemic. And there's been incredible efforts in some places to look at the efficacy of interventions as clinicians have been caring for patients in intensive care units affected by severe COVID. And I would say that the best evidence about how to take care of these patients, specifically for COVID, has come out of the production of randomized trials in the course of taking care of patients, particularly in the UK.
So that kind of real-world evidence I think is quite interesting and I think produces large evidence and is less expensive than some of these commercial trials that we've been mentioning before. And they're usually conducted without much industrial influence. And as a result, I think they represent a major development that I would like to see more of. But the more common real-world evidence is observational. And it takes advantage of the exhaust of the clinical engine.
The medical records that are produced in the course of caring for people, the administrative datasets that are produced as a result of the billing and paying for those medical services, and the combination of these databases and their combination with biological databases, laboratory results, bio banks and so forth. And I think the main challenge with this real-world evidence is bias; both biases that are typical of observational studies, mostly the problem of confounders and the incredible challenge of accounting for them and accounting for the way people choose to take treatments versus allowing chance to determine that.
And the other problem is the bias of reporting and publishing, which I suspect very strongly that in the case of observational studies is absolutely massive, given that the range of analysis that can be produced in the computer, in the privacy of one's own now working from home office is infinite. And yet the number of analyses that get published in a nifty package with very parsimonious set of analysis and very discrete set of analysis when in fact it's quite likely that the investigator manipulated the data in multiple ways to produce that particular result.
And we don't have the results of all those ways in which the data was analyzed to begin with. So I suspect that bias and publication bias are the main concerns with real-world evidence. And once that real-world evidence has been produced for publication, I think we would see that the majority of the spin approach is that we've been discussing in this program will be applied to those results in the same way. The fact that we sometimes need to use very large evidence, observational evidence to uncover small relationships, would be I think stimulus enough for investigators to try to spin those small relationships as being very meaningful.
>> Victor, some of our audience might be saying, "My goodness, how can I possibly pick up all these problems myself? What can I really do in a practical way about all this?" And one answer might be, find yourself trustworthy guidelines produced by people who are alert to these issues and that can interpret the evidence and provide appropriate recommendations that actually deal with the issues where experts, both content and methodologists, are alert to these issues.
What do you think of that as a potential solution? >> Since the early days of teaching critical appraisal, just when we were focusing on issues of bias, I think the biggest risk that we were facing was the possibility that clinicians in practice and their patients would become nihilists, all evidence is corrupt, no evidence is trustworthy, why even review the evidence and let's just rely on other things to make decisions. And again, we're taping this in the midst of the COVID pandemic and we have seen how this approach where people have dismissed clinical research and have gone with their gut, so to speak, has led whole countries into investing in interventions to deal with COVID that have proven to be wasteful and potentially harmful.
As you point out, and as I mentioned before when we were discussing the end of evidence-based medicine talk, it is in the tools of this field that we find some solutions. One of those tools has been the production of guidelines based on the GRADE approach and the GRADE approach requires methodologists working with experts together and patients together to figure out how do we draw from the evidence to form recommendations. And many of the GRADE components address directly some of the contributors to this misleading presentation of results, elements that we have been discussing, including comparators, subgroups, large and small effects, and so forth.
So if one finds guidelines that have been produced using the GRADE approach with the methodologist as one of the authors, these are things that might indicate a trustworthy guideline and users may want to look at how the evidence is presented at summarizing those guidelines in lieu of directly reading the primary paper. And I think that might be a potential way of avoiding misleading presentations. And I think it's in line with other pre-appraised or digested forms of evidence presentation.
But those are hard because those may come in late. And many times, we find clinicians feeling pressured to pick up the results of trial the minute they are out. In fact, even before the results themselves have published in a peer reviewed journal, sometimes even responding to press releases. So the idea of taking your time and looking at things carefully and so forth in a world that has been put into overdrive seems a little counterintuitive. But I think a slower practice of medicine will always be a wiser practice of medicine.
I think here it applies as well. >> The COVID example has come up several times as highlighted that with the hydroxychloroquine example and just yesterday WHO put out a recommendation that ivermectin should be used only in the context of randomized trials, casting appropriate skepticism. So I think your message of not jumping early and waiting for things to be processed and considered carefully is a great message.
>> But, Gordon, think about if every cardiologist and every oncologist that is actively involved in their national or international meetings knows that the place to be in those meetings is at the late-breaking session. You know, this is the session where the studies that were conducted and completed after the deadline for submission of the regular paper is passed. So these are like right, you know, these are late breaking. This is, you know, you have to be there because it just came out.
Nobody knows about the results and the results are going to be announced in those presentations. And there will be a big to-do about them. So, you know, those rooms again, pre-pandemic, those rooms will be filled with people. They'll be the most popular sessions and big announcements are made. So it tells you that we have this incredible attraction, even in medical practice, for what's new, what's late breaking. And here, you and I are making the case that we need to let the evidence settle, mature, accumulate, become balanced, wait out all the publication bias, you know, which delays a publication of negative findings and so forth.
So we are a bit swimming against the current, don't you think? >> Yes, but I would add at least wait until somebody dissects for spin. At least wait until somebody dissects for spin. Even if you're feeling a little bit impatient, a trustworthy guideline following the approaches that you see just and would be at least something that might well be worth waiting for before you expose your patients to interventions that may in fact do more harm than good.
Victor, anything that we've missed that you think is really important for our audience to hear about misleading presentations? >> Look for bias. Look for imprecision. Be attentive to fraud but don't forget spin. And I think this is the main thing. If you want to draw evidence to form care with your patients, alert yourself and your patients to the possibility the spin could be misleading. >> Thank you so much.
I'm Gordon Guyatt, Editor of the JAMA Users' Guide Series. I've been speaking with Victor Montori, a Professor of Medicine at the Mayo Clinic who has been educating us about misleading presentations and what one can do to avoid being misled. You can find more JAMA podcasts at jamanetworkaudio.com. Many of those I'm sure you'll find of interest. [ Music ]
Segment:0 .
[ Music ] >> Hello. I'm Gordon Guyatt. I'm a Professor of Medicine and Clinical Epidemiology at McMaster University in Canada and the editor of the JAMA Users' Guides to the Medical Literature. Today I'm going to be talking about one of our users' guides related to misleading publications with Dr. Victor Montori who is a Professor of Medicine at the Mayo Clinic.
And Victor has played a very prominent, perhaps the major role in our users' guides to misleading presentations in publications and has also made enormous contributions to the Users' Guide to the Medical Literature. Welcome, Victor. >> Thanks, Gordon. It's a pleasure to be here with you today. >> Great. So thinking of misleading presentations of research, your first major publication regarding that appeared quite a while ago in the BMJ in 2004.
When we talk about misleading presentations of evidence, what are we actually talking about? >> Yes, you say it was 2004. I think it was about ten years old when we published that. And what we're talking about is that there are clinical trials that are publishing results and these results, when we take them in, we think of them as potentially being true. And if they're not true, then we have a biased result here or we have random error associated with the result.
It's an imprecise result. But then we start seeing studies that are reporting results and when we look at their usual methods to detect risk of bias and error, they're passing with flying colors. These are phenomenally strong, methodologically well-conducted trials and yet they are presenting the results in a way that they're giving us the impression that the-- for instance, that the intervention is better than it probably is in real life.
And this is achieved often as a result of choices that the investigators make in the way the study is designed, conducted, and reported. And we notice that people who became very good at looking at bias and error were missing these potential ways in which trial results were misleading them. And so we are talking about ways in which the methods and the results are described. We're talking about the way the results are analyzed, what kind of results are highlighted, where they're highlighted, and how things appear in front of users, and how those choices affect their sense of confidence about the intervention.
>> A colloquial word for everything you just said would be "spin". Is that right? >> Yes. So "spin" is an interesting word because the reality is difficult. And so then you present it with a lot of makeup on to try to make it more appealing. I learned of this -- I'm originally from Peru and I learned of this watching the Peruvian National Team get, you know, killed in soccer matches, you know, for the World Cup. And still somehow the coach will come on TV and say, "Ah, we played well and we had the great scheme.
Our players were awesome. We lost by nine goals but we were wonderful in the pitch." And so that clearly was an example of spin. And to see investigators do this for their clinical trials I think made it quite appealing to use that word "spin" to describe to this misleading presentation of trial results. >> Is it true that that would typically be making things look better than they really are? >> I think it is, unless you're trying to make a competitor look worse. One of the examples that we found was this issue of using inappropriate comparators.
So for instance, you may have a new drug, and this was done with antipsychotics. You would have a new drug and you would want to make the case that your new drug was superior to the old drug. And so you will see studies in which the investigators chose a particularly high dose of the old drug and as a result were able to give adverse effects to a large proportion of people in the control arm who were using a very large dose of the old antipsychotic to then make the claim that their new one is superior to the old one by making the old one look worse.
>> I remember that you once gave a talk related to this with a rather dramatic title since both you and I are advocates of evidence-based medicine. And it was entitled The End of Evidence-Based Medicine. Can you tell us what led you to choose that title and give that talk? >> I suspect that it was some inherent fatalism that comes from again being Peruvian. But you came up with evidence-based medicine in 1991.
And as I recall, this was about 2007 or so. And at that point there were two issues that were making us feel that evidence-based medicine, evidence-based practice was not likely to succeed. The first one had to do with the evidence itself. And so we were aware of the potential for bias in clinical research and at that time clinical trial registration had come to bear. We had standards for reporting of clinical trials.
We had a number of things in place to-- you know, further dissemination of the things that contributed to bias. So there was some idea that bias was becoming potentially either easier to detect or less of a problem. So that was something that was keeping us excited. Precision was another issue but trials were getting, particularly for important things, were getting bigger with more events. And so there was some suggestion that perhaps the problem of random error might get smaller. Systematic reviews were exploding.
A lot of people were doing meta-analysis, so we had large evidence. But then spin was still under recognized at the time and rampant. And so people were potentially being mislead into taking up interventions that were much less efficacious or much more harmful than they had thought in people who had a very low chance of benefit from those things. And then of course there was the possibility that continues to be thankfully a minority of situations, of fraud in the conduct and dissemination of results.
So there was a problem with the evidence. And we had linked that problem of the evidence fundamentally to the increasing participation of for-profit agents in the production of that evidence, where the evidence was really being used as an arm of the marketing department of for-profit, particularly pharmaceutical and device, companies. And this ensured that using the trials in this manner either through the way they were designed or disseminated that the results will be in favor of their product more often than not.
And the spin was being used for that purpose and also for the purpose of when they show that the results were less favorable than expected that they will use substantial spin to mislead people into thinking that the results were in fact favorable when they were not. And so we thought that this corruption of the evidence by shifting the reason you conducted research, which is to improve the clarity and understanding of what works and to what extent it works for which people, that mission was corrupted and now evidence was being used to improve the approval of marketing of drugs and devices.
And we thought that that would be kind of the kiss of death for the evidence in evidence-based medicine. And then the second part was the key principle of evidence-based medicine is that the evidence never tells you what to do. And you have to consider the patient's own priorities, goals, values, preference, and particular situation, and that requires the involvement of patients and co-creation with clinicians of treatment plans, something that sometimes is referred as shared decision making. And at the time we were convinced that shared decision making in practice was a unicorn, you know it really never happened or happened very seldom.
And so as a result, with the evidence being corrupted by for-profit agents using it for marketing purposes and the lack of patient involvement in decision making means that the principles of evidence-based medicine could not be and were not being practiced in the way that it was intended and that clearly indicated the end of evidence-based medicine. After a dramatic pause in the presentation, which felt like the sudden ending of one of these European movies where boy meets girl and they seem to go about, you know, they're about to kiss and then of course the movie fades to black and then the credits roll and you never know what happened.
I would then continue on to say that the reason we can tell we have a problem with evidence is because of the tools of evidence-based medicine and critical appraisal, the tools are summarized in the Usage Guides to the Medical Literature, and the fact that we can understand the nature of the problem means that we can probably begin to work hard towards a solution. Then I concluded by inviting the audience to work towards independent forms of evidence production and to work towards increasing the prevalence of patient-centered care.
>> You've already alluded to one way that the spin happens and that is in the way the people who design trials, plan them and implement them. And you've alluded to making the competitor look bad by, in the case of the antipsychotic example, effectively overdosing the control group with the competitor's drugs. Are there any other prevalent or dramatic or particularly important ways that the misleading spinning is done that you might want to highlight?
>> In our guides, the first guide that we offer, perhaps one of the most controversial, was to invite people who read the primary reports of clinical trials to bypass the discussion and conclusion sections, both of the abstract as well as of the full paper, in favor of carefully reviewing the methods and results. This is particularly useful advice for people who know how to read the methods and results, where again the users' guides are helpful.
But the invitation to bypass a discussion came from evidence that demonstrated that even after adjusting for the size of the benefit and the potential for harms, studies that are funded by for-profit organizations tend to discuss their results in the discussion section with incredible enthusiasm that seems again to be greater than justified by the size of the benefit or the size of the harm and the relevant benefits and harms of the competitors.
Phrases like "as a result of this study, the intervention should now be the treatment of choice for this condition" will be more likely to be found in studies funded by the makers of that particular intervention regardless of the size of the benefit or harm. So given that there is evidence for that and the discussion section is in part being used to reframe the results in ways that benefit the sponsor, readers probably are protected from that form of spin by directly assessing the methods for their validity and the results for their magnitude and precision and importance independently from what the authors will write.
A similar suggestion was also made instead of reading the abstract submitted by the authors to the journal for publication that they would look at abstracts prepared independently which is often the case when people are summarizing the results of studies for journal club type publications, the ACP Journal Club comes to mind, Evidence-Based Medicine Journal comes to mind, where independent editors review the paper and produce their own abstract summarizing the result. And that may be a better abstract to review than the abstract that came out with the journal itself.
So that has to do with the training or the presentation of the results in the discussion of the paper itself. And then there were other things that readers have to pay attention to and this has to do with the size of the effect. Studies that report massive benefits oftentimes are conducted in trials that have very few events, so these are very imprecise estimates. Because they're very large, they're likely to be statistically significant and gather a lot of attention.
But one looks carefully, there are very few events driving this; that makes them very susceptible to change in magnitude and even in significance and direction with just a few more events. And this sometimes can happen when trials are stopped earlier than planned because somebody looked at the interim result and found a large difference between the arms and decided to stop the trial at that point. And you and I have reported, along with other colleagues, that this kind of practice tends to produce estimates of effect that are quite larger than what eventually ends up being the estimate when the studies are taken to completion.
So when people report on large effects, there's a lot of publicity, there's a lot of hype associated with this. You almost don't need to spin them as much as long as people don't pay attention to how few events are contributing to that estimation. And then on the opposite end, and this is a much more common situation, the problem of trials that find very small treatment effects, and so they have to really-- the spinsters have to work very hard at making these very small treatment effects appear compelling.
And so there the methods usually involve drawing attention to the effect of intervention on surrogate endpoints, so endpoints that replace the patient important outcomes and usually the effect on surrogate endpoints tends to be big. They tend to draw attention to the point estimate. Even if the confidence interval around that point estimate kisses the line of no difference or approaches it very closely, the attention is drawn to the point estimate rather than to the lower end of that confidence interval or the attention is drawn to the significant P value away from the point estimate altogether.
And oftentimes these treatments that are capable only of very small treatment effects are evaluated in trials in which there's a mix of low-risk patients and high-risk patients and then the attention is drawn to the fact that there's all this large number of low-risk patients who benefit from this very small treatment effect. And so, therefore, the effect is very large when in fact the majority of the effect was concentrated in the high-risk population. So there are a number of concerns there. And I would think that of all the ones that we've discussed so far this perhaps is the most common and therefore the most dangerous because it draws attention, it draws effort, it draws resources, particularly scarce economic resources, to the funding of interventions that are capable of making only a very, very, very small difference in a very small number of people and yet tend to be touted as significant innovations worthy of large investments.
>> Going right back to that original 2004 paper, the most striking commentary that the paper received was a primary care doctor who said, "I know if the trial has more than 10,000 patients, I don't have to worry about it because the underlying effects when it was necessary to enroll 2000 patients to see them are too small for my patients to be interested in." And that was a kind of dramatic illustration of the point you've made around small effects.
>> I'm an endocrinologist, Gordon. So there's obviously the flipside of that argument which is that many, many trials are too small to have achieved any kind of prognostic balance for randomization and sometimes they stand alone for a long, long time. So there's no way of making them contribute to a bigger body of evidence because nobody reproduces those studies. So in my field of endocrinology, we have studies that produce statistically significant results, particularly on biochemical markers, that are just too small.
And so people have to, I think, gain an appreciation for large trials or at least large evidence to produce that prognostic balance that gives us confidence that the difference in prognosis at the end of the trial is due to the intervention and not due to imbalances at baseline. And so yes, I think large trials probably are capable of producing only minor differences between, you know, only minor effects and so most people shouldn't care about them. On the flipside of that, small trials should also be for the most part ignored until they contribute to a large enough body of evidence.
And so I think we need to have an appreciation for size because it does matter. >> In particular with regard to the issue that you just mentioned, you've referred a couple of times to what are called surrogate or substitute endpoints. So one way to get away with an apparent dramatic effect in a small sample size trial, which fails to focus on what we call patient-important endpoints, would be when they focus on surrogates. And I believe you've highlighted that particularly in your area of major interest, diabetes, where there is a focus on glucose or hemoglobin A1c instead of what matters to patients which are the macrovascular and microvascular complications of diabetes.
>> Yes. And that's been I think a very challenging finding for us because when you look at the large trials of glycemic control, their ability to detect any impact of years of effort to try to achieve glycemic control on outcomes that people can experience and value, you know, like death or major cardiovascular events, amputations, renal failure requiring dialysis or transplantation or blindness, those trials cannot or have not been able to detect important differences or even detectable differences in those outcomes with the majority of the evidence being favorable in relation to outcomes that are surrogates of those when they're available.
It's interesting because we have massive industry and massive clinical intervention and massive self-management effort on the part of patients to try to achieve this glycemic control and yet what is informing all those efforts is essentially a body of evidence that although it's very large, its major contribution is on its impact on the surrogate endpoints. And so this has led our group to wonder out loud whether for type two diabetes, the aim of achieving tight glycemic control, we're talking glycemic control beyond symptom control, whether the aim of achieving glycemic control and looking for more medications to lower blood sugars is an appropriate use of resources or whether we need to start thinking about medications and interventions that effect other aspects of the syndrome of type two diabetes.
And interestingly, in the last few years we now have interventions that have minimal effect on glucose values but have major impact on cardiovascular or for instance cardiovascular and kidney endpoints and they're not mediated by blood sugar but are potentially affecting the prognosis of patients with diabetes. So the story is still evolving. >> Yes, so a major theme for clinicians is to be skeptical when all that's been shown is effects in surrogates like blood glucose or bone density or in dementia scores on memory tests and focused more on patient-important outcomes like macro and microvascular complications in diabetes, like fractures and osteoporosis, like activities of daily living in people with cognitive impairment; so, great point.
There are some new areas that have emerged since your prior writings on misleading presentations of evidence. One is large observational studies based on administrative data that people have called, perhaps in a spin itself, real-world evidence. Any thoughts about that in the context of misleading evidence? >> We need to be clear that real-world evidence probably comes in like almost any other evidence seeking to draw inferences about cause-and-effect causality.
They come in two flavors, randomized and observational. And, you know, we are recording this in the midst of the COVID pandemic. And there's been incredible efforts in some places to look at the efficacy of interventions as clinicians have been caring for patients in intensive care units affected by severe COVID. And I would say that the best evidence about how to take care of these patients, specifically for COVID, has come out of the production of randomized trials in the course of taking care of patients, particularly in the UK.
So that kind of real-world evidence I think is quite interesting and I think produces large evidence and is less expensive than some of these commercial trials that we've been mentioning before. And they're usually conducted without much industrial influence. And as a result, I think they represent a major development that I would like to see more of. But the more common real-world evidence is observational. And it takes advantage of the exhaust of the clinical engine.
The medical records that are produced in the course of caring for people, the administrative datasets that are produced as a result of the billing and paying for those medical services, and the combination of these databases and their combination with biological databases, laboratory results, bio banks and so forth. And I think the main challenge with this real-world evidence is bias; both biases that are typical of observational studies, mostly the problem of confounders and the incredible challenge of accounting for them and accounting for the way people choose to take treatments versus allowing chance to determine that.
And the other problem is the bias of reporting and publishing, which I suspect very strongly that in the case of observational studies is absolutely massive, given that the range of analysis that can be produced in the computer, in the privacy of one's own now working from home office is infinite. And yet the number of analyses that get published in a nifty package with very parsimonious set of analysis and very discrete set of analysis when in fact it's quite likely that the investigator manipulated the data in multiple ways to produce that particular result.
And we don't have the results of all those ways in which the data was analyzed to begin with. So I suspect that bias and publication bias are the main concerns with real-world evidence. And once that real-world evidence has been produced for publication, I think we would see that the majority of the spin approach is that we've been discussing in this program will be applied to those results in the same way. The fact that we sometimes need to use very large evidence, observational evidence to uncover small relationships, would be I think stimulus enough for investigators to try to spin those small relationships as being very meaningful.
>> Victor, some of our audience might be saying, "My goodness, how can I possibly pick up all these problems myself? What can I really do in a practical way about all this?" And one answer might be, find yourself trustworthy guidelines produced by people who are alert to these issues and that can interpret the evidence and provide appropriate recommendations that actually deal with the issues where experts, both content and methodologists, are alert to these issues.
What do you think of that as a potential solution? >> Since the early days of teaching critical appraisal, just when we were focusing on issues of bias, I think the biggest risk that we were facing was the possibility that clinicians in practice and their patients would become nihilists, all evidence is corrupt, no evidence is trustworthy, why even review the evidence and let's just rely on other things to make decisions. And again, we're taping this in the midst of the COVID pandemic and we have seen how this approach where people have dismissed clinical research and have gone with their gut, so to speak, has led whole countries into investing in interventions to deal with COVID that have proven to be wasteful and potentially harmful.
As you point out, and as I mentioned before when we were discussing the end of evidence-based medicine talk, it is in the tools of this field that we find some solutions. One of those tools has been the production of guidelines based on the GRADE approach and the GRADE approach requires methodologists working with experts together and patients together to figure out how do we draw from the evidence to form recommendations. And many of the GRADE components address directly some of the contributors to this misleading presentation of results, elements that we have been discussing, including comparators, subgroups, large and small effects, and so forth.
So if one finds guidelines that have been produced using the GRADE approach with the methodologist as one of the authors, these are things that might indicate a trustworthy guideline and users may want to look at how the evidence is presented at summarizing those guidelines in lieu of directly reading the primary paper. And I think that might be a potential way of avoiding misleading presentations. And I think it's in line with other pre-appraised or digested forms of evidence presentation.
But those are hard because those may come in late. And many times, we find clinicians feeling pressured to pick up the results of trial the minute they are out. In fact, even before the results themselves have published in a peer reviewed journal, sometimes even responding to press releases. So the idea of taking your time and looking at things carefully and so forth in a world that has been put into overdrive seems a little counterintuitive. But I think a slower practice of medicine will always be a wiser practice of medicine.
I think here it applies as well. >> The COVID example has come up several times as highlighted that with the hydroxychloroquine example and just yesterday WHO put out a recommendation that ivermectin should be used only in the context of randomized trials, casting appropriate skepticism. So I think your message of not jumping early and waiting for things to be processed and considered carefully is a great message.
>> But, Gordon, think about if every cardiologist and every oncologist that is actively involved in their national or international meetings knows that the place to be in those meetings is at the late-breaking session. You know, this is the session where the studies that were conducted and completed after the deadline for submission of the regular paper is passed. So these are like right, you know, these are late breaking. This is, you know, you have to be there because it just came out.
Nobody knows about the results and the results are going to be announced in those presentations. And there will be a big to-do about them. So, you know, those rooms again, pre-pandemic, those rooms will be filled with people. They'll be the most popular sessions and big announcements are made. So it tells you that we have this incredible attraction, even in medical practice, for what's new, what's late breaking. And here, you and I are making the case that we need to let the evidence settle, mature, accumulate, become balanced, wait out all the publication bias, you know, which delays a publication of negative findings and so forth.
So we are a bit swimming against the current, don't you think? >> Yes, but I would add at least wait until somebody dissects for spin. At least wait until somebody dissects for spin. Even if you're feeling a little bit impatient, a trustworthy guideline following the approaches that you see just and would be at least something that might well be worth waiting for before you expose your patients to interventions that may in fact do more harm than good.
Victor, anything that we've missed that you think is really important for our audience to hear about misleading presentations? >> Look for bias. Look for imprecision. Be attentive to fraud but don't forget spin. And I think this is the main thing. If you want to draw evidence to form care with your patients, alert yourself and your patients to the possibility the spin could be misleading. >> Thank you so much.
I'm Gordon Guyatt, Editor of the JAMA Users' Guide Series. I've been speaking with Victor Montori, a Professor of Medicine at the Mayo Clinic who has been educating us about misleading presentations and what one can do to avoid being misled. You can find more JAMA podcasts at jamanetworkaudio.com. Many of those I'm sure you'll find of interest. [ Music ]
Form Title Download
Media
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html audio player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the audio player are generally tabs, one of which shows a scrolling transcript of the audio and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the audio player is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the audio by opening up an explore dialog which contains an about tab which contains rich metadata about the media, a segments tab which contains further information about the audio, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
E: Explore Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Form Title Embed Media
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the media.
You can press A to open the Accessibility Dialog.
This describes many other keyboard shortcuts for media controls.
Exit Clip Mode
The video is currently set-up to show just a clip from 0:00 to video end. You have requested to see a portion of the media outside of the clip.
Click OK to switch to the full media or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode