Cadmore media player playing audio Bayesian Analysis: Using Prior Information to Interpret the Results of Clinical Trials: Audio Interview With Melanie Quintana, PhD
Audio Player
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the audio. Pressing escape will close all displays and pressing it again will jump to the end of the player.
You can press A to open the Keyboard Shortcut Dialog which also provides an audio description of the player and all of its features.
Transcript
Result Count
of
Search
Segments
Result Count
of
Search
Name:
Bayesian Analysis: Using Prior Information to Interpret the Results of Clinical Trials: Audio Interview With Melanie Quintana, PhD
Description:
Bayesian Analysis: Using Prior Information to Interpret the Results of Clinical Trials: Audio Interview With Melanie Quintana, PhD
Thumbnail URL:
https://cadmoremediastorage.blob.core.windows.net/7b183284-d611-45c4-8d4b-31ec50c6757e/thumbnails/7b183284-d611-45c4-8d4b-31ec50c6757e.jpg?sv=2019-02-02&sr=c&sig=3tjQEz60zy4ogedQc5TCwS7yYeFaO85kf1OagUcD%2F1Y%3D&st=2025-01-15T10%3A10%3A11Z&se=2025-01-15T14%3A15%3A11Z&sp=r
Duration:
T00H27M31S
Embed URL:
https://stream.cadmore.media/player/7b183284-d611-45c4-8d4b-31ec50c6757e
Content URL:
https://cadmoreoriginalmedia.blob.core.windows.net/7b183284-d611-45c4-8d4b-31ec50c6757e/19010025.mp3?sv=2019-02-02&sr=c&sig=Q5kZ4M%2BNnW77mjSXxc0jQCSaOJa7Zpt5u8eppwjJ9Kc%3D&st=2025-01-15T10%3A10%3A11Z&se=2025-01-15T12%3A15%3A11Z&sp=r
Upload Date:
2022-02-28T00:00:00.0000000
Transcript:
Language: EN.
Segment:0 .
[ Music ] >> Hello, and welcome to JAMAEvidence, our monthly podcast based on core issues in evidence-based medicine. I'm Roger Lewis. I am a JAMA Statistical Editor, a Co-Editor of the JAMA Guide to Statistics and Methods series, and a professor of emergency medicine at Harbor-UCLA Medical Center and the David Geffen School of Medicine at UCLA. Today, we're discussing Bayesian analysis using prior information to interpret the results of clinical trials.
Joining me to talk about this topic is Dr. Melanie Quintana, and I'd like to start by asking Dr. Quintana to tell us a little bit about herself. >> Thanks, Roger! Yeah, my name is Melanie Quintana, and I am statistical scientist at Berry Consultants. At Berry Consultants we design a wide range of clinical trials. Oftentimes, these trials are innovative, adaptive, and use Bayesian statistics. I am a Bayesian statistician at heart. I did my PhD at Duke University, where I became more familiar with Bayesian statistics and kind of really grew to love it.
>> Great. Thank you. So most of our readers are really much more familiar with the frequentist approach to statistics, often using p-values or frequentist confidence intervals to evaluate treatment effects that are seen in clinical trials. I'd like to start by asking you how would you describe the differences in the structure or philosophy between Bayesian and the more common frequentist approaches to drawing conclusions from data? >> Sure. So both methods aim to really make some conclusions about some quantity of interest based on data that's been observed.
And let's just say, for sake of argument, that quantity of interest is a response rate under some sort of treatment. At its purest sense, frequentists really believe that these quantities are fixed and unknown. So the treatment response rate is, say, 33%. Bayesians believe that these quantities are random, and because of that, we assign probabilities to them. The treatment response rate is most likely 30%, but it could be anywhere between 20 or 50%.
All that being said, I really think probably the differences are best seen in practice outside of, you know, that explanation. So frequentists use repeated sampling to understand that the likelihood that their hypothesis about that quantity of interest is true. As you said, they use p-values. So they answer questions such as given a thousand similar samples, what's the likelihood that we would observe a value more extreme than what we've observed if the null hypothesis was true that the treatment didn't work.
So in this paradigm, one is really able to test specific null hypotheses and reject the null hypothesis or not. We're never able to officially accept the null hypothesis or really understand the degree to which each hypothesis is true. Because Bayesians work in this probabilistic framework and they assign probabilities to quantities, they're able to directly answer questions such as what is the probability that this treatment is effective given the data we've observed.
The Bayesian paradigm, we're really able to understand how the accumulating evidence results in updates to our understanding of quantities of interest. So we're able to understand the degree to which hypotheses are true, not just accept or reject a hypothesis. >> Okay, so I've sometimes heard it said that the difference between the frequentist and the Bayesian paradigm is that the frequentists are interested in the probability of the data given some truth about a treatment effect, while Bayesians are interested in the probability of a treatment effect existing given the data.
Do you think that's a useful way of thinking about it, or what do you think about that approach? >> Yeah, I think that's a perfect way to think about it. It's, again, the probability of the data given this fixed truth of some quantity of interest, and the Bayesians then believe that the probability of that actual quantity given the data that we've observed. So I think those two sides of it are the perfect way to explain it. >> So we often see that clinicians when reading reports of clinical trials try to determine at the end what the probability is that the treatment works, how likely is it that the treatment works, and try to use the p-value to figure that out.
Now we've often been taught that you can't do that, and that a p-value less than 5% doesn't mean there's a 95% chance that the treatment works. But what advice would you give to clinicians who really want to know what is the chance the treatment is effective for their patients? >> I would give the advice to look into Bayesian statistics. Any frequentist analysis that's been done, we can actually replicate and provide a similar analysis within the Bayesian framework. So most people worry and think about, oh, but I have to specify some sort of prior distribution to do a Bayesian analysis.
Well, you know, if we specify our prior as very noninformative, the response rate could be anywhere between 0 or 100% and any value in between, you're going to get a result very similar to the frequentist. However, we can then, since we're in this probabilistic framework, we can then report summaries that are intuitive and that the clinician would really want to know. What's the probability that the treatment is effective? What is the probability that the response rate is greater than 50%? What's the probability that the treatment is noninferior to a control?
So the beautiful part about Bayesian statistics is that just given this what we call a posterior distribution, which is the update to the prior distribution after you observe data, we can really give any intuitive summary that you would want to know. >> Great. So that is a perfect segue to talking a little bit more about the pieces of this probabilistic framework that you describe. So let's go back to the beginning of that probabilistic framework to the concept of prior information.
So what is prior information, and how does it form the foundation or the cornerstone of Bayesian inference? >> Right. So prior information at its simplest explanation is just any information that you have before you've seen some new set of data. That's your prior information. And it's really, you know, can be some distribution or it can be as simple as saying, you know, I think like -- just like I said earlier, equally likely that there's no effect or 100% effect or anywhere in between.
Or you could say something like based on past observations, I believe the response rate is probably between 30 and 80%, likely somewhere around 50%. So anything that you really think about the quantity that you're interested in estimating before you have seen new data is your prior information. >> So once you have new data, you mentioned that you can update the prior to yield a posterior distribution. So what is the posterior distribution?
>> Yeah, it's exactly what you just said. So we use something called Bayes' theorem to take the prior information, look at the data, and then update our beliefs on the distribution of that quantity of interest based on that data that's observed. So it's just an update once we have observed some data. And the really interesting thing is now this posterior distribution could become our new prior distribution for a new set of data. So it just can work sort of in a circle like that, and we can continue updating the posterior distribution to get the posterior distribution every time we see new data.
And really, as I kind of mentioned earlier, this whole -- this posterior distribution really assigns a probability to any potential value of that quantity of interest. So there's some probability that the response rate under the treatment effect is 30%. There's some probability it's 31%. And so we can use this distribution to really summarize the results in any intuitive way that we might like to. >> So to make sure I understand that correctly, what you're saying is that the posterior distribution allows us to determine the probability of any treatment effect, not just a particular one, given both our prior information and the new data that we've observed.
>> Yes, that's correct. So any treatment effect, we could ask what's the probability that the treatment has a response rate greater than 50%? What's the probability that the treatment has a response rate less than 30%? If we have multiple treatments, we could summarize what's the probability that Treatment A is the best treatment out of, say, 12 similar other treatments. Or we can say what's the probability that the treatment is noninferior to some other treatments. It's really quite flexible once we have this posterior distribution and when we're working in this probabilistic framework.
>> Today's posterior distribution using the information we may have acquired in a trial can actually be the prior distribution for further investigation of that treatment in a new trial. That sounds a lot like the philosophy of a learning health care system. Are you aware of situations in which this approach has been used to create a learning health care system? >> Yeah, absolutely. So we're doing this in COVID. We do this when we're trying to use the health care system itself to continuously update our beliefs on what a best treatment is.
So this is done quite frequently and often uses Bayesian statistics to update our beliefs on what treatment is best over time and more frequently. >> Thanks. Now coming back to the question of the prior information, you made the point that it could be virtually any information that influences our belief in the effectiveness of a therapy before we acquire the new information in a clinical trial. But people are going to differ regarding what information should be incorporated into the prior.
Some people will find additional trials to be relevant. Some will think that the setting or the patient population was too distinct to incorporate them into the prior. So what do you do when different people would have different prior beliefs in the interpretation of new data? >> Yeah, so let me start by saying that when we're conducting an experiment and when we're performing, say, a clinical trial, and let's think of this as a -- in a confirmatory setting where we're trying to establish if a treatment works or not.
We absolutely need to prespecify our prior distribution that we will be using for the primary analysis. So I'm not speaking and saying, you know, anyone can come up with their prior distribution and anyone can get different results out of this. We should prespecify what our prior distribution is that will be used in the primary analysis. This prior could be, as we talked about, noninformative or informed by previous experiences but should be justified. We should give proper justification to why we propose to use that prior in our primary analysis.
That being said, we can specify, and we do prespecify sensitivity analyses around that prior distribution to show how the results from the experiment could change for those with different prior beliefs. So someone might come along with very different prior beliefs. There you have -- they are a skeptic and they don't believe this treatment works based on their experiences. We can show what the results would have been for that type of individual. This is really no different than sensitivity analyses that should be prespecified around key modeling assumptions in the frequentist framework.
So we do this all the time in the frequentist framework, say, for performing a linear aggression. We do goodness-of-fit tests, and we also do sensitivity analyses around the assumption that the relationship really is linear. >> So there have been some examples recently in which trials that were conducted under a traditional frequentist framework but yielded inconclusive results were then reanalyzed using a Bayesian framework. But, of course, since those Bayesian analyses were not prespecified, the prior information needed to be defined after the results of that trial were known.
What do you see as the challenges or the potential pitfalls in that approach? >> Yeah, I see a challenge there could be that because they were unblinded to the original results, their prior information could have some way been informed by those results itself, and then you would worry that there was some double use of the data and that the results from that new analysis could be biased in some way. If we said sort of this very noninformative prior distribution and show how the results might be different with that noninformative prior distribution, I think that would be a totally reasonable way to go, because we know that that noninformative distribution itself likely was not informed by any results that the individual was unblinded to.
>> So you've emphasized the importance of prespecification of the prior information and avoiding the double-counted information where we use information we've seen to reconsider our prior and then use those same data to update the prior in sort of a circular, logical fallacy. So the issue of prespecification is important when clinicians try to use Bayesian approaches or they read Bayesian re-analyses of trials that have been published previously.
What can be done to interpret a trial result after the fact when a Bayesian approach wasn't predefined? >> So the prior distribution can also be defined as the distribution that you've set independent from the data. So not necessarily setting it before you've seen the data, but if it's been set independent of the data, we could call this our prior distribution as well. But we need to be very careful to document that this prior distribution has been set independent of the data that you may have already seen.
And oftentimes, it probably would work best. The easiest way to do this is to just set a very noninformative prior distribution so you can say this is independent of the data that we've actually already observed, perform your Bayesian analysis, and get a posterior update. >> So when we're giving guidance to our readership in interpreting a Bayesian analysis, what do you think our readers who are generally not statisticians, or sometimes even fans of mathematics, should look for in the description of the Bayesian analysis when it's used to interpret the results of a clinical trial?
>> I believe you should look for prespecification of the prior distribution, just as you would want prespecification of all modeling assumptions. I think you should look at it and review the summaries and understand that these mean what you think that they should mean. They mean what you hope the p-value should mean. This truly is the updated information based on the data that was observed, based on also the prior information that was given beforehand.
>> So you mentioned at the beginning that some of your own work is focused on clinical trials for rare diseases. And rare diseases are a particularly challenging area for traditional frequentist trial design. So can you help explain how the Bayesian paradigm or approach is helpful in evaluating therapies for rare diseases? >> Sure. So I think the Bayesian approach in particular really shines when we need to be as efficient as possible and use all of the resources that we have in front of us.
So take all of the information that we have, synthesize it, and give us an updated belief about the quantity that we're interested in understanding. And this is particularly true in rare diseases, unmet medical needs, or situations where we need to be as efficient as possible to get an answer as quickly as possible, such as pandemics and COVID. The Bayesian framework is so well suited to take all of this information and combine it together, whether that information means, you know, take some sort of information outside of the trial and some prior information from previous results, build that into an informative prior, and use that in your new trial.
Or if that means taking different sources of results and doing some sort of dynamic borrowing of those different sources when bring them in together. Or this could even mean using information more efficiently within the trial itself. So it could mean doing things such as what we describe in our publication, which is you could essentially take all time points. Say you're interested in estimating a treatment effect across different time periods.
You could assume that the treatment effect at, say, six hours is similar to that at seven hours and eight hours and nine hours. I'm not saying it's the same as, it can potentially be different, but the Bayesian analysis framework allows us to understand those differences, model those differences, but still be informed by those other sources. Also, for instance, in platform trials, the ones that we have seen recently in COVID, we tend to try to use all randomized controls.
So platform trials are trials where we're perpetually enrolling and trying to understand if treatments work or not. And oftentimes, we use all of the controls within this platform trial to compare all controls to a new active treatment. This often includes controls that may be from different treatment domains. They may have different modes of administration. They may be nonconcurrently randomized controls or concurrently randomized controls.
But we use all of the controls that have been enrolled in the platform trial to compare them to the active treatment to gain as much information and use all of the information as efficiently as possible to answer the question. And again, I'm not saying we believe that all of the controls are the same. They vary much, and we believe that they may differ over time or differ over mode of administration, but we can model those differences but still allow those individuals to inform the estimate of the active treatment effect.
>> So you've mentioned that these approaches have been used in COVID. Can you describe how they've helped in the rapid investigation of multiple therapies for COVID and what challenges might have existed had we not had these approaches available to us. >> Sure. So I think we described a little bit about how we can use Bayesian models in these complex settings to really get the most out of all of the information that we have. Maybe that's borrowing the information from other sources or using all the information that we have with -- inside the platform trial.
But what we haven't talked about much is adaptations. So very frequently, Bayesian statistics is used to build adaptive clinical trials. And these are trials that frequently take prespecified interim analyses and perform prespecified adaptations based on rules that we've put in place before the trial has started so that we can efficiently determine if treatments work or not. So at these interims, we may do things such as stop the treatment early for success if there's overwhelming evidence that the treatment works or stop early for futility and don't waste the resources on that treatment if there's evidence that it does not work.
And these are some of the -- these adaptations were put in place in these COVID trials and in many trials that we work on. And Bayesian statistics is really a great toolbox to use for these types of trials because it gives us, as we've talked about, the intuitive summaries that you would want or need to make these decisions. So if I asked the clinician what information would you like to know to be able to stop a trial early for success or futility, you would likely tell me, tell me the probability that the treatment has a response rate better than 30%.
And these are the exact summaries that we have in the Bayesian setting so that we can use them to make these decisions. >> So can't a frequentist approach do the same thing? >> The issue with the frequentist approach is that you have to take into consideration how the data was sampled. And so once you begin making adaptations, you have to take into consideration the potential for those adaptations when you calculate your p-value. And again, the p-value, although it provides evidence if we can accept or reject a hypothesis, it does not necessarily provide the intuitive summaries that are often needed to make informed decisions on if we should stop a trial early for success or futility.
>> So you've certainly described a variety of advantages or situations in which the Bayesian approach may be more facile or more effective than the frequentist approach, but let's talk for a second about the regulatory environment. So specifically, are Bayesian analyses accepted by the FDA to support approvals of drugs, devices, or biologics, and what has been the history of the regulatory review of trials analyzed using a Bayesian approach?
>> Yes, absolutely. We have many situations where we have confirmatory studies that have a Bayesian primary analyses, and these studies have been discussed before the start of the trial with regulatory agencies in the FDA. Oftentimes, these studies are in rare disease, pediatrics, unmet medical needs, platform trials, or places where we want to speed up drug development as quickly as possible. And oftentimes, these situations are when we have some additional complexity where the Bayesian framework can really shine and where we can model that complexity, such as borrowing information outside of the trial or using information more efficiently within the trial.
So yes, I will say that the FDA and regulatory agencies do accept Bayesian analyses. However, there is still a mountain to climb when we're trying to propose this. We are oftentimes held to a high standard to be able to show what those Bayesian analyses will produce and be able to provide evidence that they would not lead to biased results. And so oftentimes, we save these Bayesian analyses for the most complex situations in which they're really needed.
Maybe for better or for worse, but oftentimes if we're doing a very simple trial where a frequentist result will be very similar to that of a Bayesian, let's say, just a simple t-test on the change from baseline, then we don't tend to fight that uphill battle to propose a Bayesian analysis or propose an analysis that individuals are less comfortable with, although, you know, maybe we should to make these more commonplace. >> So I think it's been clear to those of us at JAMA that the number of trials that are now being designed with a primary Bayesian approach is increasing over time.
What do you think the primary motivation for that is, and do you expect that to continue? >> Yeah, I think that although frequentist approaches were the standard, I think that clinicians especially are starting to understand how interpretable Bayesian results can be and are. I also think that the world is getting increasingly complex. And so I think that the Bayesian framework is naturally suited to handle a lot of these complexities and handle situations where we want to answer questions as efficiently as possible in, say, an adaptive trial.
And so we're really starting to see that there are situations where yes, we can, you know, manipulate the frequentist framework to get it to perform how we would want it to perform, but the Bayesian framework is really more naturally suited to do these things. >> So I think at the beginning you described yourself as a Bayesian at heart, so I'm going to ask you a very difficult question. What is a situation in which a Bayesian approach should not be used?
>> This is a hard question. So I would personally say that not necessarily a Bayesian approach shouldn't be used, but a frequentist approach may be preferred if we think that the result from that approach is going to be very similar to the Bayesian result and that there is a very large hurdle to overcome with the audience to get them to accept the Bayesian approach.
So if it's an extremely important study, if the Bayesian result is going to be very similar to the frequentist result, and the audience would be very, very skeptical of the Bayesian result, I may be willing to say that the frequentist approach is the right way to go there. >> [Music] Again, this is Roger Lewis, Senior Statistical Editor at JAMA and the Co-Editor of the JAMA Guide to Statistics and Methods series, and I'd like to thank our guest today, Dr. Melanie Quintana, for talking to us about the Bayesian paradigm for drawing information in clinical trials.
You can find further information about this and other topics on our website at jamaevidence.com or in a recent book from the JAMA Guide to Statistics and Methods. For more audio, please visit us at jamanetworkaudio.com, and you can listen and follow wherever you get your podcasts. Thanks again.
Segment:0 .
[ Music ] >> Hello, and welcome to JAMAEvidence, our monthly podcast based on core issues in evidence-based medicine. I'm Roger Lewis. I am a JAMA Statistical Editor, a Co-Editor of the JAMA Guide to Statistics and Methods series, and a professor of emergency medicine at Harbor-UCLA Medical Center and the David Geffen School of Medicine at UCLA. Today, we're discussing Bayesian analysis using prior information to interpret the results of clinical trials.
Joining me to talk about this topic is Dr. Melanie Quintana, and I'd like to start by asking Dr. Quintana to tell us a little bit about herself. >> Thanks, Roger! Yeah, my name is Melanie Quintana, and I am statistical scientist at Berry Consultants. At Berry Consultants we design a wide range of clinical trials. Oftentimes, these trials are innovative, adaptive, and use Bayesian statistics. I am a Bayesian statistician at heart. I did my PhD at Duke University, where I became more familiar with Bayesian statistics and kind of really grew to love it.
>> Great. Thank you. So most of our readers are really much more familiar with the frequentist approach to statistics, often using p-values or frequentist confidence intervals to evaluate treatment effects that are seen in clinical trials. I'd like to start by asking you how would you describe the differences in the structure or philosophy between Bayesian and the more common frequentist approaches to drawing conclusions from data? >> Sure. So both methods aim to really make some conclusions about some quantity of interest based on data that's been observed.
And let's just say, for sake of argument, that quantity of interest is a response rate under some sort of treatment. At its purest sense, frequentists really believe that these quantities are fixed and unknown. So the treatment response rate is, say, 33%. Bayesians believe that these quantities are random, and because of that, we assign probabilities to them. The treatment response rate is most likely 30%, but it could be anywhere between 20 or 50%.
All that being said, I really think probably the differences are best seen in practice outside of, you know, that explanation. So frequentists use repeated sampling to understand that the likelihood that their hypothesis about that quantity of interest is true. As you said, they use p-values. So they answer questions such as given a thousand similar samples, what's the likelihood that we would observe a value more extreme than what we've observed if the null hypothesis was true that the treatment didn't work.
So in this paradigm, one is really able to test specific null hypotheses and reject the null hypothesis or not. We're never able to officially accept the null hypothesis or really understand the degree to which each hypothesis is true. Because Bayesians work in this probabilistic framework and they assign probabilities to quantities, they're able to directly answer questions such as what is the probability that this treatment is effective given the data we've observed.
The Bayesian paradigm, we're really able to understand how the accumulating evidence results in updates to our understanding of quantities of interest. So we're able to understand the degree to which hypotheses are true, not just accept or reject a hypothesis. >> Okay, so I've sometimes heard it said that the difference between the frequentist and the Bayesian paradigm is that the frequentists are interested in the probability of the data given some truth about a treatment effect, while Bayesians are interested in the probability of a treatment effect existing given the data.
Do you think that's a useful way of thinking about it, or what do you think about that approach? >> Yeah, I think that's a perfect way to think about it. It's, again, the probability of the data given this fixed truth of some quantity of interest, and the Bayesians then believe that the probability of that actual quantity given the data that we've observed. So I think those two sides of it are the perfect way to explain it. >> So we often see that clinicians when reading reports of clinical trials try to determine at the end what the probability is that the treatment works, how likely is it that the treatment works, and try to use the p-value to figure that out.
Now we've often been taught that you can't do that, and that a p-value less than 5% doesn't mean there's a 95% chance that the treatment works. But what advice would you give to clinicians who really want to know what is the chance the treatment is effective for their patients? >> I would give the advice to look into Bayesian statistics. Any frequentist analysis that's been done, we can actually replicate and provide a similar analysis within the Bayesian framework. So most people worry and think about, oh, but I have to specify some sort of prior distribution to do a Bayesian analysis.
Well, you know, if we specify our prior as very noninformative, the response rate could be anywhere between 0 or 100% and any value in between, you're going to get a result very similar to the frequentist. However, we can then, since we're in this probabilistic framework, we can then report summaries that are intuitive and that the clinician would really want to know. What's the probability that the treatment is effective? What is the probability that the response rate is greater than 50%? What's the probability that the treatment is noninferior to a control?
So the beautiful part about Bayesian statistics is that just given this what we call a posterior distribution, which is the update to the prior distribution after you observe data, we can really give any intuitive summary that you would want to know. >> Great. So that is a perfect segue to talking a little bit more about the pieces of this probabilistic framework that you describe. So let's go back to the beginning of that probabilistic framework to the concept of prior information.
So what is prior information, and how does it form the foundation or the cornerstone of Bayesian inference? >> Right. So prior information at its simplest explanation is just any information that you have before you've seen some new set of data. That's your prior information. And it's really, you know, can be some distribution or it can be as simple as saying, you know, I think like -- just like I said earlier, equally likely that there's no effect or 100% effect or anywhere in between.
Or you could say something like based on past observations, I believe the response rate is probably between 30 and 80%, likely somewhere around 50%. So anything that you really think about the quantity that you're interested in estimating before you have seen new data is your prior information. >> So once you have new data, you mentioned that you can update the prior to yield a posterior distribution. So what is the posterior distribution?
>> Yeah, it's exactly what you just said. So we use something called Bayes' theorem to take the prior information, look at the data, and then update our beliefs on the distribution of that quantity of interest based on that data that's observed. So it's just an update once we have observed some data. And the really interesting thing is now this posterior distribution could become our new prior distribution for a new set of data. So it just can work sort of in a circle like that, and we can continue updating the posterior distribution to get the posterior distribution every time we see new data.
And really, as I kind of mentioned earlier, this whole -- this posterior distribution really assigns a probability to any potential value of that quantity of interest. So there's some probability that the response rate under the treatment effect is 30%. There's some probability it's 31%. And so we can use this distribution to really summarize the results in any intuitive way that we might like to. >> So to make sure I understand that correctly, what you're saying is that the posterior distribution allows us to determine the probability of any treatment effect, not just a particular one, given both our prior information and the new data that we've observed.
>> Yes, that's correct. So any treatment effect, we could ask what's the probability that the treatment has a response rate greater than 50%? What's the probability that the treatment has a response rate less than 30%? If we have multiple treatments, we could summarize what's the probability that Treatment A is the best treatment out of, say, 12 similar other treatments. Or we can say what's the probability that the treatment is noninferior to some other treatments. It's really quite flexible once we have this posterior distribution and when we're working in this probabilistic framework.
>> Today's posterior distribution using the information we may have acquired in a trial can actually be the prior distribution for further investigation of that treatment in a new trial. That sounds a lot like the philosophy of a learning health care system. Are you aware of situations in which this approach has been used to create a learning health care system? >> Yeah, absolutely. So we're doing this in COVID. We do this when we're trying to use the health care system itself to continuously update our beliefs on what a best treatment is.
So this is done quite frequently and often uses Bayesian statistics to update our beliefs on what treatment is best over time and more frequently. >> Thanks. Now coming back to the question of the prior information, you made the point that it could be virtually any information that influences our belief in the effectiveness of a therapy before we acquire the new information in a clinical trial. But people are going to differ regarding what information should be incorporated into the prior.
Some people will find additional trials to be relevant. Some will think that the setting or the patient population was too distinct to incorporate them into the prior. So what do you do when different people would have different prior beliefs in the interpretation of new data? >> Yeah, so let me start by saying that when we're conducting an experiment and when we're performing, say, a clinical trial, and let's think of this as a -- in a confirmatory setting where we're trying to establish if a treatment works or not.
We absolutely need to prespecify our prior distribution that we will be using for the primary analysis. So I'm not speaking and saying, you know, anyone can come up with their prior distribution and anyone can get different results out of this. We should prespecify what our prior distribution is that will be used in the primary analysis. This prior could be, as we talked about, noninformative or informed by previous experiences but should be justified. We should give proper justification to why we propose to use that prior in our primary analysis.
That being said, we can specify, and we do prespecify sensitivity analyses around that prior distribution to show how the results from the experiment could change for those with different prior beliefs. So someone might come along with very different prior beliefs. There you have -- they are a skeptic and they don't believe this treatment works based on their experiences. We can show what the results would have been for that type of individual. This is really no different than sensitivity analyses that should be prespecified around key modeling assumptions in the frequentist framework.
So we do this all the time in the frequentist framework, say, for performing a linear aggression. We do goodness-of-fit tests, and we also do sensitivity analyses around the assumption that the relationship really is linear. >> So there have been some examples recently in which trials that were conducted under a traditional frequentist framework but yielded inconclusive results were then reanalyzed using a Bayesian framework. But, of course, since those Bayesian analyses were not prespecified, the prior information needed to be defined after the results of that trial were known.
What do you see as the challenges or the potential pitfalls in that approach? >> Yeah, I see a challenge there could be that because they were unblinded to the original results, their prior information could have some way been informed by those results itself, and then you would worry that there was some double use of the data and that the results from that new analysis could be biased in some way. If we said sort of this very noninformative prior distribution and show how the results might be different with that noninformative prior distribution, I think that would be a totally reasonable way to go, because we know that that noninformative distribution itself likely was not informed by any results that the individual was unblinded to.
>> So you've emphasized the importance of prespecification of the prior information and avoiding the double-counted information where we use information we've seen to reconsider our prior and then use those same data to update the prior in sort of a circular, logical fallacy. So the issue of prespecification is important when clinicians try to use Bayesian approaches or they read Bayesian re-analyses of trials that have been published previously.
What can be done to interpret a trial result after the fact when a Bayesian approach wasn't predefined? >> So the prior distribution can also be defined as the distribution that you've set independent from the data. So not necessarily setting it before you've seen the data, but if it's been set independent of the data, we could call this our prior distribution as well. But we need to be very careful to document that this prior distribution has been set independent of the data that you may have already seen.
And oftentimes, it probably would work best. The easiest way to do this is to just set a very noninformative prior distribution so you can say this is independent of the data that we've actually already observed, perform your Bayesian analysis, and get a posterior update. >> So when we're giving guidance to our readership in interpreting a Bayesian analysis, what do you think our readers who are generally not statisticians, or sometimes even fans of mathematics, should look for in the description of the Bayesian analysis when it's used to interpret the results of a clinical trial?
>> I believe you should look for prespecification of the prior distribution, just as you would want prespecification of all modeling assumptions. I think you should look at it and review the summaries and understand that these mean what you think that they should mean. They mean what you hope the p-value should mean. This truly is the updated information based on the data that was observed, based on also the prior information that was given beforehand.
>> So you mentioned at the beginning that some of your own work is focused on clinical trials for rare diseases. And rare diseases are a particularly challenging area for traditional frequentist trial design. So can you help explain how the Bayesian paradigm or approach is helpful in evaluating therapies for rare diseases? >> Sure. So I think the Bayesian approach in particular really shines when we need to be as efficient as possible and use all of the resources that we have in front of us.
So take all of the information that we have, synthesize it, and give us an updated belief about the quantity that we're interested in understanding. And this is particularly true in rare diseases, unmet medical needs, or situations where we need to be as efficient as possible to get an answer as quickly as possible, such as pandemics and COVID. The Bayesian framework is so well suited to take all of this information and combine it together, whether that information means, you know, take some sort of information outside of the trial and some prior information from previous results, build that into an informative prior, and use that in your new trial.
Or if that means taking different sources of results and doing some sort of dynamic borrowing of those different sources when bring them in together. Or this could even mean using information more efficiently within the trial itself. So it could mean doing things such as what we describe in our publication, which is you could essentially take all time points. Say you're interested in estimating a treatment effect across different time periods.
You could assume that the treatment effect at, say, six hours is similar to that at seven hours and eight hours and nine hours. I'm not saying it's the same as, it can potentially be different, but the Bayesian analysis framework allows us to understand those differences, model those differences, but still be informed by those other sources. Also, for instance, in platform trials, the ones that we have seen recently in COVID, we tend to try to use all randomized controls.
So platform trials are trials where we're perpetually enrolling and trying to understand if treatments work or not. And oftentimes, we use all of the controls within this platform trial to compare all controls to a new active treatment. This often includes controls that may be from different treatment domains. They may have different modes of administration. They may be nonconcurrently randomized controls or concurrently randomized controls.
But we use all of the controls that have been enrolled in the platform trial to compare them to the active treatment to gain as much information and use all of the information as efficiently as possible to answer the question. And again, I'm not saying we believe that all of the controls are the same. They vary much, and we believe that they may differ over time or differ over mode of administration, but we can model those differences but still allow those individuals to inform the estimate of the active treatment effect.
>> So you've mentioned that these approaches have been used in COVID. Can you describe how they've helped in the rapid investigation of multiple therapies for COVID and what challenges might have existed had we not had these approaches available to us. >> Sure. So I think we described a little bit about how we can use Bayesian models in these complex settings to really get the most out of all of the information that we have. Maybe that's borrowing the information from other sources or using all the information that we have with -- inside the platform trial.
But what we haven't talked about much is adaptations. So very frequently, Bayesian statistics is used to build adaptive clinical trials. And these are trials that frequently take prespecified interim analyses and perform prespecified adaptations based on rules that we've put in place before the trial has started so that we can efficiently determine if treatments work or not. So at these interims, we may do things such as stop the treatment early for success if there's overwhelming evidence that the treatment works or stop early for futility and don't waste the resources on that treatment if there's evidence that it does not work.
And these are some of the -- these adaptations were put in place in these COVID trials and in many trials that we work on. And Bayesian statistics is really a great toolbox to use for these types of trials because it gives us, as we've talked about, the intuitive summaries that you would want or need to make these decisions. So if I asked the clinician what information would you like to know to be able to stop a trial early for success or futility, you would likely tell me, tell me the probability that the treatment has a response rate better than 30%.
And these are the exact summaries that we have in the Bayesian setting so that we can use them to make these decisions. >> So can't a frequentist approach do the same thing? >> The issue with the frequentist approach is that you have to take into consideration how the data was sampled. And so once you begin making adaptations, you have to take into consideration the potential for those adaptations when you calculate your p-value. And again, the p-value, although it provides evidence if we can accept or reject a hypothesis, it does not necessarily provide the intuitive summaries that are often needed to make informed decisions on if we should stop a trial early for success or futility.
>> So you've certainly described a variety of advantages or situations in which the Bayesian approach may be more facile or more effective than the frequentist approach, but let's talk for a second about the regulatory environment. So specifically, are Bayesian analyses accepted by the FDA to support approvals of drugs, devices, or biologics, and what has been the history of the regulatory review of trials analyzed using a Bayesian approach?
>> Yes, absolutely. We have many situations where we have confirmatory studies that have a Bayesian primary analyses, and these studies have been discussed before the start of the trial with regulatory agencies in the FDA. Oftentimes, these studies are in rare disease, pediatrics, unmet medical needs, platform trials, or places where we want to speed up drug development as quickly as possible. And oftentimes, these situations are when we have some additional complexity where the Bayesian framework can really shine and where we can model that complexity, such as borrowing information outside of the trial or using information more efficiently within the trial.
So yes, I will say that the FDA and regulatory agencies do accept Bayesian analyses. However, there is still a mountain to climb when we're trying to propose this. We are oftentimes held to a high standard to be able to show what those Bayesian analyses will produce and be able to provide evidence that they would not lead to biased results. And so oftentimes, we save these Bayesian analyses for the most complex situations in which they're really needed.
Maybe for better or for worse, but oftentimes if we're doing a very simple trial where a frequentist result will be very similar to that of a Bayesian, let's say, just a simple t-test on the change from baseline, then we don't tend to fight that uphill battle to propose a Bayesian analysis or propose an analysis that individuals are less comfortable with, although, you know, maybe we should to make these more commonplace. >> So I think it's been clear to those of us at JAMA that the number of trials that are now being designed with a primary Bayesian approach is increasing over time.
What do you think the primary motivation for that is, and do you expect that to continue? >> Yeah, I think that although frequentist approaches were the standard, I think that clinicians especially are starting to understand how interpretable Bayesian results can be and are. I also think that the world is getting increasingly complex. And so I think that the Bayesian framework is naturally suited to handle a lot of these complexities and handle situations where we want to answer questions as efficiently as possible in, say, an adaptive trial.
And so we're really starting to see that there are situations where yes, we can, you know, manipulate the frequentist framework to get it to perform how we would want it to perform, but the Bayesian framework is really more naturally suited to do these things. >> So I think at the beginning you described yourself as a Bayesian at heart, so I'm going to ask you a very difficult question. What is a situation in which a Bayesian approach should not be used?
>> This is a hard question. So I would personally say that not necessarily a Bayesian approach shouldn't be used, but a frequentist approach may be preferred if we think that the result from that approach is going to be very similar to the Bayesian result and that there is a very large hurdle to overcome with the audience to get them to accept the Bayesian approach.
So if it's an extremely important study, if the Bayesian result is going to be very similar to the frequentist result, and the audience would be very, very skeptical of the Bayesian result, I may be willing to say that the frequentist approach is the right way to go there. >> [Music] Again, this is Roger Lewis, Senior Statistical Editor at JAMA and the Co-Editor of the JAMA Guide to Statistics and Methods series, and I'd like to thank our guest today, Dr. Melanie Quintana, for talking to us about the Bayesian paradigm for drawing information in clinical trials.
You can find further information about this and other topics on our website at jamaevidence.com or in a recent book from the JAMA Guide to Statistics and Methods. For more audio, please visit us at jamanetworkaudio.com, and you can listen and follow wherever you get your podcasts. Thanks again.
Form Title Download
Media
Transcript
Form Title Bookmark
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Share
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Form Title Accessibility
The cadmore media player is a professional video and audio player experience designed to deliver media as well as relevant transcript, segmentation, and metadata. On the screen is a traditional html audio player with buttons such as play, pause, forward, captioning, language and play speed selection and more. Adjacent to the audio player are generally tabs, one of which shows a scrolling transcript of the audio and the other shows any segmentation of the media, sometimes referred to as chapters. At the top of the audio player is a menu bar containing an Explore button and Tools button. The Explore button allows you to dig deeper into the audio by opening up an explore dialog which contains an about tab which contains rich metadata about the media, a segments tab which contains further information about the audio, and a related tab which contains links to other media related to this media. The tools button opens a menu with options for sharing the media on social media, creating bookmarking links, creating embed codes for your website, generating citations, and more.
Keyboard Shortcuts
Keyboard shortcuts only work while you are in forms mode. By entering forms mode, you will be able to quickly access portions of the media player with single key commands.
Spacebar: Play/Pause Toggle
E: Explore Menu,
O: Tools Menu
T: Transcript Tab,
S: Segments Tab
F: Forward 15,
R: Rewind 15
L: Languages,
P: Playspeed
M: Mute Toggle,
V: Volume Bar
N: Video Only Toggle,
B: Full Screen Toggle
C: Captions Toggle
Escape: Closes dialogs and menus
Z: Next Transcript Chapter
X: Previous Transcript Chapter
Q: Next Transcript Paragraph
W: Previous Transcript Paragraph
Form Title Embed Media
Entire Media
Segment(s)
Custom Clip
Start At:
End At:
Embed Size
-
You Size - Responsive
-
We Size - Responsive
-
Picture Overlay Popup
-
400 x 225
-
512 x 288
-
400 x 700
-
768 x 1024
-
1024 x 512
-
1280 x 608
Form Title Cite
No citation provided.
Info
Segments
Related
Thumbnails
Accessibility
Using forms mode, you can utilize keyboard shortcuts to make usability more efficient.
Space bar will play and pause the media.
You can press A to open the Accessibility Dialog.
This describes many other keyboard shortcuts for media controls.
Exit Clip Mode
The video is currently set-up to show just a clip from 0:00 to video end. You have requested to see a portion of the media outside of the clip.
Click OK to switch to the full media or click if you would like to stay inside the clip. You can always go back to the clip with Tools and Clip Mode