Instrumental variable meta-analysis of aggregated randomized drug trial data for evaluating proposed target mechanismsBMJ 2021; 372 doi: https://doi.org/10.1136/bmj.n346 (Published 25 February 2021) Cite this as: BMJ 2021;372:n346
- Sarah F Ackley, postdoctoral scholar1,
- Fanny Elahi, assistant professor2,
- M Maria Glymour, professor1
- 1Department of Epidemiology and Biostatistics, University of California, San Francisco, 550 16th Street, San Francisco, CA, USA
- 2UCSF Weill Institute for Neurosciences, University of California, San Francisco, CA, USA
- Correspondence to: M M Glymour @mariaglymour on Twitter) (or
Instrumental variable approach
In our study (doi:10.1136/bmj.n156), we used a novel set of methods to combine evidence from previously reported randomized trials that share a common biological mechanism.1 An instrumental variable approach can be applied to randomized controlled trials to adjust intention-to-treat effect estimates for adherence to randomly assigned treatment. Adherence in this setting is defined as a biological measure of change in the treatment target. These estimates adjusted for adherence can then be meta-analyzed to estimate the effect of the biological target on the outcome of interest. In our paper, we used the example of treatments for Alzheimer’s disease that target amyloid, where adherence is change in brain amyloid levels. Mathematically, the analysis estimates how average cognitive decline among study participants in each randomly assigned treatment group varies with average amyloid reduction in that treatment group. Since assignment to treatment group is randomized, differences between groups cannot be attributed to other factors, so this approach preserves the key advantage of randomization for evaluating causation. We used this approach to combine estimates from all 14 trials with available results on both reductions in amyloid levels and cognitive change.
When and why to use
This method is appealing in the context of multiple randomized trials with evidence of drug engagement with a specific biological target and wanting to draw conclusions about the mechanistic importance of that biological target for patients’ outcomes. Individual drugs might be unsuccessful in clinical trials because the effect sizes are typically too small for any one reasonably sized trial to detect an effect. Conversely, the success of any individual drug should not be considered as evidence for the putative biological mechanism because the chance that a single drug shows a statistically significant benefit increases as more and more drugs targeting the same mechanism are tested. Using conventional statistical significance thresholds, an average of one in 40 trials of a non-effective biological target will appear effective, so it is essential to aggregate results across multiple trials.
In addition to combining results from multiple trials, our method is useful because not all drugs affect the same biological target to the same extent. To evaluate the effect of a specific change in a biological target, we cannot use the intention-to-treat estimate of the effect of a drug because different drugs will affect the target to differing degrees. We must correct for these differences. Although meta-analysis is the most common tool to combine effect estimates, a conventional meta-analysis of intention-to-treat effect estimates is not appropriate in this setting because the effect of each drug on the outcome should be proportional to the effect of that drug on the biological target. Some drugs had relatively large effects on amyloid, whereas others had quite small effects. Any combined analysis must account for the difference in how much each drug influenced the biological target.
For illustrative purposes, we plot the data from a single trial—the EMERGE trial of aducanumab, along with the fit from our estimation procedure (fig 1). We plot the mean change in amyloid measured using the standardized uptake value ratio and the mean change in cognition for each randomization arm, with 95% confidence intervals. The slope of the line connecting these points is the expected change in cognition per unit change in standardized uptake value ratio. We then estimate the best fitting line for each trial, accounting for uncertainty in both change in standardized uptake value ratio and in cognition, as well as differences in sample size in the full and positron emission tomography subsamples.
For each trial, we obtain an estimate for the effect of a unit change in the biological target on the outcome of interest. The result from the analysis pools results from all trials for a more precise estimate. We also can obtain pooled estimates from any subset of studies—for example, trials with published data or trials that used a particular outcome measure.
Strengths and limitations
Clinical trials provide an opportunity to understand not only drug efficacy but also disease biology. This method provides an approach combining all results into a single estimate of the effect of modifying a biological target on an outcome of interest. Such a combined result is much more precise than results from any individual trial—for example, the findings in our study indicate that large cognitive benefits of amyloid reduction are unlikely within the time frame of the conducted trials.1
This type of analysis has limitations. The pooled result across all trials is easily interpreted but only if the effect of changing the biological target on the outcome does not vary across drugs. This analysis assumes that the drug has no influence on the outcome that is not mediated by the biological target. By making the effect estimates directly comparable, this analysis will nonetheless help to identify drugs that might influence the outcome through mechanisms unrelated to the target. Owing to data limitations, it might not be possible to account for the covariance between the biological target and the outcome that might be induced by common causes of both. Finally, this analysis requires postulating the timeframe on which the biological target influences the outcome.
Our methods highlight the importance of full data reporting from clinical trials, including terminated trials, along with consistent measurement and reporting practices across trials. Given the cost and potential harm to study participants of clinical trials and the urgency of finding treatments that work, these data should be leveraged to yield biological insight and accelerate progress towards finding effective drugs.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from the National Institutes of Health National Institute on Aging for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Provenance and peer review: Commissioned; not externally peer reviewed.