A hallmark of science is reproducibility: if an experiment is conducted twice using the same methods, the result should be the same. In most epidemiologic research, reproducibility usually extends the confirmation using multiple data sources. Of course, it does not matter how many times a result is reproduced if the methodology used to derive the result is faulty.

In October 2018, Stanton Glantz and his team published a study that showed an association between myocardial infarction (MI), or heart attacks, and e-cigarette use. The paper received significant attention, much of which questioned the methodology and the authors’ interpretation of the results. In fact, the first R Street Responds piece was a critical commentary of the work.

Despite the concerns, corrections and criticisms of the first paper, Glantz and colleauge Dharma N. Bhatta have published a new study “confirming” the results of the first. Unfortunately, as was the case the first time, this new study merely doubles down on the flawed assumptions and methods of the original.

Many studies attempting to evaluate the health effects of e-cigarettes, including this one, fail to adequately control for the effect of past smoking behavior. Often, simply including a “current smoking status” variable in the model fails to capture the effect of smoking and/or cessation duration. So, for example, the “former smoker” definition would include people who quit the previous month and those who haven’t smoked in decades. Such an oversight matters, because for risk of MI, duration of cessation has a strong influence on outcomes. Specifically, after only one year, former smokers cut their risk of coronary heart disease (including MI) to half that of current smokers and after 15 years former smokers experience no greater risk than non-smokers. Failing to take this into account influences the accuracy of the study, requiring additional measures of smoking behavior to correct.

It is difficult to circumvent this limitation because most available datasets do not include questions about duration or intensity of smoking. However, given the strong association between smoking and e-cigarette use, these measures are potential confounders of any association between e-cigarettes and a slow-developing health condition such as coronary artery disease (1 percent prevalence among people under 40) and other smoking-related diseases, none of which are accounted for in Bhatta and Glantz.

Another disturbing aspect of this study relates to e-cigarettes entering the United States market in 2007. Accounting for the fact that the majority of e-cigarette users are current or former smokers, and 69 percent of the sample used both products concurrently, it is difficult to believe that e-cigarette use could have independently and significantly increased the odds of MI after fewer than 10 years of availability (data set ends in 2015). This is especially significant since e-cigarettes did not even truly gain popularity until about 2011.

Bhatta and Glantz wantonly avoid this reality by including MIs that occurred before 2007 in all but one of their analyses. Put simply, this is bad science. It is obvious that e-cigarettes could not have contributed to the MIs that occurred prior to their introduction in the United States, however, the authors ignore this and justify their choice because restricting the analysis to just MIs occurring in 2007 or later would limit their sample size and ability to detect an association. The reality is that doing so would limit their ability to detect the association they assumed. This is made further evident because they reported significant associations between MI and current combustible cigarette use, which shows that it is possible to detect an association for a known risk factor for MI using the restricted sample.

Further evidence that the authors have found a spurious correlation is that, when attempting to establish a direct, unadjusted relationship between e-cigarette usage and increased MI, the study failed to find an association using the full dataset, and did not include an analysis of that relationship for only MIs occurring after 2007. However, rather than conclude that perhaps there was no association, the authors instead attribute their poor methodological choices to inadequate sample size, without acknowledging that they, themselves, chose to use this dataset. A case can be made for this being some of the “best available data,” but it is irresponsible to frame your findings as definitive while simultaneously blaming sample size for failure to find hypothesized associations.

Another issue is that many of the paper’s arguments hinge on the probability of MI among e-cigarette users and the probability of MI among combustible cigarette users being mathematically independent—which is to say that the occurrence of one event does not influence the occurrence of the other, for example, flipping a fair coin twice and it landing on heads the first flip does not impact the probability of it landing on tails for the second flip. It is plausible to assume that the odds of a combustible cigarette user having an MI do not influence the odds of an e-cigarette user having an MI. However, given that the vast majority of e-cigarette users are current or former smokers, knowledge of past smoking behavior almost certainly impacts the probability of MI from e-cigarette use. Furthermore, their proof of mathematical independence relies on a statistical procedure that requires a much larger sample size than is required for testing for a direct association between the two exposures and MI. But, at any rate, if a study’s sample size is inadequate for the exposure of interest, it is irresponsible to report results as conclusive and to perform more sophisticated analyses that inherently require more observations. It is likewise irresponsible to employ faulty methodology and then use it to force your data to validate your hypotheses.

Much like the previous paper from Glantz and his team, the present study utilizes questionable methodology to arrive at the desired conclusion and mistakes correlation for causation. Indeed, rather than confirming the results of their past study, they merely replicated much of their own bad science. Scientists have an ethical obligation to conduct objective research, and that is impossible if the study methods are chosen specifically to obtain a desired result that happens to align with previous findings. Setting aside ethics, bad science permeates beyond the walls of academia and influences medical and policy decisions and impacts individual lives. Irresponsible research that jumps to unsubstantiated conclusions can have very negative effects and, in this case, it may very well deter smokers from completely switching to less harmful e-cigarettes.