Signal detection of adverse events with imperfect confirmation rates in vaccine safety studies using self‐controlled case series design

Using ICD-coded cases without individual confirmation yielded a lower point estimate of risk for IS post rotavirus vaccination; however, the risk remained statistically compatible with that for chart confirmed cases only. Analysis using healthcare databases to evaluate risk of IS if conducted without case confirmation may be insufficient to confirm a low-level risk. IS episodes after vaccination were not more severe.

Vaccine safety studies are often electronic health record (EHR)-based observational studies. These studies often face significant methodological challenges, including confounding and misclassification of adverse event. Vaccine safety researchers use self-controlled case series (SCCS) study design to handle confounding effect and employ medical chart review to ascertain cases that are identified using EHR data. However, for common adverse events, limited resources often make it impossible to adjudicate all adverse events observed in electronic data. In this paper, we considered four approaches for analyzing SCCS data with confirmation rates estimated from an internal validation sample: (1) observed cases, (2) confirmed cases only, (3) known confirmation rate, and (4) multiple imputation (MI). We conducted a simulation study to evaluate these four approaches using type I error rates, percent bias, and empirical power. Our simulation results suggest that when misclassification of adverse events is present, approaches such as observed cases, confirmed case only, and known confirmation rate may inflate the type I error, yield biased point estimates, and affect statistical power. The multiple imputation approach considers the uncertainty of estimated confirmation rates from an internal validation sample, yields a proper type I error rate, largely unbiased point estimate, proper variance estimate, and statistical power.

Healthcare databases have been used in the past four decades to identify, refine, and evaluate potential safety signals of marketed medical products. Critics have challenged this research because data are from secondary sources and because some published studies have lacked robust methods for exposure and outcome definition and failed to adequately control for biases. We review the history of healthcare databases used in pharmacovigilance for quantifying adverse outcomes associated with therapeutics, methods to improve the quality of this research, and best practices for pharmacoepidemiologic studies. Drug and vaccine safety studies increasingly use information from multiple healthcare databases, with analyses that aim to keep patient-level identifying data with local research custodians. Analytic methods, including high-dimensional exposure propensity scores, use large numbers of variables to reduce confounding and further anonymize patient data. However, due to gaps in and complexities of the available databases, the value of the research depends on experts with knowledge about the clinical context (e.g., how products are prescribed and taken, how outcomes are diagnosed and recorded, what risk factors must be considered), understanding the nuances of individual databases and the clinical practice patterns they represent, and utilizing study designs that minimize bias, particularly confounding by medication indication.