Evaluation of Serological Tests for SARS-CoV-2: Implications for Serology Testing in a Low-Prevalence Setting

Abstract

Background

Robust serological assays are essential for long-term control of the COVID-19 pandemic. Many recently released point-of-care (PoCT) serological assays have been distributed with little premarket validation.

Methods

Performance characteristics for 5 PoCT lateral flow devices approved for use in Australia were compared to a commercial enzyme immunoassay (ELISA) and a recently described novel surrogate virus neutralization test (sVNT).

Results

Sensitivities for PoCT ranged from 51.8% (95% confidence interval [CI], 43.1%–60.4%) to 67.9% (95% CI, 59.4%–75.6%), and specificities from 95.6% (95% CI, 89.2%–98.8%) to 100.0% (95% CI, 96.1%–100.0%). ELISA sensitivity for IgA or IgG detection was 67.9% (95% CI, 59.4%–75.6%), increasing to 93.8% (95% CI, 85.0%–98.3%) for samples >14 days post symptom onset. sVNT sensitivity was 60.9% (95% CI, 53.2%–68.4%), rising to 91.2% (95% CI, 81.8%–96.7%) for samples >14 days post symptom onset, with specificity 94.4% (95% CI, 89.2%–97.5%).

Conclusions

Performance characteristics for COVID-19 serological assays were generally lower than those reported by manufacturers. Timing of specimen collection relative to onset of illness or infection is crucial in reporting of performance characteristics for COVID-19 serological assays. The optimal algorithm for implementing serological testing for COVID-19 remains to be determined, particularly in low-prevalence settings.

The coronavirus disease (COVID-19) pandemic caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is a global public health emergency on an unprecedented scale. First reports in late December 2019 described a cluster of patients with pneumonia, linked to a live animal market in Wuhan, China [1–3]. To date, laboratory testing has comprised detection of SARS-CoV-2 virus using reverse-transcriptase PCR (RT-PCR) assays, predominantly from patients meeting specific epidemiological criteria. However, the immense scale of RT-PCR diagnostic testing has placed extraordinary demands on laboratories, with challenges relating to supply chains of swabs, laboratory reagents, and the human and financial resource required to support population-level testing.

Over the past 2 months, there has been rapid development of serological assays for COVID-19 in a number of countries [4]. Serological tests rely on detection of specific antiviral antibodies (immunoglobulin M [IgM], immunoglobulin G [IgG], immunoglobulin A [IgA], or total antibody) in patient serum, plasma, or whole blood. The broad array of serological tests now available vary both in analytical performance and in their particular utility in the overall public health response to COVID-19. The most publicized serological tests for COVID-19 have been lateral flow immunoassays, also known as serological point of care tests (PoCT), which have been manufactured and deployed in several countries. Most available PoCT involve detection of anti-SARS-CoV-2 IgM or IgG antibodies through binding to immobilized antigen (generally domains of the spike [S] protein) attached to colloidal gold, followed by detection of the conjugates by an anti-human IgM or IgG antibody. In addition, a control line is usually included in the assay, which helps determine whether the test result is valid. The relatively cheap and simple nature of lateral flow assays means that production is suited to scaling-up for increased testing capacity.

In many countries, including the United States and Australia, the rapid development and implementation of COVID-19 diagnostics has meant that normally stringent regulatory criteria have not been applied to all tests, with limited published data supporting assay performance in clinical settings. Here, in order to inform the deployment of PoCT in Australia, we compared the performance characteristics of 5 commercially available PoCT with a commercially available enzyme-linked immunosorbent assay (ELISA) and a recently described surrogate virus neutralization assay (sVNT), using samples from (1) patients with RT-PCR-confirmed COVID-19; (2) patients who were RT-PCR negative but presented with respiratory symptoms during the peak of the pandemic in Australia, and (3) patients before the COVID-19 pandemic.

METHODS

Clinical Samples and Patient Populations

A testing panel was specifically developed to test PoCT devices for this study (Supplementary Material), consisting of 3 patient populations: (1) sera from 91 patients with SARS-CoV-2 detected by RT-PCR from upper and/or lower respiratory tract specimens; (2) sera from 36 patients with seasonal coronavirus infections or other acute infections (eg, dengue, cytomegalovirus, Epstein-Barr virus); and (3) serum from a random cohort (56 patients) of the Australian population obtained in 2018.

One of the devices was also tested against serum samples from 1217 patients who were SARS-CoV-2 RT-PCR negative but presented to a hospital emergency department between 6 February and 15 April 2020, spanning the initial peak of the COVID-19 pandemic in Australia.

Serum samples were obtained from a large academic hospital in Melbourne, Australia (Royal Melbourne Hospital, RMH), or the Victorian state reference laboratory for virology (Victorian Infectious Diseases Reference Laboratory, VIDRL). Convalescent patients were followed up at home by the RMH@Home Hospital in the Home Team. Information on each cohort is provided in the Supplementary Material.

Cases were classified clinically as mild (not admitted to hospital), moderate (admitted to a hospital ward, but not the intensive care unit [ICU]), or severe (admitted to ICU). Of the 91 cases, 71 were mild, 17 were moderate, and 3 were severe.

RT-PCR

SARS-CoV-2 RNA was detected using the Coronavirus Typing assay (AusDiagnostics), a 2-step, heminested multiplex tandem PCR, with 7 coronavirus RNA targets plus a proprietary artificial sequence as an internal control. All positive samples underwent additional confirmatory testing for SARS-CoV-2 at VIDRL, using previously published primers [5].

Serological PoCT

Serological PoCT devices were tested exactly as per the manufacturer’s stated instructions for use, including use of plastic droppers and buffers provided in the kits. Devices were provided through the Australian Government Therapeutic Goods Administration, based on device availability at the time of the study (Supplementary Table 1). In brief, 10 μL of serum was added to the device, with addition of between 60 and 100 μL of the manufacturer’s provided buffer. Devices were incubated at room temperature according to the time period defined in the instructions for use (generally 10–15 minutes). All results were read as per the instructions for use. Testing was performed by laboratory technicians, all of whom had undergone competency training in the use of lateral flow assays. Testing of each sample in the serum panel was performed in duplicate, with a triplicate deciding test for discordant results. Any faint line present at test termination was considered a positive result. Results were recorded in a password-protected database available only to study investigators. All patient samples were deidentified.

Five PoCT were evaluated: OnSite COVID-19 IgG/IgM Rapid Test; VivaDiag COVID-19 IgM/IgG Rapid Test; Hangzhou AllTest COVID-19 test; Wondfo SARS-CoV-2 Antibody Test; and Hightop SARS-CoV-2 IgM/IgG Antibody Rapid Test. Additional information is provided in Supplementary Table 1.

Enzyme-Linked Immunosorbent Assay

ELISA testing was performed using the EUROIMMUN Anti-SARS-CoV-2 ELISA, a commercially available ELISA (Supplementary Table 1). The assay involves semiquantitative detection of anti-SARS-CoV-2 IgA or IgG antibodies in serum through binding to a recombinant structural antigen (S1 domain of the Spike protein) fixed to reagent wells. If test sera contain anti-SARS-CoV-2 antibodies, a second incubation step using enzyme-labelled anti-human IgA or anti-human IgG will catalyze a color reaction, detected by an optical density reader. Semiquantitative results were reported as a ratio as per the manufacturer’s instructions for use and interpreted as follows: (1) ratio <0.8, negative result; (2) ratio ≥0.8 to <1.1, borderline result; and (3) ratio ≥1.1, positive result. Performance characteristics were determined using the same sera panel as the PoCT, along with 36 additional samples (33 samples from 19 patients with COVID-19 confirmed by RT-PCR; 2 samples from MERS-CoV–positive patients, and 1 serum from a SARS-CoV-1–positive patient; Supplementary Material).

SARS-CoV-2 Surrogate Virus Neutralization Test

To further assess antibody response, we used a recently described sVNT, that detects circulating antibodies directed against the spike protein receptor binding domain (RBD) in an isotype- and species-independent manner, based on antibody-mediated blockage of interaction between the ACE2 receptor protein and the RBD [6]. In brief, 10 μL of test serum was diluted with 90 μL of sample dilution buffer and incubated with horseradish peroxidase conjugated SARS-CoV-2 RBD protein (HRP-RBD); the test solution was added to wells coated with fixed ACE2 receptor. The degree to which test serum inhibited binding of the HRP-RBD to ACE2 receptors, compared to control serum, was determined by optical density reading, with 20% inhibition and above considered a positive result. Sera tested in the sVNT included samples from 110 patients with RT-PCR–confirmed COVID-19 and 142 samples from 142 control patients, of which 36 samples were from patients with seasonal coronavirus or other acute infections, and 106 samples were from a random cohort of the Australian population obtained in 2016 and 2018 (Supplementary Material). A first round of testing on all samples followed the instructions for use; subsequently, samples within the 10% coefficient of variation (CV) range as stated in the instructions for use (inhibition cutoff of 18%–22%, n = 21) were repeated in duplicate to assess for interrun variation.

Microneutralization Assay

An in-house microneutralization assay was performed at the University of Melbourne. SARS-CoV-2 isolate CoV/Australia/VIC01/2020 [7] passaged in Vero cells was stored at −80°C. Serial 2-fold dilutions of heat-inactivated plasma were incubated with 100 TCID₅₀ (50% tissue culture infectious dose) of SARS-CoV-2 for 1 hour and residual virus infectivity was assessed in quadruplicate wells of Vero cells; viral cytopathic effect was read on day 5. The neutralizing antibody titer was calculated using the Reed/Muench method as previously described [8, 9].

Statistical Analysis

All statistical analyses were conducted using R (version 3.6.3) or GraphPad Prism (version 8.4.2). Binomial 95% confidence intervals (CI) were calculated for all proportions. Differences in nonnormally distributed numerical data were calculated using the Wilcoxon Rank sum test. Receiver operating characteristic (ROC) area under the curve (AUC) analysis was performed in GraphPad Prism (version 8.4.2).

Ethics

Ethical approval for this project was obtained from the Melbourne Health Human Research Ethics Committee (RMH HREC QA2020052).

RESULTS

Comparison of Commercial ELISA vs RT-PCR

The overall sensitivity for either IgA or IgG detection was 67.9% (95% CI, 59.4%–75.6%) and specificity was 72.8% (95% CI, 62.6%–81.6%) (Table 1). The sensitivity for IgA or IgG detection increased to 93.8% (95% CI, 85.0%–98.3%) when only samples collected >14 days post symptom onset were considered (Table 2), and a significant rise in signal/cutoff ratio was observed for both IgA and IgG over time (P < .001) (Figure 1).

ROC AUC analysis was performed for both IgA and IgG. Overall, the IgA ROC AUC was 0.74 (95% CI, 0.69–0.81) and the IgG ROC AUC was 0.66 (95% CI, 0.59–0.72) (Supplementary Figure 1A and 1B). However, when only samples collected >14 days post symptom onset were included, the ROC AUC increased to 0.92 (95% CI, 0.87–0.96) for IgA and 0.97 (95% CI, 0.94–0.99) for IgG (Supplementary Figure 1C and 1D). No cross-reactivity with seasonal coronavirus infection was observed for IgG, although 4/17 (23.5%) samples from patients with seasonal coronavirus (2 HKU1, 1 NL63, and 1 OC43) were positive for IgA. Neither of the 2 samples with anti-MERS-CoV antibodies displayed cross-reactivity for SARS-CoV-2 IgA or IgG, but 1 sample with anti-SARS-CoV-1 antibodies had positive results for SARS-CoV-2 IgA and IgG (ratios 3.81 and 1.26, respectively; Figure 2).

Comparison of PoCT and RT-PCR

We compared the sensitivity and specificity of 5 PoCT devices, using RT-PCR as our reference standard, and interpreting PoCT results as positive when either an IgM or IgG result was read as positive. Overall, the sensitivities ranged from 51.8% (95% CI, 43.1%–60.4%) to 68.6% (95% CI, 60.1%–76.3%), and specificities from 95.6% (95% CI, 89.2%–98.8%) to 100.0% (95% CI, 96.1%–100.0%) (Table 1 and Figure 3A and 3B). When only samples collected >14 days were considered, the sensitivities ranged from 78.5% (95% CI, 66.5%–87.7%) to 93.8% (95% CI, 85.0%–98.3%) (Table 2).

Using the OnSite device (for which there was a surplus of kits), additional testing was conducted on 1217 samples from patients who presented with respiratory symptoms but tested RT-PCR negative for SARS-CoV-2. In total, 39/1217 (3.2%) samples tested positive for IgM and/or IgG. On further testing, 6/39 samples (15.4%) tested positive to IgA and/or IgG using the ELISA assay, of which 1 was confirmed by sVNT (inhibition 63.9%) when an inhibition cutoff of 20% was employed (see below). In addition, this sample was confirmed as positive using a whole-virus microneutralization assay. This patient presented 21 days following symptom onset, with significant epidemiological risk factors for SARS-CoV-2 acquisition, and likely represents a true infection.

Using the highest performance device characteristics (sensitivity 68.6% [Wondfo] and specificity >99.9% [Hightop]) and lowest performance characteristics (sensitivity 51.8% [VivaDiag] and specificity 95.6% [OnSite]) as hypothetical best and worse scenarios, respectively, the performance of PoCT was assessed across a range of SARS-CoV-2 population prevalence estimates (0.1%, 1%, 5%, and 10%; Table 3 and Supplementary Figure 1). With the best performing PoCT characteristics at an estimated SARS-CoV-2 period prevalence in Australia of 0.03%, the positive predictive value was only 17.1%.

Comparison of sVNT and RT-PCR

In total, 311 samples were also tested using the sVNT assay. Applying a 20% inhibition cutoff and using RT-PCR as the reference standard, the sensitivity of sVNT was 62.7% (95% CI, 55.0%–70.0%); this increased to 91.2%% (95% CI, 81.8%–96.7%) when only samples collected >14 days post symptom onset were considered (Table 1 and Table 2). Specificity was 94.4% (95% CI, 89.2%–97.5%), with cross-reaction observed for 8 samples (Figure 4 and Supplementary Table 2). Increasing the inhibition cutoff to 25%, or repeating samples with an initial inhibition score between 18% and 22% improved the specificity to 99.3% (95% CI, 96.1%–99.9%) with little change in sensitivity (Table 1 and Table 2). The % CV for the in-house control sample with respect to the percentage inhibition was 10.8% between runs and 5.8% within run.

DISCUSSION

Accurate laboratory testing is integral to the prevention and control of COVID-19. The unprecedented scale of diagnostic testing has led to the rapid development and implementation of a large range of diagnostic assays for SARS-CoV-2, including serological tests. However, there are limited peer-reviewed data on the performance characteristics of serological tests, and in order to best inform the implementation of these assays, high-quality postmarket validation data are urgently needed to guide laboratories, public health agencies, and governments in the appropriate and responsible use of such tests [10].

In this study, we assessed the performance characteristics of 5 serological PoCT, a commercial ELISA, and a commercial novel sVNT against a large serum panel from a cohort of over 100 patients with RT-PCR–confirmed SARS-CoV-2. In keeping with other studies [11], the sensitivity of all assays was low (<70%) when all sample collection time points were considered. However, as expected given the reported antibody response to SARS-CoV-2 infection, sensitivity increased considerably when samples collected >14 days post symptom onset were assessed [12], with the majority achieving sensitivities over 90%. Our findings provide further support for recent commentary suggesting that current serological assays have limited, if any, role in the diagnosis of acute COVID-19, with RT-PCR remaining the gold standard for diagnosis in the acute setting [13, 14]. Specificities for PoCT ranged from 92.4% to 100%; it is possible this may reflect differences in the antigens used in each assay, although specific information about the SARS-CoV-2 recombinant antigen used in the assay was not described for most PoCT. In keeping with previous reports [15, 16], when both IgA and IgG components of the ELISA were considered, specificity was low (72.8%), but considering IgG alone, specificity increased to 97.8%.

This study is one of the first to utilize a recently described sVNT assay [6]. Previous work describing the development of this assay reported a 95%–100% sensitivity and 100% specificity using cohorts in Singapore and China [7]. In our cohort at >14 days post symptom onset the test achieved sensitivity of 91.2% (95% CI, 81.8%–96.7%) and specificity of 94.4% (95% CI, 89.2%–97.5%). Although limited clinical data are available on the cohort used to develop and validate the assay [6], it is possible that our relatively mild clinical cohort may generate lower antibody titers than a more severely unwell cohort, potentially influencing sensitivity of the assay [12, 17]. Of note, the majority of our nonspecific (false positive) samples in the sVNT assay recorded inhibition just over the 20% cutoff. However, specificity improved when either (1) a higher inhibition cutoff was used or (2) samples within an arbitrary range (based on the instructions for use reported % CV of the assay) were tested in triplicate. In our low-prevalence setting where the test is more likely to act as a confirmatory assay, raising the inhibition cutoff to 25% increased the specificity to 99.3% (95% CI, 96.1%–99.9%), thus improving clinical utility. Alternatively, introduction of an equivocal range for the assay with repeat testing for samples within this range, would be another approach to mitigate potential assay variation.

In contrast to acute diagnosis, there are settings where high-quality serological assays will have utility, including (1) defining antibody prevalence in key populations such as frontline workers; (2) determining the extent of COVID-19 infection within the community; (3) identifying individuals for further evaluation of therapeutic immunoglobulin donation; and (4) vaccine development and evaluation. For (3) and (4), it is essential to have a good quantification of the functional neutralizing antibodies among donors or vaccines and the PoCT and ELISA assays do not provide an endpoint titer. However, in order to appropriately deploy serological testing, it is critical to understand the limitations of test performance in the epidemiological context in which tests are used. This is particularly important in a setting such as Australia, which, based on the number of reported cases of COVID-19 (8001 cases as of 2 July 2020), has an estimated COVID-19 period prevalence of 0.03% [18]. As such, even with highly sensitive and specific serological tests, the majority of positive results are likely to represent false positives. When considering the use of serology to inform policies relating to relaxing of physical distancing interventions, specificity of the assay becomes critical. If the majority of those identified as immune are actually false-positive results, then the threshold to maintain immunity within the community will not be achieved [19].

Analogous to HIV testing in low-prevalence settings [20], serological testing for SARS-CoV-2 may require a 2-step approach, whereby a sensitive high-throughput screening assay is followed by a high-specificity assay for confirmation (eg, neutralization testing or western blot). This approach could facilitate seroepidemiological studies in low-prevalence settings, which are required to better understand the extent of COVID-19 infection at a population level. Ongoing questions remain, however, about the duration and type of antibody response to SARS-CoV-2, particularly around the protective effect of neutralizing antibodies against future reinfection [12]. Accordingly, the concept of an “immunity passport” that facilitates return to workplaces or school should be interpreted with caution, and the World Health Organization currently recommends the use of PoCT immunodiagnostic assays in research settings only, and not for clinical decision making until further evidence is available [21].

A key strength of this study was our systematic collection of convalescent samples. By establishing a community collection platform, we tested over 50 patients who were more than 21 days post symptom onset. Ideally, validation of serological assays should be performed against a testing panel that includes samples from (1) patients at acute and convalescent stages of infection (to assess sensitivity), and (2) patients with other human coronavirus infections (to assess specificity). Given the range of serological assays now available, there is a critical need for standardized protocols, including reference standards, across laboratories when conducting evaluations of emerging serological assays. Further, the relatively recent emergence of SARS-CoV-2 means there are limited data on the sensitivities of serological assays at 3–6 months post infection. Future work should assess any potential drop in sensitivity at varying time points post infection.

In summary, our data describe the performance characteristics of 5 PoCT devices, a commercially available ELISA assay, and a newly developed sVNT. Overall, our findings are in keeping with recent position statements that note serological assays have limited, if any, role in the diagnosis of acute COVID-19 infection. Our data strongly suggest that current PoCT devices should not be used in the diagnosis of acute COVID-19 or as the sole assay in population-level serosurveys. Nevertheless, there are settings where high-quality serological assays will have clinical utility. The curated panel of samples assembled for this study is being expanded and provides a valuable repository for rapid validation of new serological assays as they become available.

Supplementary Data

Supplementary materials are available at The Journal of Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Acknowledgments. We thank the scientific staff of the Pathology Department at the Royal Melbourne Hospital, particularly Maria Bisignano, Siobhan Carne, Elizabeth Shilling and Merina Gurung; the Serology Department of the Victorian Infectious Diseases Reference Laboratory; and the RMH@Home staff, including Beverley Cox and Samuel Strapps and the staff of the RMH Emergency Department, particularly Dr Mark Putland. We especially thank the patients who contributed to this study. We thank Professor Linfa Wang and his team at Duke-NUS Medical School for providing the sVNT kit to us before it is commercially available.

Financial support. This work was supported by the National Health and Medical Research Council (NHMRC) of Australia Medical Research Future Fund (grant number APP2002317); the Jack Ma Foundation (grant to K. S.); and the A2 Milk Company (grant to K. S.). The Melbourne World Health Organization Collaborating Centre for Reference and Research on Influenza is supported by the Australian Government Department of Health. D. A. W. and K. S. are supported by Investigator Grants from the NHMRC (grant numbers APP1174555 and APP1177174, respectively). B. P. H. is supported by a NHMRC Practitioner Fellowship (grant number APP1105905). K. B. is supported by an NHMRC Postgraduate Scholarship (grant number GNT1191321).

Potential conflicts of interest. All authors: No reported conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.

References

Author notes