Use of SNP chips to detect rare pathogenic variants: retrospective, population based diagnostic evaluationBMJ 2021; 372 doi: https://doi.org/10.1136/bmj.n214 (Published 16 February 2021) Cite this as: BMJ 2021;372:n214
- Weedon MN, associate professor,
- Jackson L, lecturer,
- Harrison JW, post-doctoral research associate,
- Ruth KS, post-doctoral research associate,
- Tyrrell J, senior lecturer,
- Hattersley AT, professor,
- Wright CF, professor
- Institute of Biomedical and Clinical Science, University of Exeter College of Medicine and Health, Royal Devon and Exeter Hospital, Exeter EX2 5DW, UK
- Correspondence to:
- Accepted 13 January 2021
Objective To determine whether the sensitivity and specificity of SNP chips are adequate for detecting rare pathogenic variants in a clinically unselected population.
Design Retrospective, population based diagnostic evaluation.
Participants 49?908 people recruited to the UK Biobank with SNP chip and next generation sequencing data, and an additional 21 people who purchased consumer genetic tests and shared their data online via the Personal Genome Project.
Main outcome measures Genotyping (that is, identification of the correct DNA base at a specific genomic location) using SNP chips versus sequencing, with results split by frequency of that genotype in the population. Rare pathogenic variants in the BRCA1 and BRCA2 genes were selected as an exemplar for detailed analysis of clinically actionable variants in the UK Biobank, and BRCA related cancers (breast, ovarian, prostate, and pancreatic) were assessed in participants through use of cancer registry data.
Results Overall, genotyping using SNP chips performed well compared with sequencing; sensitivity, specificity, positive predictive value, and negative predictive value were all above 99% for 108?574 common variants directly genotyped on the SNP chips and sequenced in the UK Biobank. However, the likelihood of a true positive result decreased dramatically with decreasing variant frequency; for variants that are very rare in the population, with a frequency below 0.001% in UK Biobank, the positive predictive value was very low and only 16% of 4757 heterozygous genotypes from the SNP chips were confirmed with sequencing data. Results were similar for SNP chip data from the Personal Genome Project, and 20/21 individuals analysed had at least one false positive rare pathogenic variant that had been incorrectly genotyped. For pathogenic variants in the BRCA1 and BRCA2 genes, which are individually very rare, the overall performance metrics for the SNP chips versus sequencing in the UK Biobank were: sensitivity 34.6%, specificity 98.3%, positive predictive value 4.2%, and negative predictive value 99.9%. Rates of BRCA related cancers in UK Biobank participants with a positive SNP chip result were similar to those for age matched controls (odds ratio 1.31, 95% confidence interval 0.99 to 1.71) because the vast majority of variants were false positives, whereas sequence positive participants had a significantly increased risk (odds ratio 4.05, 2.72 to 6.03).
Conclusions SNP chips are extremely unreliable for genotyping very rare pathogenic variants and should not be used to guide health decisions without validation.
Single gene disorders are usually caused by genetic variants that are very rare in the population (<1 in 10?000 people).1 Finding one of these rare pathogenic variants confers a high probability of disease in an individual and their family that requires referral for clinical follow-up. For example, a confirmed pathogenic variant in one of the breast cancer genes BRCA1 or BRCA2 would need urgent follow-up with additional screening and potentially prophylactic surgical mastectomy and oophorectomy.2 Molecular diagnostic laboratories typically use highly accurate DNA sequencing technologies to test for these types of rare pathogenic variants.34
SNP chips are DNA microarrays that test genetic variation at many hundreds of thousands of specific locations across the genome.5 They were initially designed for testing single nucleotide polymorphisms (SNPs) that are common in the population (>1 in 100 people). SNP chips have proven to be excellent for studying common genetic variation, which can be used to assess ancestry,6 as well as predisposition to many complex multifactorial diseases such as type 2 diabetes.78 The genetics community generally recognises that SNP chips perform poorly for genotyping rare genetic variants owing to their reliance on data clustering (fig 1).91011 Clustering data from multiple individuals with similar genotypes works very well when variants are common, as large numbers of data points are available (fig 1, top). However, clustering becomes more difficult as the number of people with a particular genotype decreases; most people will have the reference genotype (normal allele), and distinguishing an alternative genotype (variant allele) from experimental noise is extremely difficult when only a single carrier is present (fig 1, bottom).
Despite this problem, in recent years many SNP chip designs, including those used by many direct to consumer companies, have been augmented to include rare pathogenic variants that cause single gene disorders. As a consequence, consumers of such tests are increasingly being screened for many rare single gene disorders and are potentially receiving medically actionable results, for which they often seek advice from healthcare professionals (fig 2).12 False positive results for rare clinically actionable variants detected by direct to consumer SNP chips have been described in practice guidance,13 several case reports,1415 and two small case series.1617 However, no systematic evaluation of the performance of SNP chips for assaying rare genetic variants has been published. It has been estimated that more than 26 million people had accessed direct to consumer genetic testing at the start of 2019,18 so knowing how accurate these results are is likely to be is crucial in order to interpret rare pathogenic variants detected using SNP chips.
In this study, we used sequencing data from 49?908 UK Biobank participants as a reference standard to do a large scale, systematic evaluation of how well SNP chips detect rare genetic variants.10 We sought to answer two questions. (1) How well do SNP chips perform at detecting pathogenic genetic variants in individuals compared with the sequencing data in the UK Biobank? (2) Are the sensitivity and specificity of SNP chips adequate for rare variants? We used rare pathogenic variants in the BRCA1 and BRCA2 genes (collectively termed BRCA henceforth) that cause hereditary cancers as an exemplar to evaluate the performance of SNP chips in the UK Biobank for genotyping clinically actionable variants. We replicated our findings using data from 21 people who had had direct to consumer genetic testing and shared their data online via the Personal Genome Project.19
Study design, participants, and test methods
We did a retrospective comparison of SNP chip genotyping (index test) with next generation sequencing (here defined as the reference standard) from the UK Biobank and Personal Genome Project participants for whom both datasets were available. We studied 49?960 individuals (55% female) from the UK Biobank. The UK Biobank is a population based research cohort of approximately 500?000 participants recruited in the UK between 2006 and 2010. Approximately 9.2 million people aged 40-69 years who lived within 40 km of one of 22 assessment centres were invited and 5.5% participated.10 The Personal Genome Project is a community project in which participants are invited to publicly share their genetic data.19 We compared variants genotyped by using SNP chips (index test) with next generation sequencing data (reference standard) in the same individual.
Next generation exome sequencing data were available on 49?960 participants, of whom 49?908 also had SNP chip data that had passed quality control. SNP chip data were generated centrally by the UK Biobank, and the exome sequencing data were generated externally by Regeneron and returned to the UK Biobank resource as part of an external access application request.20 A subset of 4037 participants were previously genotyped using the Applied Biosystems UK BiLEVE Axiom Array by Affymetrix (807?411 genetic markers), and the other 45?871 participants were previously genotyped using the Applied Biosystems UK Biobank Axiom Array (825?927 genetic markers) that shares 95% of its marker content with the BiLEVE.10 Participants were genotyped in 106 batches of around 5000 samples. We included samples that passed central UK Biobank quality control on either of the UK Biobank SNP chips and used standard quality metrics to exclude problematic SNPs (missingness rate <5% and Hardy Weinberg P<1×10?6).11 We used the UCSC genome browser liftover tool to convert SNP chip variant positions that were reported in human genome build 37 to 38 coordinates for direct comparison with sequencing data.
Personal Genome Project
We analysed publicly available datasets within the Personal Genomes Project (https://my.pgp-hms.org/) to determine which individuals had both direct to consumer SNP chip and sequencing data in genome build 37. We subsequently downloaded SNP chip data (provided by 23andMe from 2012 to 2019 using Illumina arrays) and genome sequencing data (provided by Veritas Genetics) for 21 people.
We compared variants directly genotyped on the SNP chips with the equivalent positions in the sequencing data from the same individual, excluding sites that were not well sequenced. For genome-wide comparison with SNP chip genotypes, we included only directly genotyped single nucleotide variants with genomic positions present in the genomic Variant Call Format (gVCF) files and covered by more than 15 reads in the sequencing data. For UK Biobank data, we used the minor allele frequency from all 488?377 SNP chip genotyped UK Biobank participants.10 For Personal Genome Project data, we used VCF converter to analyse whether variants detected by the SNP chip were present in the sequence data, and we used the minor allele frequencies from gnomAD and the 1000 genomes project.2122 We tested the genotyping quality of each individual variant on the SNP chips versus sequencing and calculated average performance metrics per variant overall and for common and rare variant subsets in the UK Biobank.
For detailed gene specific comparison with SNP chip genotypes in the UK Biobank, we included directly genotyped single nucleotide variants, as well as small insertions and deletions in the BRCA1 and BRCA2 genes. We defined variants as pathogenic if they were predicted to result in a truncated protein or had previously been classified as likely or definitely pathogenic in the ClinVar database23; we included variants with conflicting reports of pathogenicity that included pathogenic assertions made within the previous five years. We visually examined sequencing data with the Integrative Genomics Viewer for all index and reference positive results to determine whether the variant was present.24 We extracted cancer registry data for breast, ovarian, prostate, and pancreatic cancer for all participants in April 2019 to coincide with the most recent release of Hospital Episode Statistics data. We used logistic regressions to assess the relation between participants who tested positive and any BRCA related cancer. We included age at cancer registry, sex, recruitment centre, death, breast screening, and non-cancer related mastectomy in the regression.
Results are presented in accordance with the standard framework for the validation and verification of clinical molecular genetic tests and STARD guidelines for reporting diagnostic accuracy studies, using sensitivity, specificity, positive predictive value, and negative predictive value to evaluate assay performance.2526
Patient and public involvement
Patients and the public were not directly involved in the design or implementation of this study, as we used previously generated data. As part of the consent process for the UK Biobank, National Health Service patients gave their consent for the collection, storage, and use of their genetic data and medical records by all approved researchers. All of the participant records are linked-anonymised.
Performance of SNP chips for all variants
In the 49?908 UK Biobank participants, we compared genotypes for 108?574 single nucleotide variants that were classified as heterozygous by the SNP chip to reference standard sequencing data. Of the 49?908 participants, 45?871 were genotyped using the Axiom chip and 4037 using the BiLEVE chip. Overall performance across both chips for all variants was very good (table 1), with 3.1×108 true positives, 4.6×109 true negatives, 3.2×106 false positives, and 2.7×106 false negatives. Performance for genotyping common SNPs with a frequency of more than 1% was especially good (table 1), both for the Axiom chip (average sensitivity 99.8%, specificity 99.7%, positive predictive value 99.0%, and negative predictive value 99.9%) and the BiLEVE chip (average sensitivity 99.7%, specificity 99.7%, positive predictive value 98.7%, and negative predictive value 99.9%).
Performance of SNP chips in relation to allele frequency
The genotyping performance of the SNP chips in the UK Biobank was strongly related to the frequency of the variant in the population (table 1; fig 3; supplementary figure A). We found 10?891 (Axiom) and 7408 (BiLEVE) variants on the two UK Biobank SNP chips with a frequency below 0.001% that were also genotyped by exome sequencing in the UK Biobank. For these very rare variants, the sensitivity of the SNP chips to detect heterozygous genotypes was low (29.5% for Axiom and 4.4% for BiLEVE). However, because of the large excess of true negatives, the specificity and negative predictive value both remained high (>99.7% for both chips). The positive predictive value was also strikingly reduced for rare variants compared with common variants—that is, we found a very high proportion of false positives for which the SNP chip (index test) detected a variant allele that was not present in the sequence data (reference standard). For the very rare variants present in UK Biobank, including 4757 heterozygous genotypes across both SNP chips, only 16.1% of Axiom SNP chip heterozygous genotypes (708 true positives in 4239 participants at 3422 variants) were confirmed by the sequencing data, as were only 9.4% (46 true positives in 518 participants at 488 variants) for the BiLEVE chip. We observed a similar performance for very rare variants in the Personal Genome Project data (supplementary figure B), with a positive predictive value of 14% for variants with a population frequency below 0.01% in 21 people (83 true positives at 594 variants).
Performance of SNP chips for rare pathogenic variants
We went on to evaluate the performance of the SNP chips in the UK Biobank for 1139 pathogenic and likely pathogenic variants in BRCA that were included on the chips (fig 4; supplementary figure C); 916 (80%) of these are rare with an allele frequency below 0.01% in the UK Biobank. The performance of both chips was very poor for genotyping pathogenic BRCA variants: overall sensitivity 34.6%, specificity 98.3%, positive predictive value 4.2%, and negative predictive value 99.9% (table 2). Across both SNP chips, 425 pathogenic BRCA variants were detected in 889 UK Biobank participants. Of these, just 17 variants in 37 participants were present in the sequencing data, and the others were false positives; the most common true positive was present in 10 participants and had conflicting and uncertain interpretations in ClinVar. A further 43 pathogenic BRCA variants were present in the sequencing data of 70 participants but were not detected by either SNP chip despite being assayed. The performance of both chips for genotyping pathogenic BRCA variants was very poor (table 2): sensitivity 33.0% and specificity 99.7% for the Axiom chip; sensitivity 50.0% and specificity 82.7% for the BiLEVE chip.
Risk of BRCA related hereditary cancer in UK Biobank participants with a pathogenic BRCA variant genotyped by SNP chips was similar to the risk of those without an SNP chip BRCA variant. As expected, given the low positive predictive value, the risk of BRCA related cancers in UK Biobank participants with a positive SNP chip result for any pathogenic BRCA variant was similar to the age matched risk in the UK Biobank (odds ratio 1.31, 95% confidence interval 0.99 to 1.71). In contrast, those with a positive sequencing result—including the 107 participants with BRCA variants assayed by either chip, plus another 137 with pathogenic BRCA variants not assayed by either chip—had a markedly increased risk (odds ratio 4.05, 2.72 to 6.03).
We also investigated rare pathogenic and likely pathogenic variants in the Personal Genome Project, where the SNP chips used were made by a different manufacturer and based on a different design than those used in the UK Biobank. Across 21 individuals, 100% (47/47) of rare (minor allele frequency <0.01%) pathogenic single nucleotide variants and 74% (25/34) of likely protein truncating insertions and deletions in known disease causing genes were incorrectly genotyped by the SNP chips. These included a pathogenic variant in KCNQ2 that causes early infantile epileptic encephalopathy and protein truncating variants in MSH2 and MSH6 that confer a very high risk of colorectal cancer.2728 A rare pathogenic variant in ABCC8 that causes congenital hyperinsulinism was incorrectly genotyped as homozygous in 43% (9/21) of individuals.29 Overall, 95% (20/21) of people investigated had a least one false positive rare pathogenic variant compared with sequencing.
We have shown that SNP chips are extremely poor for correctly genotyping very rare variants compared with sequencing data and that, for an individual person, a positive result for a very rare pathogenic variant is more likely to be wrong than right. This finding can be explained as follows. An individual rare variant is very unlikely to be present in any clinically unselected individual, so most results for that variant are true negatives. However, because SNP chips typically assay many thousands of rare variants simultaneously, and have a specificity that is less than 100%, false positive results will occur and outnumber true positives across all rare variants. Any individual person is therefore more likely to have a false positive result across all the rare variants than a true positive result at that variant.
Strengths and limitations of study
We present the largest evaluation to date of the quality of genotyping for rare variants, using retrospective data from the UK Biobank. Although our analyses were limited by the array designs used in the UK Biobank, our results reflect a fundamental property of SNP chip technology, so we expect our findings to be broadly applicable to most SNP chip datasets. SNP chips are widely recognised as not being good at genotyping very rare variants.91011 Some more recent SNP chips have been designed to include only low and intermediate frequency coding variants (>1 in 5000), for which the SNP chips perform relatively well. However, SNP chips are increasingly being augmented with very rare pathogenic variants, which, as we have shown, are not well genotyped. This is an inherent problem of using SNP chip technology to genotype very rare variants and is caused by both the rarity of the variant and the data clustering method on which SNP chips rely (fig 1). Relying on data clustering means that both variant frequency and batch size will affect accuracy, with fewer individuals per batch leading to more genotyping errors for rare variants. As a result, although the performance of chips from different manufacturers may differ owing to different underlying chemistries,5 our findings are likely to be generalisable to most SNP chips, and the results are strikingly similar across different SNP chips used in both our UK Biobank and Personal Genome Project datasets. Sequencing is not affected by the same technical problem as SNP chips and thus provides a much more accurate method for genotyping rare variants.30
The problem of SNP chips incorrectly genotyping very rare variants can be partially remedied through improved probe design, removal of poorly performing probes, using custom variant detection definitions,31 using multiple probes for individual variants, or adding positive laboratory controls to improve clustering of variants. Many consumer genetic testing companies use these additional quality control methods, supplemented by validation of important variants through DNA sequencing, to improve the accuracy of variants they advertise and report directly to consumers. However, most direct to consumer companies (including those focused on ancestry or other non-medical traits) also allow customers to download and analyse their raw data, which will often include many thousands of additional rare variants assayed on their SNP chip that have not undergone stringent quality control and are therefore much more likely to be false positives.15 A recent study found that 89% of consumers of genetic tests downloaded their raw data and 94% of those used at least one third party interpretation service to analyse the results.32 We have therefore focused on the direct to consumer scenario because errors in raw SNP chip data could cause significant harm to consumers without further validation. However, erroneous results from SNP chips used in research biobanks can also lead to false associations and wasted resource in the development of new treatments against the wrong targets.333435
The inherent technical limitation of SNP chips for correctly detecting rare genetic variants is further exacerbated when the variants themselves are linked to very rare diseases. As with any diagnostic test, the positive predictive value for low prevalence conditions will necessarily be low in most individuals. For pathogenic BRCA variants in the UK Biobank, the SNP chips had an extremely low positive predictive value (1-17%) when compared with sequencing. Were these results to be fed back to individuals, the clinical implications would be profound. Women with a positive BRCA result face a lifetime of additional screening and potentially prophylactic surgery that is unwarranted in the case of a false positive result. Conversely, although the false negative rate of SNP chips is generally low, many very rare pathogenic variants are not included in the design and will therefore be missed.36 Women who receive a false negative BRCA result but have a strong family history of breast and/or ovarian cancer are at high risk of developing cancer that could be greatly reduced through preventive surgeries and other interventions.37
Using a large population research cohort and a small consumer genetic testing cohort, we have shown that positive results from SNP chips for very rare variants are more likely to be wrong than right. We therefore urge clinicians to validate any SNP chip results from direct to consumer companies or research biobanks by using a standard diagnostic test before recommending any clinical action. In addition, people with symptoms or a family history of breast and/or ovarian cancer who have received a negative SNP chip result should not be reassured that their risk is low,13 and standard referral guidelines should be followed for diagnostic testing (see https://cks.nice.org.uk/breast-cancer-managing-fh). We suggest that, for variants that are very rare in the population being tested, genotyping results from SNP chips should not be routinely reported back to individuals or used in research without validation. Clinicians and researchers should be aware of the poor performance of SNP chips for genotyping very rare genetic variants to avoid counselling patients inappropriately or investing limited resources into investigating false associations with badly genotyped variants.
What is already known on this topic
SNP chips are an accurate and affordable method for genotyping common genetic variants across the genome
They are often used by direct to consumer genetic testing companies and research studies
However, several case reports suggest that they perform poorly for genotyping rare genetic variants when compared with sequencing
What this study adds
This study, using large scale SNP chip and sequencing data from the UK Biobank, confirms that SNP chips are highly inaccurate for genotyping rare, clinically actionable variants
SNP chips had a very low positive predictive value of less than 16% for detecting very rare variants; most variants with population frequency below 0.001% were false positives
Very rare variants assayed using SNP chips should not be used to guide health decisions without validation
Allele: each of two or more alternative forms of DNA that are found at the same location on a chromosome
Exome: ~1-2% of the human genome that codes for proteins
Genotyping: method for determining the base (A, G, T, or C) present at a specific location in a person’s DNA
Heterozygous: two different alleles in an individual
Homozygous: two identical alleles in an individual.
Negative predictive value: proportion of normal alleles found by the index test that are confirmed by the reference standard (true negative/(true negative + false negative))
Positive predictive value: proportion of variant alleles found by the index test that are confirmed by the reference standard (true positive/(true positive + false positive))
Sensitivity: proportion of variant alleles detected by the reference standard that are also found by the index test (true positive/(true positive + false negative))
Sequencing: method for determining the order of bases in a DNA sample
Single gene disorder: disease caused by, or with a high probability of developing due to, a rare genetic variant in a specific single gene
Single nucleotide polymorphism (SNP): type of single nucleotide variant that is common and present in more than 1% of the population (pronounced “snip”)
SNP chip: DNA microarray that is used to genotype known genetic variants (typically SNPs) in the population
Specificity: proportion of normal alleles detected by the reference standard that are also found to be normal by the index test (true negative/(true negative + false positive))
Variant (single nucleotide variant): position in the genome where an individual differs from the reference human genome by a single base change (ie, a substitution of a single letter of DNA). A variant may be rare or common in the population
We thank Charles Warden for pointing us towards the Personal Genome Project resource and Tim McDonald for useful advice on clinical performance metrics. We also thank participants of both the UK Biobank and Personal Genome Project for generously donating their data for research.
Contributors: MNW and LJ contributed equally to this work. CFW, MNW, and ATH designed the study. CFW wrote the first draft of the manuscript. All authors reviewed and edited the manuscript. MNW, LJ, JWH, KSR, JT, ATH, and CFW were involved in data processing, statistical analysis, and interpretation. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. MNW and CFW are the guarantors.
Funding: This research has been conducted using the UK Biobank Resource under Application Number 49847, using the University of Exeter High-Performance Computing (HPC) facility. ATH is a Wellcome Trust Senior Investigator (098395) and an NIHR senior investigator. The funders had no influence on study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: The UK Biobank has received ethics approval from the National Health Service National Research Ethics Service (ref 11/NW/0382).
The lead author (the manuscript’s guarantor) affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
Data sharing: The data reported in this paper are available via application directly to the UK Biobank. Direct to consumer data are available from the Personal Genome Project website.
Dissemination to participants and related patient and public communities: We will disseminate this research through the UK Biobank network. The Personal Genome Project participants have freely shared their data online to be used by the research community.
Provenance and peer review: Not commissioned; externally peer reviewed.
This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.