Background

The aim of genetic testing is to determine the etiology of a suspected genetic disorder to influence medical management or provide a diagnosis that can guide genetic counseling. For this reason, genetic variants must be classified accurately and consistently. Professional societies and governmental regulatory agencies have provided general guidance regarding the classification, reporting, and long-term follow-up of variants. Guidelines for the classification of variants were provided in 2007 by the American College of Medical Genetics and Genomics (ACMG) (Richards et al. 2008). The ACMG, in collaboration with the Association for Molecular Pathology (AMP) and the College of American Pathologists (CAP), is currently updating their guidelines towards a more evidence-based system that provides semi-quantitative guidelines for variant classification. In publicly circulated near-final drafts of these guidelines (Lyon et al. 2013; Richards et al. 2014), the ACMG/AMP/CAP will be recommending a multi-tier classification system, grouping variants based upon perceived risk of disease association: pathogenic, likely pathogenic, benign, likely benign, and variant of uncertain significance. These guidelines will recommend that the reporting of novel sequence variants from a clinical diagnostic laboratory must include a clinical interpretation based on the best data available at the time of testing. In clinical practice, laboratories with additional gene-specific information and expertise can use these guidelines to complement their findings. For example, our laboratory has previously outlined its approach to variant interpretation for the BRCA1 and BRCA2 genes (Eggington et al. 2014).

Locus-specific variant databases (LSDBs) have evolved as a resource for both researchers and clinicians to interpret the clinical relevance of genetic sequence variants associated with Mendelian disorders. The aim of LSDBs is to facilitate variant interpretation through the use of aggregated data, with variant specific data and classifications provided by researchers, laboratories, and clinicians. The new ACMG/AMP/CAP sequence variant classification guidelines will likely continue to recommend the cautious use of LSDBs to establish whether a genetic variant has been reported as associated with disease (Richards et al. 2008). Recently, a group of researchers and clinicians evaluated and curated the largest public Lynch syndrome (MLH1, MSH2, MSH6, and PMS2) variant database and demonstrated that the majority of user-submitted clinical classifications were incorrectly classified within the database when evaluated by evidence-based standards (Thompson et al. 2014). This research demonstrates that a cautious use of LSDBs warrants a thorough review of each database’s evidence-based methodologies for classification. In addition, there are many LSDBs available for clinical variant classification and it is generally well accepted that differences between databases exist (Celli et al. 2012; Mitropoulou et al. 2010). However, there has been no comprehensive review to systematically determine the extent of these discrepancies and, therefore, the potential impact on clinical classification.

In this study, we undertook a cross-comparison of LSDBs to measure concordance across and within five publicly accessible BRCA1 and BRCA2 variant databases. Our aim was to analyze whether LSDBs provide a consistent classification for the possible disease association of genetic variants. Our results show substantial disparity of variant classifications among and within publicly accessible variant databases. We also show that the LSDBs in this study failed to provide sufficient evidence to verify the databases’ pathogenic classifications for less straightforward variant classifications, with the exception of genetic variants that have been thoroughly characterized in the literature.

Methods

To establish an unbiased dataset for comparison of clinical classifications within and between public databases, we performed a retrospective database query of BRCA1 and BRCA2 variants identified among consecutive patients who were referred to Myriad Genetic Laboratories, Inc., for sequencing and large rearrangement testing in November and December of 2013 (Fig. 1). Using this snapshot of data as of November 6, 2013, we then cross-referenced each variant in the dataset with its classification in five publicly accessible databases: (1) the Breast Cancer Information Core (BIC), an online, open-access breast cancer mutation database maintained by the National Human Genome Research Institute at the National Institutes of Health (NIH) (Szabo et al. 2000); (2) the Leiden Open Variation Database 2.0 (LOVD, chromium.liacs.nl/LOVD2/cancer/home.php?select_db=BRCA1, chromium.liacs.nl/LOVD2/cancer/home.php?select_db=BRCA2), maintained by the Leiden University Medical Center, the Netherlands (Fokkema et al. 2011); (3) ClinVar, a freely accessible public archive maintained by The National Center for Biotechnology Information (NCBI) at the NIH with the goal of reporting relationships between human variations and phenotypes (Landrum et al. 2014); (4) the BRCA1 and BRCA2 Universal Mutation Database (UMD, http://www.umd.be/BRCA2/), which contains published and unpublished information about BRCA1 and BRCA2 mutations reported in a network of 16 French diagnostic laboratories (Beroud et al. 2000); and (5) the Human Gene Mutation Database (HGMD), a paid subscription database maintained by the Institute of Medical Genetics in Cardiff (Stenson et al. 2009).

Fig. 1
figure 1

Flow diagram of the study design showing BRCA1 and BRCA2 variants identified for study due to detection in the sequencing of 24,650 consecutive patient samples presumed to be unrelated or only distantly related. The number of unique variants among these patients and the number of these variants shared across the BIC, ClinVar, HGMD (paid version), LOVD, and UMD databases are represented

Full size image

Variants and their respective classifications in LSDBs were compiled to analyze discrepancy rates between and within databases. To facilitate comparison between different classification schemes, we grouped classifications within databases into three major categories: pathogenic (pathogenic and likely pathogenic), benign (benign and likely benign), and variants of uncertain clinical significance (VUS). The criteria used for group classifications from each database are listed in Table 1. Multiple instances of the same variant, within the same database, were considered as “conflicting” if they were assigned both a pathogenic and benign classification. Classifications were not considered conflicting within the same database if a variant was classified as pathogenic and VUS or benign and VUS. In these cases, the pathogenic or benign clinical classification was used in cross-database comparisons. The variants with classifications that were found in the “Other Classifications” category were excluded from the comparison.

Table 1 Definitions of clinical classifications for comparison across databases
Full size table

For quality assurance of the data, once the classifications in the data set were recorded, a blinded review was performed by two independent reviewers who were given a list of 100 variants from the overall dataset. These 100 variants were randomly selected in order to obtain a representative subset of the variants analyzed here. Each reviewer queried all five databases, noting the classification if available. Once the review was complete, the subset of variants was compared to the initial list of classifications to verify their accuracy. This approach allowed for the discovery and correction of any systematic errors, which arose primarily from inconsistent nomenclature across databases.

Agreement across databases

To investigate the degree to which LSDBs agree on variant classifications, we stratified the 2017 unique variants based on the number of databases in which they were present (e.g., all variants that were seen in four databases formed one group). For each database in which a variant occurred, its classification was noted. Agreement was defined as all LSDBs in which a variant occurred assigning the same classification. The frequency of agreement (%) was calculated as the number of variants where agreement was found against the number of variants assigned the specified classification in one or more databases (for sample sizes in the different categories, see Fig. 2). For example, a variant was judged to be in the “pathogenic” subset if at least one database assigned that classification. We recorded this agreement separately for all variants judged to be pathogenic, benign, and VUS.

Fig. 2
figure 2

Agreement between database classifications of variants from an unbiased set of 2017 BRCA1 and BRCA2 unique genetic variants identified in patients. The proportion of agreement is presented according to the number of databases in agreement (x-axis) for (a) all databases examined (BIC, ClinVar, HGMD, LOVD, and UMD) and (b) all databases except HGMD, which had the highest degree of discrepancy. Variants listed in all databases as pathogenic and likely pathogenic were grouped into the “pathogenic” subset (red). Variants listed in all databases as benign or likely benign were grouped into the “benign” subset (blue). “VUS” (grey) represents variants for which all databases described assigned a classification of variants of uncertain significance

Full size image

Discrepancies within databases

We also examined the potential for conflicting variant classifications to occur within the same database. BIC, HGMD, and UMD provide a single “master” clinical classification per unique variant on the primary variant report page; however, ClinVar and LOVD currently do not provide a single “master” classification and instead list the conflicting entries. The variants with conflicting entries in these databases are listed in the Supplemental Material. We totaled the number of variants seen for each classification (pathogenic, benign, VUS) in each of the five LSDBs and recorded the conflicting classifications when multiple instances of the exact same variant were observed in the same database.

Analysis of additional evidence utilized by databases

The use of literature and unique empirical data is an important facet of variant classification. However, the clinical utility of such information may be subject to debate when, e.g., the raw data are unavailable to support a reported conclusion or an intermediate functional effect is reported in a biochemical assay. To assess the use of literature and unique empirical data by LSDBs, we first collected a subset of variants from the 124 unique variants found in all five LSDBs that could be conservatively classified as VUS. The following criteria were used to select this subset: (1) variant listed in all five of the databases, (2) variant receives a default classification of VUS when excluding literature and other empirical evidence, and (3) definitively classified by at least one of the five databases as pathogenic (or likely pathogenic). The draft form of updated guidelines from the ACMG/AMP/CAP Interpretation of Sequence Variations Workgroup was used for this analysis (Table 3). To establish a default VUS classification for criteria #2, the following types of variants were considered: missense variants, intronic variants greater than two nucleotides inside the intron from the native RNA splice acceptor or donor site, in-frame insertions/deletions, and variants within the 5' untranslated region (UTR). We next noted the reported classification of these variants in the five LSDBs and plotted all pathogenic and benign classifications (Fig. 3). Any variants that already held a VUS clinical classification in a given database were excluded from further analysis.

Fig. 3
figure 3

Analysis of the LSDB clinical classifications (benign or pathogenic) of variants considered “VUS” by the criteria outlined in the “Methods” section. Sixty-three of the 124 unique variants shared across the BIC, ClinVar, HGMD (paid version), LOVD, and UMD databases met these criteria. To determine how LSDBs vary in their treatment of these “challenging to classify” variants, we compared the clinical classifications of each “VUS” in all five databases. Variants with conflicting entries of VUS and either pathogenic or benign in the same database are categorized as either pathogenic or benign in the figure (see Table 1). Variants with conflicting classifications of both pathogenic and benign in the same database were excluded from this figure

Full size image

We also evaluated the content of the BIC, ClinVar, LOVD and UMD databases to determine whether they provided sufficient, verifiable supporting data for an independent reviewer to concur with the databases’ pathogenic classification. As HGMD had the highest discrepancy rate when all databases were compared, it was excluded from this analysis. To perform this analysis, we chose a set of “challenging to classify” variants, which we defined as variants present in all four databases (BIC, ClinVar, LOVD, and UMD) that would be conservatively classified as VUS per the criteria listed previously but that were classified as “pathogenic” in one or more databases. The evidence listed in the database was then evaluated using the evidence-based criteria released in draft form by the ACMG/AMP/CAP Interpretation of Sequence Variations Workgroup, which are soon to be published (Lyon et al. 2013; Richards et al. 2014). Sufficient supporting data was defined as verifiable data contained or referenced in the database that met the minimal requirements for a “Likely Pathogenic” classification (note—for the purposes of this evaluation, a variant’s listing in a public database was not used as a supporting line of evidence for classification).

Results

During the 2-month study period (Nov–Dec 2013), we evaluated the variants identified in 24,650 consecutive patient samples referred to our laboratory for sequencing and large rearrangement testing in the BRCA1 and BRCA2 genes. From these patient samples, we obtained 2017 unique BRCA1 and BRCA2 variants among the study population. We then categorized these variants by the number of databases in which they were identified. Of the 2017 unique finds, 1327 were present in one or more databases, 931 were present in two or more databases, 633 were present in three or more databases, 328 were present in four or more databases, and 124 were present in all five databases (Fig. 1).

Agreement across databases

The ability of multiple LSDBs to agree on a given variant classification is illustrated in Fig. 2a. For pathogenic and VUS classifications, it is clear that the level of agreement consistently decreases as more database evaluations are added. For VUSs in particular, there is no agreement once the variant is observed in a least four of the five databases in this study. Although the percentage agreement increases slightly when a “benign” variant is observed in three databases, it falls off again when observed in four. Figure 2b illustrates the agreement analysis, excluding HGMD which had the highest rate of discrepancy, and shows the same general trend as Fig. 2a.

Discrepancies within databases

Of the 2017 variants evaluated in this study, ClinVar contained 455 variants with 18 of those having conflicting classifications (Table 2). LOVD contained 470 variants with 61 assigned conflicting classifications. These results indicate that when differing classifications are allowed within a database, as is the case for ClinVar and LOVD, discrepancies in clinical classifications may occur.

Table 2 Discrepancies within BRCA1/BRCA2 databases for the 1327 variants evaluated
Full size table

Analysis of additional evidence utilized by databases

Of the 124 variants shared across all five LSDBs, 63 variants would be conservatively classified as VUS according to the simplified guidelines outlined in the “Methods” section. However, Fig. 3 illustrates that different databases vary significantly in the reported clinical classification for these variants when additional evidence is considered (i.e., literature and unique empirical data). Our results show that HGMD favors pathogenic clinical classifications for 61 of 63 variants in this subset that other databases classify as either VUS or benign (note—HGMD currently states on its Frequently Asked Questions (FAQ) webpage that its current database is contaminated with variants listed as pathogenic that are now known to be benign and that “to resolve this issue is likely to be a slow iterative process.”).

Within the subset of 63 variants identified, 18 were found that could be considered “challenging to classify” (Table 3). Among these 18 variants, without exception, those with sufficient empirical data to support the LSDB pathogenic classifications were those that had been published in peer-reviewed literature. In this subset, UMD and LOVD performed the best in supporting their pathogenic classifications through the use of appropriate literature references (71 % and 40 % of the pathogenic claims, respectively) (Table 3). BIC and ClinVar performed equally poorly, both failing to support any of the pathogenic classifications with empirical evidence at the time of this study.

Table 3 Evaluation of the supporting data provided in BRCA1/2 databases for the pathogenic classification of 18 challenging variants as measured by evidence-based guidelines for variant interpretation of “likely pathogenic”(Lyon et al. 2013; Richards et al. 2014)
Full size table

Of special note, in Table 3 are the conflicting classifications for BRCA1 c.5408G>C (p.Gly1803Ala) between LOVD and UMD. The latter classifies the variant as pathogenic and the former as benign. The two databases provide overlapping literature citations to support their classifications. However, while UMD classifies the variant as pathogenic, the comments listed for the literature cited in the variant information suggest that the variant is trending towards a benign classification. The pathogenic classification is therefore likely to be a data entry error or an instance where incomplete information is provided to support the given classification. Similar differences were discovered in other databases. For example, a LOVD entry for the well-characterized pathogenic mutation BRCA1 c.181T>G (p.Cys61Gly) is listed as benign (Caligo et al. 2009; Goldgar et al. 2004). Also discovered were likely data entry errors for two of the 18 variants in Table 3, which were submitted to ClinVar through the Sharing Clinical Reports Project (see Table 3 footnotes). Collectively, this highlights the importance of reviewing all available literature and supporting information regarding a variant classification before issuing a clinical report, as database classifications can be discordant and incomplete.

Discussion

Accurate and consistent data curation is critical to the value derived from LSDBs in categorizing the relationship between genetic variation and disease, particularly for clinical applications. In this study, we performed a retrospective analysis of 2017 BRCA1 and BRCA2 variants identified in our laboratory. We compared the clinical classifications of these variants across BIC, ClinVar, HGMD (paid version), LOVD, and UMD to determine whether LSDBs provide a consistent clinical classification.

Our results show substantial disparity of variant classifications among databases. In our evaluation, it was rare for all five databases to agree on variant classification. For example, of 116 variants present in all five databases that received a classification of “pathogenic” in at least one database, only four variants were classified as pathogenic in all five databases (Fig. 2a). Even when the outlier database (HGMD) was removed from the comparison, this did not mitigate the decreased level of agreement we observed as more databases are considered. These results not only highlight an absence of increased accuracy when a variant is observed in multiple databases but also the disparity in classification schemes used across LSDBs. Although two thirds of the variants identified in the patient samples were listed in one or more BRCA1 and BRCA2 databases, the remainders were not listed in any LSDB, likely owing to their rarity.

Two of the five evaluated databases do not provide a “master” clinical classification and allow for internal database conflicts (pathogenic and benign) for the same variant. These conflicts limit the overall utility of LSDBS and highlight the inconsistent methodologies used to populate the database.

As information from genetic testing may be used to make important clinical decisions that may involve prophylactic surgery, the ACMG/AMP/CAP guidelines recommend a conservative and evidence-based approach to variant interpretation, with the goal of avoiding speculation. When sufficient data is absent for an evidence-based classification, the guidelines recommend classifying variants as having uncertain clinical significance. However, it is important to note that all public databases included in our analysis contain a disclaimer that the database is to be used for research purposes only. Our findings illustrate that some public databases frequently lean more towards aggressive classification when evidence-based protocols indicate that, in a clinical setting, a VUS classification would be more appropriate for particular variants.

For an unbiased subset of 18 variants evaluated in this study, application of these evidence-based guidelines to existing US government-sponsored and international-based BRCA1 and BRCA2 databases shows that the BIC, ClinVAR, LOVD, and UMD databases failed to provide sufficient verifiable content, independent of published literature, to achieve evidence-based verification of their listed “pathogenic” classifications. This might suggest that the use of a database that relies solely upon literature references would be more prudent, such as the use of HGMD. Yet, as described on the HGMD FAQ web page, this database is known to be significantly contaminated with “pathogenic” variants classified in the literature as benign. This information correlates with our findings that HGMD variant classifications were frequent outliers (Fig. 3). However, not all literature used to support a LSDB’s pathogenic classification provided sufficient evidence to meet the evidence-based standards for “likely pathogenic.” This underscores the need for users to independently evaluate referenced literature. Interestingly, even when published literature provided sufficient evidence for a pathogenic classification, the four databases failed to show concordance in the use of the best available literature.

Recently, Thompson et al. reported an evaluation of the LOVD-hosted InSIGHT database for mismatch repair variants. In this analysis, the application of evidence-based criteria resulted in the reclassification of 66 % of the eligible variant entries, including the reversal of pathogenic classifications to benign and vice versa (Thompson et al. 2014). Considering that most public LSDBs do not include a user-tracking mechanism, it is unlikely that the laboratories, clinicians, and patients who previously used the incorrect, non-evidence-based classifications were actively notified of the variant reclassifications. This poses a significant challenge and burden for the user of public LSDBs that has not yet been resolved.

The ability for laboratories to provide sufficient patient- and family-specific data in public databases is challenging because of country-specific laws protecting personal health information. To counter this, databases that attempt to provide empirical data often simplify complex phenotype/genotype information in efforts to anonymize the data for data sharing. However, such simplified data may be misleading. In BRCA1 and BRCA2 LSDBs in which only the proband disease status is provided (e.g., UMD), ascertainment bias must be considered in all assessments as the majority of tested patients are being selected for breast and/or ovarian cancers; thus, truly benign variants may seem to be associated with disease because of this bias. It is therefore critical to develop and use clinically validated methods that rely on rigorous control matching to account for this bias (Eggington et al. 2014; Pruss et al. 2014).

Conclusions

This analysis presents a systematic evaluation of several commonly used LSDBs and their limitations in clinical variant classification. Our results show substantial disparity of variant classifications among and within publicly accessible variant databases. Although LSDBs have been well established for research applications, our results suggest that several challenges inhibit their wider use in clinical practice. Healthcare providers should exercise caution when using these research tools for clinical purposes. With so many BRCA1/2-specific databases now available, it is a challenge to identify the most updated and accurate resources. The number of available databases also presents the challenge of curating the often conflicting variant classifications available (Celli et al. 2012; Mitropoulou et al. 2010). We recommend that for any test that may result in definitive and irreversible clinical intervention, the laboratory issuing the genetic report should carefully evaluate each piece of data used for variant classification.