Background

Next-generation sequencing technologies and high-performance computers define the culture-independent era of microbial ecology. High-throughput sequencing of DNA barcode marker regions, namely the bacterial 16S rRNA gene or fungal internal transcribed spacer (ITS) ribosomal regions, have allowed researchers to characterize complex microbial communities at a depth not previously possible with culture-based methods. Hypervariable regions of the 16S rRNA gene have been extensively studied and adopted by researchers to describe prokaryotic microbial communities, and a mix of ribosomal markers have been used to describe fungal communities [1] over the past 25 years [2]. The ITS region, comprising the ITS1, 5.8S, and ITS2 segments, was recently selected as the formal DNA barcode for fungi [3,4,5], although there is a lack of consensus regarding which ITS (ITS1 or ITS2) to utilize as a barcode [6,7,8]. It remains unclear which of the ITS primer sets has the best resolution for fungal diversity, and papers targeting either ITS segment have been published at near equal frequencies [8,9,10].

Pipelines for processing fungal ITS amplicon datasets such as CLOTU [11], CloVR-ITS [12], PIPITS [1], and others [13] are available in the literature, but most of the tool-development effort has been towards generating nearly automated pipelines for filtering, trimming, and clustering of amplicon reads into operational taxonomic units (OTUs). Less emphasis has been placed on assigning taxonomy to representative OTU sequences in a dataset. Linnaean taxonomy provides a controlled vocabulary that communicates ecological, biological or geographic information. Linking OTUs to functionally meaningful names, which typically depends upon species-level resolution, is key to addressing biological and ecological hypotheses. Processing sequencing reads, in addition to taxonomy assignment of sequences, can be completed using various bioinformatics pipeline tools. The most popular are Mothur [14], QIIME [15], and USEARCH [16]. There are a variety of algorithms to use for the taxonomy assignment step, which include: BLAST [17], Ribosomal Database Project (RDP) Naïve Bayesian Classifier [18], UTAX [19], and SINTAX [20]. The RDP Classifier (RDPC) uses Bayesian statistics to find 8-mers that have higher probability of belonging to a given genus. Based on these conditions, RDPC estimates the probability that an unknown query DNA sequence belongs to the genus [18]. The UTAX algorithm looks for k-mer words in common between a query sequence and a known reference sequence, and calculates a score of word counts. The score is used to estimate confidence values for each of the taxonomic levels, which are then trained on the reference database to give an estimate of error rates [19]. The SINTAX algorithm predicts taxonomy by using k-mer similarity to identify the top hit in a reference database, and provides bootstrap confidence for all ranks in the prediction [20]. Local alignment, most commonly implemented in BLAST [17], is still occasionally used for taxonomy assignment of high-throughput sequence datasets. However use of BLAST to identify OTUs in amplicon-based microbiome datasets has low accuracy as demonstrated previously [20,21,22], and discussed by Wang et al. [18].

The UNITE reference database is a curated database of all International Nucleotide Sequence Database Collaboration (INSDC) fungal sequences, and is the most commonly used reference database for fungal amplicon analyses [23,24,25]. Recently the Ribosomal Database Project released the Warcup Fungal Database [26], a curated version of UNITE and INSDC. Apart from previously published database comparisons which showed the accuracy of UNITE [23] and Warcup fungal databases [26], all comparative studies of taxonomy classifiers of which we are aware, have analyzed only prokaryotic organisms [22, 27, 28]. Since only a small fraction of microbial species estimated to be on the planet have been described, taxonomic classification is not a trivial task and no algorithm is 100% precise. Several types of classification errors are possible, as highlighted in Table 1. The RDPC, UTAX, and SINTAX classifiers report a confidence value for the classification given to an OTU so that the user can set a cutoff value below which no name is given. Even though a number of databases and tools have been developed to enable high-throughput analyses of environmental sequences, researchers still need to solve the problems caused by misidentified or insufficiently identified sequences [5]. Further, some poorly sampled fungal lineages reduce the ability of a classifier to confidently assign OTUs to the correct fungal lineage regardless of the classification algorithm used.

Table 1 Types of classifications
Full size table

This study tested whether established taxonomic classifiers for fungal ITS DNA sequences generate similar profiles of the fungal community. Specifically, we compared the power (proportion of assigned OTUs at a given level) and consistency (agreement of OTU assignment across classifiers) of the RDPC, UTAX, and SINTAX classification algorithms. Power and consistency were compared across i) ITS1 or ITS2 regions, ii) OTU-clustering approaches, and iii) merged or single stranded reads. Further, we created a Python tool that functions independently of OTU-picking method to merge taxonomy assignments from multiple classifier programs into an improved consensus taxonomy, and generates several output files that can be used for subsequent community analysis.

Methods

Data accessibility

Sample origins, barcode regions, and accession numbers for all datasets used in the current study can be found in Table 2. Implementation of the tool presented in this paper requires users to download and install the following software: RDPC [https://github.com/rdpstaff/classifier], USEARCH version 8 for UTAX, and USEARCH version 9 or later for SINTAX [http://drive5.com/usearch/download.html], R v2.15.1 or later [https://www.r-project.org], Python version 2.7 [https://www.python.org]. Detailed installation and analysis instructions, including all custom scripts used in the analysis and a test dataset are available in Additional file 1, or for download from GitHub: [ https://github.com/natalie-vandepol/compare_taxonomy ]. All of the custom Python scripts described in the methods section can be downloaded from the CONSTAX.tar.gz file (Additional file 2). All the steps described in the methods section are automated through the constax.sh script, but are included as independent scripts in CONSTAX.tar.gz so they can be easily modified to suit the user’s needs. An overview of the data analysis workflow is available in Fig. 1.

Table 2 Sample origins, barcode regions, and accession numbers for datasets
Full size table
Fig. 1
figure 1

Overview of CONSTAX workflow. Bubbles highlighted by gray box are automated through constax.sh

Full size image

Generation of operational taxonomic units

For the ITS1-soil and ITS2-soil datasets (Table 2), forward and reverse reads were merged with PEAR version 0.9.8 [29]. Merged reads were randomly sampled to one million reads to reduce computational time. Reads were quality-filtered, trimmed, dereplicated, clustered at 97% similarity (the standard sequence similarity value), and OTU-calling was performed using USEARCH version 8.1.1831 [16]. Analysis of plant datasets (ITS1-plant and ITS2-plant) began with the processed 97% similarity OTUs provided by the authors [10].

For the ITS1/2-BC and ITS1/2-UN datasets, reads were quality-filtered as above, but differed in OTU-generation method. First, a clustering algorithm that generated OTUs using the UPARSE [19] algorithm was used to call OTUs for ITS1-BC and ITS2-BC. Second, the UNOISE2 algorithm [30] that performed denoising and generated exact sequence variants (ESVs) [31] was used for ITS1-UN and ITS2-UN. Each set of OTUs and ESVs were randomly sampled to 500 for the comparative taxonomic analysis described in the next section. Sample and abundance data were not used in this study. The code for the OTU-picking pipeline described above is available in Additional file 3.

Database formatting and classifier training

The UNITE fungal database [23], release 31–01-2016, containing 23,264 sequences was used in the current study. A custom script (FormatRefDB.py) was developed in Python 2.7 to format the database, starting from the general fasta release, for each classifier to ensure training was completed with identical databases. For RDPC training, custom Python scripts (subscript_lineage2taxonomyTrain.py, subscript_fasta_addFullLineage.py) were used to give each Species Hypothesis a unique name and remove special text characters. Prior to UTAX training and SINTAX classification, custom Python scripts were used to make minor changes to header lines of the fasta file. After formatting, these versions of the UNITE database were used to train classifiers. All the formatting and training scripts above are automated through the constax.sh script, users need only specify the location of the reference database.

Taxonomy assignment

Taxonomy was assigned to the OTUs with RDPC version 11.5 [18, 32], UTAX from USEARCH version 8.1.1831 [19, 33], and SINTAX from USEARCH version 9.2 [16]. This step generated three tables (one from each classifier) with a taxonomic assignment at each of the seven ranks of the hierarchy (Kingdom, Phylum, Class, Order, Family, Genus, Species). We used the default settings, a 0.8 cut-off, to serve as a baseline for comparison. Researchers may choose to use less stringent cut-offs, depending on the goals of their studies. The cut-off can be specified in the config file contained in CONSTAX.tar.gz (Additional file 2).

Post-taxonomy data processing

A custom Python script (CombineTaxonomy.py) was developed to standardize the taxonomy table formats, filter the output files at the recommended quality score, and create the consensus taxonomy. Additionally, the script produces a combined and improved (higher power) taxonomy table by concatenating the information contained in the taxonomy tables from RDPC, UTAX, and SINTAX. Rules developed to merge the taxonomy assignments implemented in the Python script are detailed in Table 3. Briefly, a majority rule (two out of three OTUs classified) was used when classifiers did not assign the same name to a representative sequence. When there was not a clear majority rule, the name with the highest quality score was chosen. The CombineTaxonomy.py script is also automated through the constax.sh script. All analyses downstream of the consensus OTU assignments were completed in R version 3.3.2 [34] and graphs were generated with the R package ‘ggplot2’ [35]. R code used to generate the graphs is also available in the CONSTAX.tar.gz, and automated through constax.sh script.

Table 3 Rules adopted to generate the combined taxonomy table
Full size table

Results

Power of classifiers

Classification power differed across RDPC, UTAX, and SINTAX (Fig. 2). Also, the total number of assigned OTUs varied across datasets, ITS region, and OTU-generation approach. In general, the highest number of assignments at each level of the taxonomic hierarchy was observed in the ITS1-soil dataset shown in Fig. 2a [36]. Classifications for the ITS2-soil dataset [37] follow the same general pattern as ITS1-soil, but overall had lower power (Fig. 2b). Although, UTAX had higher classification power for some ITS1 datasets at Kingdom level (Fig. 2c, Additional file 4: Figure S1A), generally, SINTAX had the highest classification power (Fig. 2a-b, d). ITS1-plant (Fig. 2c) and ITS2-plant (Fig. 2d) [10] datasets generated a greater number of unidentified OTUs by all three of the classifiers when compared with the soil datasets (Fig. 2, Additional file 4: Figure S1). A larger number of identified OTUs were detected for the ITS1-BC and ITS2-BC datasets when OTUs were generated by denoising (Additional file 4: Figure S1A-B) instead of clustering (Additional file 4: Figure S1C-D), at all levels except Species. Moreover, a similar pattern was observed with the ITS1-BC and ITS2-BC datasets, more assigned OTUs were observed for ITS2-BC in comparison to ITS1-BC, but not at every rank level (Additional file 4: Figure S1).

Fig. 2
figure 2

Power of classifiers. Distribution of classified and unclassified OTUs for each classifier and across taxonomic level. a ITS1-soil dataset from Smith & Peay [36]. b ITS2-soil dataset from Oliver et al. [37]. c ITS1-plant and d ITS2-plant datasets from Angler et al. [10]

Full size image

Depending on the dataset, the number of unidentified OTUs gradually, or sharply, increased at other ranks higher than Kingdom level. Percent improvement of the consensus taxonomy assignments were calculated from maximum and minimum numbers of classifications obtained at any given rank (Table 4). With CONSTAX, there was ~1% mean improvement at Kingdom level when the consensus taxonomy was compared with an individual classifier program. At other rank levels, there was 7–35% mean improvement. For ITS2 datasets, there was a 1–61% percent improvement at Family level (Table 4). For ITS1 datasets there was a 1 to 59% improvement at Family level (Table 4). At Species level there was a 35% mean improvement across all datasets (Table 4). The higher end of these ranges is due to poor classification of OTUs, especially ITS2 OTUs, using UTAX. If the percent improvement is recalculated without UTAX the maximum percent improvement drops from 98% to 52% (Table 4).

Table 4 Range of percent improvement using CONSTAX
Full size table

Consistency of classifiers

Generally, all the classifiers were consistent in OTU assignments. Based on the consensus taxonomy tables, no bias was observed toward a fungal lineage from any of the classifiers. Nearly all OTUs were identified at Kingdom level (Table 5, Additional file 5: Table S1). There were few examples across the datasets where a single OTU was placed into a unique lineage by one or more of the classifiers. Only 1.24% ∓ 0.006 (st. dev.) of OTUs were differentially assigned across the datasets. This differential assignment phenomenon was most frequently observed at Kingdom level where OTUs were placed with low confidence into either Kingdom Fungi or Protista (Table 5). These OTUs were rarely assigned at a higher level after Kingdom, and never higher than Class; they may be novel sequences, PCR, or sequencing errors. Across all datasets used in the present study (4000 OTUs/ESVs), there were two examples of OTUs assigned to unique fungal lineages. These were found only in ITS1-BC and ITS2-BC datasets (Table 5). The ITS1-BC OTU diverged at Class; the OTU was assigned to Eurotiomycetes and Sordariomycetes by RDPC and UTAX, respectively, and unidentified by SINTAX. This OTU did not have an assignment lower than family. The assignment of the ITS2-BC OTU diverged at Phylum; RDPC and SINTAX placed the ITS2-BC OTU into the Basidiomycota, and UTAX placed this OTU in the Ascomycota. The assignment diverged again at Class, where it was placed into the Pucciniomycetes by RDPC, and the Agaricomycetes by SINTAX.

Table 5 Distribution of identically classified, uniquely classified, and unidentified OTUs across all taxonomic ranks for data presented in Fig. 2
Full size table

Python tool outputs

CONSTAX is implemented in Python and provided as a Bourne Shell executable, constax.sh. After installation of the required dependencies, the user must modify paths and parameters in constax.sh and the config file, both of which can be found in CONSTAX.tar.gz (Additional file 2). The Python scripts called by constax.sh are provided independently and can be easily modified for use with other classifiers or reference databases. After implementation of constax.sh, filtered versions of all taxonomy tables for the given cutoff are generated, alongside the four main output files: i) consensus_taxonomy.txt, the final higher power taxonomy table; ii) combined_taxonomy.txt, which is a large table of all three taxonomy tables side-by-side in addition to the consensus taxonomy; iii) otu_taxonomy_CountClassified.txt, which details assigned and unidentified OTUs at each rank level; and iv) Classification_Summary.txt, which lists the total counts of all unique taxa at a given rank level.

Discussion

Factors that influence the composition and structure of microbial communities are mainly confined to three different groups: sample origin (e.g., soil or water), laboratory methods (e.g., primer selection, PCR conditions, library preparation), and post-sequencing bioinformatic analysis. Since there are sample or methodological challenges at several steps of microbial community studies that can ultimately influence taxonomic classification; we standardize and improve the taxonomic classification step of fungal microbiome studies with CONSTAX. CONSTAX improves taxonomy assignment of fungal OTUs regardless of the strategies researchers choose to reduce the sample or methodological challenges. Linking OTUs to functionally informative names, which largely requires genus- or species-level resolution, is key to addressing biological and ecological hypotheses in fungal community studies. Considerable time should be invested into choosing optimal tools for taxonomic analysis. In this study, eight fungal amplicon datasets were assigned taxonomy using the same reference database [23] and three taxonomy assignment programs were compared: RDPC [18, 32], UTAX [19, 33], and SINTAX [20]. The taxonomic classification step is arguably one of the most delicate steps of the pipeline for amplicon-based microbial ecology studies, because taxon names are largely the basis by which scientists attach biological interpretation to the data. Our results showed minor differences across taxonomic classification approaches using thresholds chosen a priori.

The UTAX classifier generated greater numbers of unidentified OTUs compared with RDPC and SINTAX, a pattern that is pronounced in the ITS2 dataset. We also found more non-fungal OTUs were recovered from the ITS2 sequences; indicating primers for this region may be less fungi-specific than those used for amplifying the ITS1 region. The ITS1 region has been shown to be more conserved in sequence and length for most fungal lineages compared with ITS2 [38,39,40]. Whether the ITS1 or ITS2 region provides the best taxonomic resolution has been investigated previously with Sanger sequence data [3, 37] and pyrosequence data [9, 41]. Apart from the small bias of ITS1 against early diverging fungi, these regions yield similar profiles of fungal communities and either region is considered suitable for community studies. Regardless of primer choice, we showed that use of multiple taxonomy assignment algorithms resulted in consistent classifications when an appropriate OTU-clustering threshold level is used.

Our tool, CONSTAX, implements the following best practice tips for taxonomy assignment of ITS datasets: i) Use more than one classifier program, as not one is clearly superior to others; ii) Obtain a consensus taxonomy after running multiple classifiers; iii) Use the most recent release of software. The classifier programs tested here differ slightly in power, so performing taxonomic classifications with multiple programs, and combining the results will result in a stronger assignment with higher resolution.

When designing experiments, it behooves researchers to carefully consider their target organisms when choosing the ITS barcode region and selecting primers. When investigating broad patterns of fungi, use of ITS alone should be sufficient, but if there is interest in a specific group of fungi, additional markers for those lineages (such as 18S rRNA gene for arbuscular mycorrhizal fungi) may be needed [42]. Further, there are limitations in making functional inferences from fungal ITS amplicon data. If the research questions are aimed at specific species or functions, metagenomics may be a more appropriate approach than amplicon-based community analyses.

Conclusion

We provide a tool, CONSTAX, for generating consensus taxonomy of targeted amplicon sequence data, and demonstrate that it improves taxonomy assignments of environmental OTUs. Taxonomic assignment will improve as database completeness improves, especially the RDPC, since that algorithm functions best when there are multiple representatives for a group (genus or species). The mycological community should continue to generate high quality ITS reference sequences for their research organisms and from Herbarium specimens, which will further enhance the performance of taxonomy assignment algorithms.