Skip to main content
SpringerLink
Log in
SpringerLink
Log in

COBS: A Compact Bit-Sliced Signature Index

  • Conference paper
  • First Online:
String Processing and Information Retrieval (SPIRE 2019)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11811))

Included in the following conference series:

Abstract

We present COBS, a COmpact Bit-sliced Signature index, which is a cross-over between an inverted index and Bloom filters. Our target application is to index k-mers of DNA samples or q-grams from text documents and process approximate pattern matching queries on the corpus with a user-chosen coverage threshold. Query results may contain a number of false positives which decreases exponentially with the query length. We compare COBS to seven other index software packages on 100 000 microbial DNA samples. COBS’ compact but simple data structure outperforms the other indexes in construction time and query performance with Mantis by Pandey et al. in second place. However, unlike Mantis and other previous work, COBS does not need the complete index in RAM and is thus designed to scale to larger document sets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Almodaresi, F., Pandey, P., Patro, R.: Rainbowfish: a succinct colored de Bruijn graph representation. In: 17th International Workshop on Algorithms in Bioinformatics (WABI). LIPIcs, vol. 88, pp. 18:1–18:15. Schloss Dagstuhl, August 2017. preprint bioRxiv:138016

    Google Scholar 

  2. Almodaresi, F., Sarkar, H., Srivastava, A., Patro, R.: A space and time-efficient index for the compacted colored de Bruijn graph. Bioinformatics 34(13), i169–i177 (2018)

    Article  Google Scholar 

  3. Bingmann, T.: NVMe “disk” bandwidth and latency for batched block requests, March 2019. Online Article, http://panthema.net/2019/0322-nvme-batched-block-access-speed

  4. Bloom, B.H.: Space/time trade-offs in hash coding with allowable errors. Commun. ACM 13(7), 422–426 (1970)

    Article  Google Scholar 

  5. Bradley, P., den Bakker, H.C., Rocha, E.P.C., McVean, G., Iqbal, Z.: Ultrafast search of all deposited bacterial and viral genomic data. Nat. Biotechnol. 37, 152–159 (2019)

    Article  Google Scholar 

  6. Brin, S., Page, L.: The anatomy of a large-scale hypertextual web search engine. Comput. Networks ISDN Syst. 30(1–7), 107–117 (1998)

    Article  Google Scholar 

  7. Broder, A.Z., Mitzenmacher, M.: Network applications of Bloom filters: a survey. Internet Math. 1(4), 485–509 (2003)

    Article  MathSciNet  Google Scholar 

  8. Chikhi, R., Holub, J., Medvedev, P.: Data structures to represent sets of \(k\)-long DNA sequences. Computing Research Repository (CoRR), arXiv:1903.12312:1–16, March 2019

  9. Collet, Y.: xxHash: extremely fast non-cryptographic hash algorithm, 2014. Git repository. https://github.com/Cyan4973/xxHash. Accessed July 2019

  10. Cook, C.E., Lopez, R., Stroe, O., Cochrane, G., Brooksbank, C., Birney, E., Apweiler, R.: The European Bioinformatics Institute in 2018: tools, infrastructure and training. Nucleic Acids Res. 47(D1), D15–D22 (2019)

    Article  Google Scholar 

  11. Crainiceanu, A., Lemire, D.: Bloofi: multidimensional bloom filters. Inf. Syst. 54, 311–324 (2015)

    Article  Google Scholar 

  12. Faloutsos, C., Christodoulakis, S.: Signature files: an access method for documents and its analytical performance evaluation. ACM Trans. Inf. Syst. (TOIS) 2(4), 267–288 (1984)

    Article  Google Scholar 

  13. Gauger, F.: Engineering a compact bit-sliced signature index for approximate search on genomic data. Master Thesis. Karlsruhe Institute of Technology, Germany, February 2018

    Google Scholar 

  14. Gog, S., Beller, T., Moffat, A., Petri, M.: From theory to practice: plug and play with succinct data structures. In: Gudmundsson, J., Katajainen, J. (eds.) SEA 2014. LNCS, vol. 8504, pp. 326–337. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07959-2_28

    Chapter  Google Scholar 

  15. Goodwin, B., et al.: BitFunnel: revisiting signatures for search. In: 40th ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 605–614. ACM, August 2017

    Google Scholar 

  16. Harris, R.S., Medvedev, P.: Improved representation of sequence Bloom trees. bioRxiv, pp. 501452, December 2018

    Google Scholar 

  17. Harrison, P.W., et al.: The european nucleotide archive in 2018. Nucleic Acids Res. D47(1), D84–D88 (2019)

    Article  Google Scholar 

  18. Heinz, S., Zobel, J., Williams, H.E.: Burst tries: a fast, efficient data structure for string keys. ACM Trans. Inf. Syst. (TOIS) 20(2), 192–223 (2002)

    Article  Google Scholar 

  19. Holley, G., Wittler, R., Stoye, J.: Bloom filter trie: an alignment-free and reference-free data structure for pan-genome storage. Algorithms Mol. Biol. 11(1), 3 (2016)

    Article  Google Scholar 

  20. Iqbal, Z., Caccamo, M., Turner, I., Flicek, P., McVean, G.: De novo assembly and genotyping of variants using colored de Bruijn graphs. Nat. Genet. 44(2), 226 (2012)

    Article  Google Scholar 

  21. Iqbal, Z., Turner, I., McVean, G.: High-throughput microbial population genomics using the cortex variation assembler. Bioinformatics 29(2), 275–276 (2012)

    Article  Google Scholar 

  22. Krugel, J.: Approximate Pattern Matching with Index Structures. Ph.D. thesis, Technische Universität München, Germany, February 2016

    Google Scholar 

  23. Marçais, G., Kingsford, C.: A fast, lock-free approach for efficient parallel counting of occurrences of \(k\)-mers. Bioinformatics 27(6), 764–770 (2011)

    Google Scholar 

  24. Mitzenmacher, M., Upfal, E.: Probability and Computing: Randomized Algorithms and Probabilistic Analysis. Cambridge University Press, Cambridge (2005)

    Book  Google Scholar 

  25. Mohamadi, H., Khan, H., Birol, I.: ntCard: a streaming algorithm for cardinality estimation in genomics data. Bioinformatics 33(9), 1324–1330 (2017)

    Google Scholar 

  26. Muggli, M.D., et al.: Succinct colored de Bruijn graphs. Bioinformatics 33(20), 3181–3187 (2017). preprint bioRxiv:040071

    Article  Google Scholar 

  27. Navarro, G., Baeza-Yates, R.A., Sutinen, E., Tarhio, J.: Indexing methods for approximate string matching. IEEE Bull. Tech. Committee Data Eng. 24(4), 19–27 (2001)

    Google Scholar 

  28. Pandey, P., Almodaresi, F., Bender, M.A., Ferdman, M., Johnson, R., Patro, R.: Mantis: a fast, small, and exact large-scale sequence-search index. Cell Systems, June 2018. preprint bioRxiv:217372

    Google Scholar 

  29. Pandey, P., Bender, M.A., Johnson, R., Patro, R.: A general-purpose counting filter: making every bit count. In: ACM International Conference on Management of Data, pp. 775–787. ACM (2017)

    Google Scholar 

  30. Pandey, P., Bender, M.A., Johnson, R., Patro, R.: Squeakr: an exact and approximate k-mer counting system. Bioinformatics 34(4), 568–575 (2018). preprint bioRxiv:122077

    Article  Google Scholar 

  31. Raman, R., Raman, V., Srinivasa Rao, S.: Succinct indexable dictionaries with applications to encoding \(k\)-ary trees and multisets. In: 13th ACM-SIAM Symposium on Discrete Algorithms (SODA), pp. 233–242. SIAM, January 2002

    Google Scholar 

  32. Solomon, B., Kingsford, C.: Fast search of thousands of short-read sequencing experiments. Nat. Biotechnol. 34(3), 300–312 (2016)

    Article  Google Scholar 

  33. Solomon, B., Kingsford, C.: Improved search of large transcriptomic sequencing databases using split sequence Bloom trees. J. Comput. Biol. 25(7), 755–765 (2018)

    Article  MathSciNet  Google Scholar 

  34. Sun, C., Harris, R.S., Chikhi, R., Medvedev, P.: AllSome sequence Bloom trees. J. Computat. Biol. 25(5), 467–479 (2018)

    Article  Google Scholar 

  35. Turner, I., Garimella, K.V., Iqbal, Z., McVean, G.: Integrating long-range connectivity information into de Bruijn graphs. Bioinformatics 34(15), 2556–2565 (2018)

    Article  Google Scholar 

  36. Ukkonen, E.: Approximate string-matching with \(q\)-grams and maximal matches. Theoret. Comput. Sci. 92(1), 191–211 (1992)

    Google Scholar 

  37. Wong, H.K.T., Liu, H.-F., Olken, F., Rotem, D., Wong, L.: Bit transposed files. In 11th International Conference on Very Large Data Bases (VLDB), pp. 448–457. VLDB Endowment, August 1985

    Google Scholar 

  38. Ye, Y., Belazzougui, D., Qian, C., Zhang, Q.: Memory-efficient and ultra-fast network lookup and forwarding using othello hashing. IEEE/ACM Trans. Networking 26(3), 1151–1164 (2018)

    Article  Google Scholar 

  39. Ye, Y., et al.: SeqOthello: querying RNA-seq experiments at scale. Genome Biol. 19(1), 167 (2018). preprint bioRxiv:258772

    Article  Google Scholar 

  40. Zobel, J., Moffat, A.: Inverted files for text search engines. ACM Comput. Surveys (CSUR) 38(2), 6 (2006)

    Article  Google Scholar 

  41. Zobel, J., Moffat, A., Ramamohanarao, K.: Inverted files versus signature files for text indexing. ACM Trans. Database Syst. (TODS) 23(4), 453–490 (1998)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Theoretical Informatics, Karlsruhe Institute of Technology, Karlsruhe, Germany

    Timo Bingmann & Florian Gauger

  2. European Molecular Biology Laboratory, European Bioinformatics Institute, Cambridge, UK

    Phelim Bradley & Zamin Iqbal

Authors
  1. Timo Bingmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Phelim Bradley
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Florian Gauger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zamin Iqbal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Timo Bingmann .

Editor information

Editors and Affiliations

  1. University of A Coruña, A Coruña, Spain

    Nieves R. Brisaboa

  2. University of Helsinki, Helsinki, Finland

    Simon J. Puglisi

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bingmann, T., Bradley, P., Gauger, F., Iqbal, Z. (2019). COBS: A Compact Bit-Sliced Signature Index. In: Brisaboa, N., Puglisi, S. (eds) String Processing and Information Retrieval. SPIRE 2019. Lecture Notes in Computer Science(), vol 11811. Springer, Cham. https://doi.org/10.1007/978-3-030-32686-9_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32686-9_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32685-2

  • Online ISBN: 978-3-030-32686-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation