Impact of Training Dataset Size on Neural Answer Selection Models

  • Trond LinjordetEmail author
  • Krisztian Balog
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11437)


It is held as a truism that deep neural networks require large datasets to train effective models. However, large datasets, especially with high-quality labels, can be expensive to obtain. This study sets out to investigate (i) how large a dataset must be to train well-performing models, and (ii) what impact can be shown from fractional changes to the dataset size. A practical method to investigate these questions is to train a collection of deep neural answer selection models using fractional subsets of varying sizes of an initial dataset. We observe that dataset size has a conspicuous lack of effect on the training of some of these models, bringing the underlying algorithms into question.


  1. 1.
    Arpit, D., et al.: A closer look at memorization in deep networks. In: Proceedings of ICML 2017, pp. 233–242 (2017)Google Scholar
  2. 2.
    Cho, J., Lee, K., Shin, E., Choy, G., Do, S.: How much data is needed to train a medical image deep learning system to achieve necessary high accuracy? ArXiv e-prints (2015)Google Scholar
  3. 3.
    Fan, Y., Pang, L., Hou, J., Guo, J., Lan, Y., Cheng, X.: MatchZoo: A Toolkit for Deep Text Matching. ArXiv e-prints, July 2017Google Scholar
  4. 4.
    Guo, J., Fan, Y., Ai, Q., Croft, W.B.: A deep relevance matching model for ad-hoc retrieval. In: Proceedings of CIKM 2016, pp. 55–64 (2016)Google Scholar
  5. 5.
    Hestness, J., et al.: Deep Learning Scaling is Predictable. Empirically, ArXiv e-prints, December 2017Google Scholar
  6. 6.
    Hu, B., Lu, Z., Li, H., Chen, Q.: Convolutional neural network architectures for matching natural language sentences. In: Proceedings of NIPS 2014, pp. 2042–2050 (2014)Google Scholar
  7. 7.
    Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., Heck, L.: Learning deep structured semantic models for web search using clickthrough data. In: Proceedings of CIKM 2013, pp. 2333–2338 (2013)Google Scholar
  8. 8.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436 (2015)CrossRefGoogle Scholar
  9. 9.
    Mitra, B., Craswell, N.: Neural Models for Information Retrieval. ArXiv e-prints, May 2017Google Scholar
  10. 10.
    Mitra, B., Diaz, F., Craswell, N.: Learning to match using local and distributed representations of text for web search. In: Proceedings of WWW 2017, pp. 1291–1299 (2017)Google Scholar
  11. 11.
    Onal, K.D., et al.: Neural information retrieval: at the end of the early years. Inf. Retrieval J. 21, 111–182 (2017)CrossRefGoogle Scholar
  12. 12.
    Pang, L., Lan, Y., Guo, J., Xu, J., Wan, S., Cheng, X.: Text matching as image recognition. In: Proceedings of AAAI 2016, pp. 2793–2799 (2016)Google Scholar
  13. 13.
    Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Networks 61, 85–117 (2015)CrossRefGoogle Scholar
  14. 14.
    Shen, Y., He, X., Gao, J., Deng, L., Mesnil, G.: Learning semantic representations using convolutional neural networks for web search. In: Proceedings of WWW 2014, pp. 373–374 (2014)Google Scholar
  15. 15.
    Sun, C., Shrivastava, A., Singh, S., Gupta, A.: Revisiting unreasonable effectiveness of data in deep learning era. In: Proceedings of ICCV 2017, pp. 843–852 (2017)Google Scholar
  16. 16.
    Wan, S., Lan, Y., Guo, J., Xu, J., Pang, L., Cheng, X.: A deep architecture for semantic matching with multiple positional sentence representations. In: Proceedings of AAAI 2016, pp. 2835–2841 (2016)Google Scholar
  17. 17.
    Yang, L., Ai, Q., Guo, J., Croft, W.B.: aNMM: ranking short answer texts with attention-based neural matching model. In: Proceedings of CIKM 2016, pp. 287–296 (2016)Google Scholar
  18. 18.
    Yang, Y., Yih, S.W.-T., Meek, C.: WikiQA: a challenge dataset for open-domain question answering. In: Proceedings of EMNLP 2015, pp. 2013–2018 (2015)Google Scholar
  19. 19.
    Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.: Understanding deep learning requires rethinking generalization. In: Proceedings of ICLR 2017, pp. 1–2 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of StavangerStavangerNorway

Personalised recommendations