The Influence of Entropy on the Classification Performance of a Non-linear Convolutional Neural Network

  • Iryna DzieciuchEmail author
  • Daniel Gebhardt
Conference paper
Part of the Understanding Complex Systems book series (UCS)


The influence of noise levels on image classification with neural networks has been studied before. However, little is known about how different levels of entropy affect the performance of non-linear systems such as Convolutional Neural Networks (CNN), where the initial and final system states are predetermined and entropy represents a performance function. This study provides understanding on how a CNN system evolves from the original to the final state and explains the sensitive dependence on initial training conditions using the publicly available architecture and the MNIST dataset and also discusses the effects of entropy on side-scan sonar imagery. This paper describes a method of testing the effects of varying degrees of entropy on the performance of a non-linear neural network system. This approach allows the comparison of performance of the “black box” system under four states: (1) original non-altered dataset with minimal interclass variance, when a CNN trained on an original dataset is tested on images with added levels of entropy, (2) when a CNN trained on a dataset with varying levels of entropy and (3) tested to recognize the original labeled class and (4) to recognize the labeled class with varying levels of entropy. The advantage of this approach is that we can trace the performance of a single architecture CNN under varying levels of entropy, we can demonstrate the ability of the system to use noise to learn more abstract and complex features of the input space, and we can discuss the results in the light of a theoretical physical system.


Feature selection convolution neural network (CNN) Entropy Accuracy Information theory Black box Physical system 


  1. 1.
    Entropy. Wikipedia, Wikimedia Foundation (2018). 11 June 2018
  2. 2.
    D.J.C. MacKay, Information Theory, Inference and Learning Algorithms (Cambridge University Press, Cambridge, 2017)Google Scholar
  3. 3.
    I. Dzieciuch et al., Non-Linear Convolutional Neural Network for Automatic Detection of Mine-Like Objects in Sonar Imagery (Springer, Dordrecht, 2016). 28 Aug 2016Google Scholar
  4. 4.
    College Physics Textbook Equity Edition Volume 2 of 3: Chapters 13–24. Google Books.

Copyright information

© This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply 2019

Authors and Affiliations

  1. 1.Space and Naval Warfare Systems Center PacificSan DiegoUSA

Personalised recommendations