Estimating and Factoring the Dropout Induced Distribution with Gaussian Mixture Model

Abstract

The analytical method to capture the dropout induced distribution of forwarding output in a neural network as Gaussian mixture model (GMM) was proposed. In dropout Bayesian DNN, if the network is dropout-trained and a test data is dropout-forwarded for inference, then its output, usually approximated as a single mode Gaussian, becomes a posterior whose variance tells uncertainty of its inference [1]. Here, the proposed method can capture the arbitrary distribution analytically with high accuracy without Monte Carlo (MC) method for any network equipped with dropout and fully connected (FC) layers. Therefore, it is applicable to the general non-Gaussian posterior case for a better uncertainty estimate. The proposed method also has the advantage to provide a multimodal analysis in distribution by factoring which can be tuned with a user defined expressibility parameter while a MC estimate provides only a “flat” image. This helps to understand how the FC layer tries to code a dropout injected highly multimodal data into a single mode Gaussian while the unknown data becomes a complicated distribution.