Estimation of Differential Entropy for Positive Random Variables and Its Application in Computational Neuroscience

Summary

We use the differential entropy concept and methods related to differential entropy estimation in this chapter. In the beginning, we define the basic terms: entropy, differential entropy, the Kullback–Leibler distance and the refractory periods. We show relations between differential entropy and the Kullback–Leibler distance as well.

Hereafter a detailed description of the methods used is given. These methods can be divided into three groups: parametric methods of entropy estimation, “plug-in” entropy estimators based on nonparametric density estimation and direct entropy estimators. The formulas for direct entropy estimation based on the first four sample moments are introduced.

The results are illustrated by comparing the methods of the entropy estimation, combined with two refractory period estimates. We compare the estimates based on the histogram, the kernel density estimator, the sample spacing method, Vasicek’s method, the nearest neighbor distance method and the methods based on sample moments.