Network Edge Entropy from Maxwell-Boltzmann Statistics

Abstract

In prior work, we have shown how to compute global network entropy using a heat bath analogy and Maxwell-Boltzmann statistics. In this work, we show how to project out edge-entropy components so that the detailed distribution of entropy across the edges of a network can be computed. This is particularly useful if the analysis of non-homogeneous networks with a strong community as hub structure is being attempted. To commence, we view the normalized Laplacian matrix as the network Hamiltonian operator which specifies a set of energy states with the Laplacian eigenvalues. The network is assumed to be in thermodynamic equilibrium with a heat bath. According to this heat bath analogy, particles can populate the energy levels according to the classical Maxwell-Boltzmann distribution, and this distribution together with the energy states determines thermodynamic variables of the network such as entropy and average energy. We show how the entropy can be decomposed into components arising from individual edges using the eigenvectors of the normalized Laplacian. Compared to previous work based on the von Neumann entropy, this thermodynamic analysis is more effective in characterizing changes of network structure since it better represents the edge entropy variance associated with edges connecting nodes of large degree. Numerical experiments on real-world datasets are presented to evaluate the qualitative and quantitative differences in performance.

1 Introduction

There has been a considerable recent interest in computing the entropy associated with different types of network structure [2, 3, 5]. Network entropy has been extensively used to characterize the salient features of the structure of static and dynamic of network systems arising in biology, physics, and the social sciences [1,2,3]. For example, the von Neumann entropy can be used as an effective characterization of network structure, commencing from a quantum analog in which the Laplacian matrix on graphs [1] plays the role of the density matrix. Further development of this idea has shown the link between the von Neumann entropy and the degree statistics of pairs of nodes forming edges in a network [2], which can be efficiently computed for both directed and undirected graphs [3]. Since the eigenvalues of the density matrix reflect the energy states of a network, this approach is closely related to the heat bath analogy in statistical mechanics. This provides a convenient route to network characterization [3, 5]. By populating the energy states with particles which are in thermal equilibrium with a heat bath, this thermalization, of the occupation statistics for the energy states can be computed using the Maxwell-Boltzmann distribution [4, 5]. The properties of this physical heat bath system are described by a partition function with the energy microstates of the network represented by a suitably chosen Hamiltonian. Usually, the Hamiltonian is computed from the adjacency or Laplacian matrix of the network, but recently, Ye et al. [4], have shown how the partition function can be computed from a characteristic matrix polynomial instead.

Although entropic analysis of the heat bath analogy provides a useful global characterization of network structure, it does not allow the entropy of edge or subnetwork structure to be easily computed. In this paper, we explore a novel edge entropy projection which can be applied to the global network entropy computed from Maxwell-Boltzmann statistics. We use this technique to analyze the distribution of edge entropy within a network and explore how this distribution encodes the intrinsic structural properties of different types of network.

The remainder of the paper is organized as follows. In Sect. 2, we briefly introduce the von Neumann entropy with its approximate degrees of nodes connected by an edge. In Sect. 3, we develop an entropic network characterization from the heat bath analogy and Maxwell-Boltzmann statistics, and then describe our edge entropy projection. In Sect. 4, we undertake experiments to demonstrate the usefulness of this novel method. Finally, in Sect. 5 we conclude our paper with a summary of our contribution and suggestions for future work.

2 Preliminaries

2.1 Von Neumann Entropy

Let G(V, E) be an undirected graph with node set V and edge set $E \subseteq V \times V$, and let ${\vert V \vert }$ represent the total number of nodes on graph G(V, E). The ${\vert V \vert } \times {\vert V \vert }$ adjacency matrix A of a graph is defined as

$$\begin{aligned} A = {\left\{ \begin{array}{ll} 0&{} \text {if} (u,v) \in E \\ 1&{} \text {otherwise.} \end{array}\right. } \end{aligned}$$

(1)

Then the degree of node u is $d_u = \sum _{v \in V} A_{uv}$.

The normalized Laplacian matrix $\tilde{L}$ of the graph G is defined as

$$\begin{aligned} \tilde{L} = D^{-\frac{1}{2}}LD^{\frac{1}{2}} = \varPhi \tilde{\varLambda } \varPhi ^T \end{aligned}$$

(2)

where $L = D - A$ is the Laplacian matrix and D denotes the degree diagonal matrix whose elements are given by $D(u,u) = d_u$ and zeros elsewhere. $\tilde{\varLambda } = diag(\lambda _1, \lambda _2, \ldots \lambda _{\vert V \vert })$ is the diagonal matrix with the ordered eigenvalues as elements and $\varPhi = (\varphi _1, \varphi _2, \ldots , \varphi _{\vert V \vert })$ is the matrix with the ordered eigenvectors as columns.

In quantum mechanics, the density matrix is used to describe a system with the probability of pure quantum states. Passerini and Severini [1] have extended this idea to the graph domain. Specifically, they show that a density matrix for a graph or network can be obtained by scaling the combinatorial Laplacian matrix by the reciprocal of the number of nodes in the graph.

With this notation, the specified density matrix is obtained by scaling the normalized Laplacian matrix by the number of nodes, i.e.

$$\begin{aligned} \varvec{\rho } = \frac{\tilde{L}}{{\vert V \vert }} \end{aligned}$$

(3)

When defined in this way the density matrix is Hermitian i.e. $\varvec{\rho } = \varvec{\rho \dagger }$ and $\varvec{\rho } \ge 0, \text {Tr} \varvec{\rho } = 1$. It plays an important role in the quantum observation process, which can be used to calculate the expectation value of measurable quantity.

The interpretation of the scaled normalized Laplacian as a density operator, opens up the possibility of characterizing a graph using the von Neumann entropy from quantum information theory. The von Neumann entropy is defined as the entropy of the density matrix associated with the state vector of a system. As noted above, Passerini and Severini [1] suggest how the von Neumann entropy can be computed by scaling the normalized discrete Laplacian matrix for a network. As a result the von Neumann entropy is given in terms of the eigenvalues $\lambda _1$, ....., $\lambda _{\vert V \vert }$ of the density matrix $\varvec{\rho }$,

$$\begin{aligned} S^{^{VN}} = - \text {Tr}(\varvec{\rho } \log \varvec{\rho }) = - \sum _{i=1}^{{\vert V \vert }}\frac{\hat{\lambda }_i}{{\vert V \vert }}\log \frac{\hat{\lambda }_i}{{\vert V \vert }} \end{aligned}$$

(4)

The von Neumann entropy [1] computed from the normalized Laplacian spectrum has been shown to be effective for network characterization. In fact, Han et al. [2] have shown how to approximate the calculation of von Neumann entropy in terms of simple degree statistics. Their approximation allows the cubic complexity of computing the von Neumann entropy from the Laplacian spectrum, to be reduced to one of quadratic complexity using simple edge degree statistics, i.e.

$$\begin{aligned} S^{^{VN}} = 1 - \frac{1}{{\vert V \vert }} - \frac{1}{{\vert V \vert }^2}\sum _{(u,v)\in E}\frac{1}{d_u d_v} \end{aligned}$$

(5)

This expression for the von Neumann entropy allows the approximate entropy of the network to be efficiently computed and has been shown to be an effective tool for characterizing structural property of networks, with extremal values for the cycle and fully connected graphs.

Thus, the edge entropy decomposition is given as

$$\begin{aligned} S^{^{VN}}_{_{edge}}(u,v) = \frac{1}{{\vert E \vert }} - \frac{1}{{\vert V \vert }{\vert E \vert }} - \frac{1}{{\vert E \vert } {\vert V \vert }^2}\frac{1}{d_u d_v} \end{aligned}$$

(6)

where $S^{^{VN}} = \sum _{(u,v)\in E} S^{^{VN}}_{_{edge}}(u,v)$. This expression decomposes the global parameter of von Neumann entropy on each edge with the relation to the degrees from the connection of two vertexes.

3 Network Entropy in Maxwell-Boltzmann Statistics

3.1 Thermodynamic Representation

Thermodynamic analogies provide powerful tools for analyzing complex networks. The underpinning idea is that statistical thermodynamics can be combined with network theory to characterize both static and time-evolving networks [6].

Here we consider the thermodynamic system specified by a system of N particles with energy states given by the network Hamiltonian and immersed in a heat bath with temperature T. The ensemble is represented by a partition function $Z(\beta , N)$, where $\beta =1/k_B T$ is an inverse of temperature parameter [5].

When specified in this way, the various thermodynamic characterizations of the network can be computed. For instance, the average energy of the network can be expressed in terms of the density matrix and the Hamiltonian operator,

$$\begin{aligned} \langle U \rangle = \langle H \rangle = \text {Tr} \left( \rho H \right) = \left[ -\frac{\partial }{\partial \beta } \log Z \right] _N \end{aligned}$$

(7)

and the thermodynamic entropy by

$$\begin{aligned} S = k_B \left[ \log Z + \beta \langle U \rangle \right] \end{aligned}$$

(8)

Both the energy and the entropy can be regarded as weighted functions of the Laplacian eigenvalues which characterize the network structure in different ways. In the following sections, we set the Boltzmann constant to the unity, i.e., $k_B=1$, and explore the thermodynamic entropy in more detail to represent the intrinsic structure of networks.

3.2 Maxwell-Boltzmann Statistics

The Maxwell-Boltzmann distribution relates the microscopic properties of particles to the macroscopic thermodynamic properties of matter [4]. It applies to systems consisting of a fixed number of weakly interacting distinguishable particles. These particles occupy the energy levels associated with a Hamiltonian and in our case the Hamiltonian of the network, which is in contact with a thermal bath [7].

Taking the Hamiltonian to be the normalized Laplacian of the network, the canonical partition function for Maxwell-Boltzmann occupation statistics of the energy levels is

$$\begin{aligned} Z_{_{MB}} = \text {Tr} \biggl [\exp (-\beta \tilde{L})^N \biggr ] = \left( \sum _{i=1}^{{\vert V \vert }} e^{-\beta \lambda _i} \right) ^N \end{aligned}$$

(9)

where $\beta =1/k_B T$ is the reciprocal of the temperature T with $k_B$ as the Boltzmann constant; N is the total number of particles and $\lambda _i$ denotes the microscopic energy of system at each microstate i with energy $\lambda _i$. Derived from Eq. (8), the entropy of the system with N particles is

$$\begin{aligned} S_{_{MB}}= & {} \log Z - \beta \frac{\partial \log Z}{\partial \beta } = -N \text {Tr} \biggl \{ \frac{\exp (- \beta \tilde{L} )}{\text {Tr} [\exp (- \beta \tilde{L} )]}\log \frac{ \exp (- \beta \tilde{L} )}{ \text {Tr} [\exp (- \beta \tilde{L} )]} \biggr \} \nonumber \\= & {} -N \sum _{i=1}^{\vert V \vert } {\frac{e^{-\beta \lambda _i}}{\sum _{i=1}^{\vert V \vert } {e^{-\beta \lambda _i}}} \log {\frac{e^{-\beta \lambda _i}}{\sum _{i=1}^{\vert V \vert } {e^{-\beta \lambda _i}}}}} \end{aligned}$$

(10)

For a single particle, the density matrix is

$$\begin{aligned} \varvec{\rho }_{_{MB}} = \frac{\exp (-\beta \tilde{L})}{\text {Tr}[\exp (-\beta \tilde{L})]} \end{aligned}$$

(11)

Since the density matrix commutes with the Hamiltonian operator, we have $\partial \rho /\partial t=0$ and the system can be viewed as in equilibrium. So the entropy in the Maxwell-Boltzmann system is simply N times the von Neumann entropy of a single particle, as we might expect.

3.3 Edge Entropy Analysis

Our goal is to project the global network entropy onto the edges of the network. In matrix form for Maxwell-Boltzmann statistics in Eq. (10), the entropy can be written as,

$$\begin{aligned} S^{^{MB}} = -\text {Tr} \left[ \varvec{\rho }_{_{MB}} \log \varvec{\rho }_{_{MB}} \right] = -\text {Tr} [\mathbf {\Sigma }_{_{MB}}] \end{aligned}$$

(12)

Since the spectral decomposition of the normalized Laplacian matrix is

$$\begin{aligned} \tilde{L} = \varPhi \tilde{\varLambda } \varPhi ^T \end{aligned}$$

(13)

We can decompose the matrix $\mathbf {\Sigma }_{_{MB}}$ as follows

$$\begin{aligned} \mathbf {\Sigma }_{_{MB}} = \varPhi \sigma _{_{MB}}(\tilde{\varLambda }) \varPhi ^T \end{aligned}$$

(14)

where

$$\sigma _{_{MB}}(\lambda _i) = -N {\frac{e^{-\beta \lambda _i}}{\sum _{i=1}^{\vert V \vert } {e^{-\beta \lambda _i}}} \log {\frac{e^{-\beta \lambda _i}}{\sum _{i=1}^{\vert V \vert } {e^{-\beta \lambda _i}}}}}$$

for Maxwell-Boltzmann statistics. As a result, we can perform edge entropy projection of the Maxwell-Boltzmann statistical model using the Laplacian eigenvectors, with the result that the entropy of edge (uv) is given as,

$$\begin{aligned} S^{^{MB}}_{_{edge}}(u,v) = \sum _{i=1}^{{\vert V \vert }} \sigma _{_{MB}}(\lambda _i) \varphi _i \varphi _i^T \end{aligned}$$

(15)

Thus, the global entropy can be projected on the edges of the network system. This provides useful measures for local entropic characterization of network structure in a relatively straightforward manner.

4 Experiments and Evaluations

4.1 Data Sets

Data-Set 1: The PPIs dataset extracted from STRING–8.2 [8] consisting of networks which describe the interaction relationships between histidine kinase and other proteins. There are 173 PPIs in this dataset and they are collected from 4 different kinds of bacteria with the following evolution order (from older to more recent). Aquifex and Thermotoga-8 PPIs from Aquifex aelicus and Thermotoga maritima, Gram-Positive-52 PPIs from Staphylococcus aureus, Cyanobacteria-73 PPIs from Anabaena variabilis and Proteobacteria-40 PPIs from Acidovorax avenae [9].

Data-Set 2: The New York Stock Exchange dataset consists of the daily prices of 3,799 stocks traded continuously on the New York Stock Exchange over 6000 trading days. The stock prices were obtained from the Yahoo! financial database (http://finance.yahoo.com) [10]. A total of 347 stock were selected from this set, for which historical stock prices from January 1986 to February 2011 are available. In our network representation, the nodes correspond to stock and the edges indicate that there is a statistical similarity between the time series associated with the stock closing prices [10]. To determine the edge structure of the network, we use a time window of 20 days to compute the cross-correlation coefficients between the time-series for each pair of stock. Connections are created between a pair of stock if the cross-correlation exceeds an empirically determined threshold. In our experiments, we set the correlation coefficient threshold to the value to $\xi = 0.85$. This yields a time-varying stock market network with a fixed number of 347 nodes and varying edge structure for each of 6,000 trading days. The edges of the network, therefore, represent how the closing prices of the stock follow each other.

4.2 Experimental Results

We first investigate the temperature dependence of edge entropy for the PPI networks. We select three different types of edges with different values of degrees at the vertices and explore how the entropy changes with temperature.

Fig. 1.

(Color online) (a) Edge entropy with a different degree on both nodes for Maxwell-Boltzmann statistics. The red line represents the high-degree edge; the blue line is the low-degree edge and the black line is the median value of degree on the edge ends; (b) Scatter plot of edge entropies from Maxwell-Boltzmann vs. von Neumann entropy with different value of temperatures.

Figure 1(a) plots three selected edge entropies versus temperature with Maxwell-Boltzmann occupation statistics. The three edges show a similar dependence of entropy on the temperature. As the inverse of temperature ($\beta$) increases, the edge entropy reaches a maximum value. The edge entropy for vertices with the high degree increases faster than that for the low-degree in the high-temperature region. In the low-temperature limit, entropy approaches zero. This is because when the temperature decreases the configuration of particle occupation becomes identical as the particles always state at the low energy levels since the thermalization effects vanish.

Figure 1(b) shows the relationship between the edge entropies in the Maxwell-Boltzmann and von Neumann cases. There is a transition in the relationship between two entropies with temperature. At high temperature (i.e., $\beta = 0.1$), the Maxwell-Boltzmann entropy is roughly in linear proportion to von Neumann entropy. However, as the temperature reduces, it takes on an approximately exponential dependence. The Maxwell-Boltzmann edge entropy decreases monotonically with the von Neumann edge entropy in the low-temperature region ($\beta = 10$).

Fig. 2.

(Color online) 3D scatter plot of edge entropy from Maxwell-Boltzmann statistics and von Neumann entropy. (a) Edge entropy in Maxwell-Boltzmann statistics. (b) Edge entropy from von Neumann formula. (c) The comparison of edge entropy between Maxwell-Boltzmann statistics and von Neumann entropy.

Further exploration of the relationship between Maxwell-Boltzmann edge entropy and von Neumann entropy is shown in Fig. 2, which shows the 3D plots of edge entropy with the vertex degree. The figure compares the edge entropy between Maxwell-Boltzmann statistics and von Neumann entropy with node degree connection for each edge in the network. The observation is that both entropies have a similar tendency with the degrees at the end. The Maxwell-Boltzmann edge entropy is more sensitive to the degree variance than the von Neumann entropy in the high degree region. The reason for this is the constant term in the von Neumann entropy formula dominates the value of edge entropy when the degrees are large. Thus, the Maxwell-Boltzmann edge entropy is better suited to represent the differences in graph structure associated with large degree nodes.

When compared to the von Neumann edge entropy, the Maxwell-Boltzmann edge entropy is distributed rather differently. Figure 3 shows two examples of PPI networks, namely Anabaena variabilis and Aquifex aelicus together with their associated edge entropy histograms. The Maxwell-Boltzmann edge entropies are more sensitive to the presence of edges associated with high degree nodes, which provides better edge discrimination. This effect is manifest in the differences of edge entropy histograms. In the Maxwell-Boltzmann case, the histogram shows two peaks in the edge entropy distribution, while the von Neumann edge entropy is concentrated at low values and has just a single peak. In other words, the von Neumann edge entropy offers less salient structure.

Fig. 3.

(Color online) Examples of protein-protein interaction networks with the edge entropy distribution from von Neumann entropy and Maxwell-Boltzmann statistics.

Fig. 4.

(Color online) Entropy from Maxwell-Boltzmann statistics and von Neumann entropy for NYSE (1987–2011). Number of particle is $N = 1$ and temperature is $\beta = 10$.

Next, we turn our attention to the time evolution of networks. We take the NYSE network as an example to explore the entropic characterization in the network structure. Figure 4 plots the total network for the Maxwell-Boltzmann and von Neumann cases. Both entropies reflect the positions of significant global financial events such as Black Monday, Friday 13th mini-crash, Early 1990s Recession, 1997 Asian Crisis, 9.11 Attacks, Downturn of 2002–2003, 2007 Financial Crisis, the Bankruptcy of Lehman Brothers and the European Debt Crisis. In each case, the entropy undergoes significant fluctuations during the financial crises, associated with dramatic structural changes. Compared to the von Neumann entropy, the Maxwell-Boltzmann case is more sensitive to fluctuations in the network structure. A good example is Black Wednesday in 1992, which is obvious in the Maxwell-Boltzmann entropy but is not clear in the von Neumann case.

We now focus in detail on one critical financial event, i.e., Black Monday in October 1987, to explore the dynamic structural difference with the entropic variance. We visualize the network structure at three-time epochs, i.e., before, during and after Black Monday, and compare the Maxwell-Boltzmann with von Neumann edge entropy. Figure 5 shows the network structure and edge entropy distribution during the crisis. Before Black Monday, the stocks are highly connected with a large number of densely connected clusters of stocks following the same trading trends. This feature is also reflected in the Maxwell-Boltzmann edge entropy distribution. However, during Black Monday, the number of connections between stock decrease significantly with large numbers of nodes becoming disconnected. Some stocks do though slightly increase their number of links with other stocks. This manifests itself as a shift of the peak to the high entropy region of the distribution. After Black Monday, the stocks begin to recover connections with another. The node degree distribution also returns to its previous shape. In contrast, the von Neuman edge entropy distribution does not completely reflect the details of these critical structural changes. Compared to the Maxwell-Boltzmann edge entropy, the distribution of von Neumann edge entropy does not change significantly during Black Monday and hence does not effectively characterize the dynamic structure on the network.

Fig. 5.

(Color online) Visualization of network structure before, during and after Black Monday. The edge entropy distribution is computed from von Neumann entropy and Maxwell-Boltzmann statistics. The statistical model such as Maxwell-Boltzmann case is more sensitive to represent the dynamic structure in the networks.

In conclusion, both the Maxwell-Boltzmann and von Neumann edge entropies can be used to represent changes in network structure. Compared to the von Neumann edge entropy, the Maxwell-Boltzmann edge entropy is more sensitive to variance associated with the degree distribution. In the low-temperature region, the Maxwell-Boltzmann edge-entropy has similar degree sensitivity to the von Neumann edge entropy. However, it is more sensitive to high degree variations.

5 Conclusion

This paper has explored the thermodynamic characterizations of networks resulting from Maxwell-Boltzmann statistics, and specifically those associated with the thermalization effects of the heat bath on the occupation of the normalized Laplacian energy states. We view the normalized Laplacian matrix as the Hamiltonian operator of the network with associated energy states which can be occupied by classical distinguishable particles. This extends the use of entropy as a tool to characterize network structures in both static and time series data. To compare with the extensively studied von Neuman entropy, we conduct the experiments which demonstrate that the thermodynamic edge entropy is better suited to represent the intrinsic structural properties associated to long-tailed degree distributions. Future work will focus on exploring non-classical alternatives to the Maxwell-Boltzmann occupation statistics and the detailed distribution of the entropic characterization for different types of complex networks.