# An alternate method of evaluating Lagrange multipliers of MEP

- 313 Downloads

**Part of the following topical collections:**

## Abstract

We present an alternate method for evaluating Lagrange multipliers of probability distributions usually used in maximum entropy principle. The Lagrange multipliers are evaluated using least square method. Both the methods are used to calculate Lagrange multipliers in fitting wind speed data. Kolmogorov–Smirnov test, Chi squared goodness of fit test, and root mean square error indicate the usefulness of this alternate method.

## Keywords

Maximum entropy principle Maximum entropy method Wind data Wind speed Hansweert Least square method Least square fitting## 1 Introduction

Maximum entropy principle has a variety of applications in modeling of stochastic data. These applications involve many branches of natural and social sciences. Pressé et al. [1] reviewed the origins and use of Maximum Entropy Principle in Statistical Physics and beyond in fields such as Biology. Karlin et al. [2] used MEP for constructing equilibria in lattice kinetic equations. Chen and Dai [3] investigated MEP of uncertainty distributions for uncertain variables. Županović et al. [4] took an alternative path to derive Kirchhoff’s loop rule where they employed MEP to show that in network branches currents are distributed to achieve maximum entropy. Alves et al. [5] found a connection between the maximum entropy principle and the Higgs boson mass without introducing any extra assumptions into the Standard Model. Hanel etal. [6] generalized the notion of entropy to complex and non-Ergodic systems and showed that the MEP is a consistent method for such systems. Roux and Weare investigated the consistency of the MEP and restrained-ensemble simulations. They showed that the conditions of the restrained-ensemble simulations were compatible with the maximum entropy principle [7]. Aldana-Bobadilla and Kuri-Morales proposed a method of clustering based upon the MEP and showed their method was more effective than supervised ones [8]. Pontzen and Governato used MEP to derive the phase-space distribution for a halo of dark matter. They used physically motivated constraints and found a good match with three simulations of dark matter [9]. Rahmati, Pourghasemi and Melesse applied random forest and maximum entropy models to map groundwater potential in the Mehran region of Iran and found MEP provided more successful predictions [10]. Ruggeri used non-linear MEP to model a polyatomic gas under changing pressure conditions. He found the framework to be in agreement with the phenomenological Extended Thermodynamics theory [11].

## 2 MEP and new method

- 1.
Define the range (x

_{min}, x_{max}) and step size dx. - 2.
Use some standard functions for \( \phi_{n} \left( x \right). \)

- 3.
Start iterative procedure with some \( \varvec{\lambda}_{\varvec{o}} . \)

- 4.
Calculate following two integrals.

$$ G_{n} \left(\varvec{\lambda}\right) = \smallint \phi_{n} \left( x \right)exp\left[ { - \mathop \sum \limits_{n = 0}^{N} \lambda_{n} \phi_{n} \left( x \right)} \right]dx = \mu_{n} , n = 0, \ldots ,N $$(4)$$ g_{nk} = g_{kn} = \frac{{\partial G_{n} \left(\varvec{\lambda}\right)}}{{\partial \lambda_{k} }} = - \smallint x^{k} x^{n} exp\left[ { - \mathop \sum \limits_{m = 0}^{N} \lambda_{m} x^{m} } \right]dx = - G_{n + k} \left(\varvec{\lambda}\right) $$(5) - 5.
Solve the equation \( \varvec{G\delta } = \varvec{v} \), where \( v = \left[ {\mu_{o} - G_{o} \left( {\lambda^{o} } \right), \mu_{1} - G_{1} \left( {\lambda^{o} } \right), \ldots \ldots , \mu_{N} - G_{N} \left( {\lambda^{o} } \right)} \right]^{t} \) to find \( \varvec{\delta} \)

- 6.
Calculate \( \varvec{\lambda}=\varvec{\lambda}^{0} +\varvec{\delta} \) The process is repeated with this new value as \( \varvec{\lambda}^{0} \) ntil \( \varvec{\delta} \) becomes negligible.

In this procedure, the first \( \varvec{\lambda}_{\varvec{o}} \) is basically an initial guess for the vector \( \varvec{\lambda} \). Then the new values of \( \varvec{\lambda} \) are taken as the new \( \varvec{\lambda}_{\varvec{o}} \cdot\varvec{\delta} \) are uncertainties in \( \varvec{\lambda}_{\varvec{o}} \).

## 3 Least square method

## 4 Wind speed data

Many different statistical distributions have been used to model wind speed data. The most widely used distribution is Weibull distribution [12, 13]. MEP more accurately models wind speed data [14, 15, 16, 17]. Djafari developed a matlab program to model wind power density using maximum entropy distributions [18]. Wind speed data for this study has been downloaded from the site [19]. This site contains wind speed data of various cities of The Netherlands and it has been made available for free for educational and research purposes. As a sample we randomly picked the city, Hansweert and used hourly wind data for the year 2000. It has a longitude of 3.998 and latitude of 51.446. Hansweert is a village in southwest Netherlands with overall pleasant weather. The average temperature in winter is 2 °C, while in summer it is 18 °C. In Hansweert winters are windier.

## 5 Goodness of fit tests

Three different goodness of fit tests have been used to determine how well the datasets fit on the classical distribution of maximum entropy principle using both methods.

## 6 Kolmogorov–Smirnov test

To check the validity of the hypothesis that the given data follows a specific distribution, a test of hypothesis with given level of significance is conducted. If \( KS < KS_{\alpha } \) then the two distributions are similar. At 99% level of significance, \( KS_{99} = \frac{1.63}{\sqrt n } \). For n = 6 and \( \alpha = 99\% \), the critical value is 0.618. The best feature of KS test is that it does not depend on sample size.

## 7 Chi square goodness of fit test

_{i}) and corresponding expected frequency (E

_{i}). The Chi square test statistic is given by

## 8 Root mean square error (RMSE)

## 9 Results and discussion

Goodness of fit statistics of MEP fit and least square fit for the month of January

Cities | Fits | Chi square | Kolmogorov–Smirnov | RMSE |
---|---|---|---|---|

Hansweert | MEP fit | 0.058554 | 0.013204 | 0.029344 |

Least square fit | 3.05E − 16 | 6.70E − 08 | 6.86E − 08 | |

Karachi | MEP fit | 2.48E − 2 | 2.66E − 3 | 3.62E − 4 |

Least square fit | 5.63E − 4 | 6.63E − 4 | 1.263E − 5 |

Month-wise test statistics for goodness of fit for Least Square fit

Month | N | Chi square | Kolmogorov–Smirnov test | RMSE |
---|---|---|---|---|

January | 5 | 3.05E − 16 | 6.70E − 08 | 6.86E − 08 |

February | 5 | 3.05E − 16 | 6.70E − 08 | 6.86E − 08 |

March | 5 | 7.88E − 18 | 1.48E − 08 | 1.26E − 08 |

April | 5 | 1.97E − 17 | 1.85E − 08 | 1.33E − 08 |

May | 5 | 1.52E − 17 | 3.11E − 08 | 1.49E − 08 |

June | 5 | 1.42E − 16 | 3.92E − 08 | 4.43E − 08 |

July | 5 | 3.35E − 17 | 3.23E − 08 | 2.54E − 08 |

August | 5 | 1.69E − 17 | 2.04E − 08 | 1.70E − 08 |

September | 5 | 8.65E − 18 | 1.86E − 08 | 1.17E − 08 |

October | 5 | 4.62E − 16 | 1.35E − 07 | 1.02E − 07 |

November | 6 | 1.44E − 11 | 3.28E − 05 | 1.64E − 05 |

December | 5 | 8.57E − 18 | 3.63E − 08 | 1.70E − 08 |

Values of Lagrange coefficients obtained by Least Square Method

\( \varvec{\lambda}_{0} \) | \( \varvec{\lambda}_{1} \) | \( \varvec{\lambda}_{2} \) | \( \varvec{\lambda}_{3} \) | \( \varvec{\lambda}_{4} \) | \( \varvec{\lambda}_{5} \) | \( \varvec{\lambda}_{6} \) | \( \varvec{\lambda}_{7} \) | |
---|---|---|---|---|---|---|---|---|

Jan | 0.2586 | − 3.7199 | 1.1559 | − 0.1965 | 1.7840E − 02 | − 7.9752E − 04 | 1.3813E − 05 | |

Feb | − 0.0962 | − 2.6092 | 0.7384 | − 0.1229 | 1.0853E − 02 | − 4.6586E − 04 | 7.7139E − 06 | |

Mar | − 2.2530 | − 1.4277 | 0.2821 | − 0.0390 | 3.7522E − 03 | − 1.9167E − 04 | 3.8228E − 06 | |

Apr | − 1.4752 | − 1.4661 | 0.0975 | 0.0120 | − 1.5274E − 03 | 4.6887E − 05 | ||

May | − 8.5501 | 8.9182 | − 4.8420 | 1.0956 | − 1.2571E − 01 | 7.7100E − 03 | − 2.4044E − 04 | 2.9919E − 06 |

Jun | 0.6504 | − 4.0858 | 1.2287 | − 0.2112 | 2.1028E − 02 | − 1.0907E − 03 | 2.2989E + 00 | |

Jul | − 1.2563 | − 2.3945 | 0.5998 | − 0.0938 | 9.9086E − 03 | − 5.8817E − 04 | 1.4429E − 05 | |

Aug | 0.1333 | − 5.0964 | 1.9391 | − 0.3807 | 3.9318E − 02 | − 1.9633E − 03 | 3.7214E − 05 | |

Sep | − 0.5050 | − 2.7926 | 0.6578 | − 0.0698 | 1.2434E − 03 | 3.0950E − 04 | − 1.5337E − 05 | |

Oct | 2.4246 | − 3.9655 | 0.9971 | − 0.1661 | 1.7848E − 02 | − 1.1256E − 03 | 3.7408E − 05 | − 5.0212E − 07 |

Nov | 38.0074 | − 31.1175 | 9.8836 | − 1.6705 | 1.5398E − 01 | − 7.2824E − 03 | 1.3832E − 04 | |

Dec | 3.1023 | − 3.7479 | 0.7224 | − 0.0822 | 5.5612E − 03 | − 1.9647E − 04 | 2.7869E − 06 |

## 10 Conclusion

- (i)
MEP should be used if statistical information of the distribution is needed, and LSM is preferable when fitting of dataset is required.

- (ii)
The results show the evaluation of Lagrange multipliers through LSM is better than that obtained by MEP (see Table 1).

A Python program has been developed that fits various degree polynomials as classical probability distributions and displays the degree for which the errors are least. It also displays the calculated values of Lagrange multipliers. For error calculation Chi square, Kolmogorov–Smirnov and Root Mean Square Error test statistics were used. The alternative method consists of using least square method to find the Lagrange multipliers of maximum entropy principle and has proven to provide a better fit of the dataset, as evidenced by lower values of tests of significance statistics.

## Notes

### Compliance with ethical standards

### Conflict of interest

The authors declare that they have no conflict of interest.

## References

- 1.Pressé S, Ghosh K, Lee J, Dill KA (2013) Principles of maximum entropy and maximum caliber in statistical physics. Rev Modern Phys 3(85):1115CrossRefGoogle Scholar
- 2.Karlin IV, Gorban AN, Succi S, Boffi V (1998) Maximum entropy principle for lattice kinetic equations. Phys Rev Lett 81(1):6CrossRefGoogle Scholar
- 3.Chen X, Dai W Maximum entropy principle for uncertain variables. Int J Fuzzy Syst 13(3):232–6 (2011)Google Scholar
- 4.Županović P, Juretić D, Botrić S. Kirchhoff’s loop law and the maximum entropy production principle. Phys Rev E 70(5):056108 (2004)Google Scholar
- 5.Alves A, Dias AG, da Silva R (2015) Maximum entropy principle and the Higgs boson mass. Phys A Stat Mech Appl 7(1):420Google Scholar
- 6.Hanel R, Thurner S, Gell-Mann M (2014) How multiplicity determines entropy and the derivation of the maximum entropy principle for complex systems. Proc Natl Acad Sci 111:6905–6910CrossRefGoogle Scholar
- 7.Roux B (2013) Weare J On the statistical equivalence of restrained-ensemble simulations with the maximum entropy method. J Chem Phys 138(8):02B616CrossRefGoogle Scholar
- 8.Aldana-Bobadilla E, Kuri-Morales A (2015) A clustering method based on the maximum entropy principle. Entropy 17(1):151–180CrossRefGoogle Scholar
- 9.Pontzen A, Governato F (2013) Conserved actions, maximum entropy and dark matter haloes. Mon Notices R Astronom Soc 1(430):121–133CrossRefGoogle Scholar
- 10.Rahmati O, Pourghasemi HR, Melesse AM (2016) Application of GIS-based data driven random forest and maximum entropy models for groundwater potential mapping: a case study at Mehran Region, Iran. CATENA 137:360–372CrossRefGoogle Scholar
- 11.Ruggeri T (2015) Non-linear maximum entropy principle for a polyatomic gas subject to the dynamic pressure. arXiv preprint arXiv:1504.05857
- 12.Khan JK, Shoaib M, Uddin Z, Siddiqui IA, Aijaz A, Siddiqui AA, Hussain E (2015) Comparison of wind energy potential for coastal locations: Pasni and Gwadar. J Basic Appl Sci 11:211CrossRefGoogle Scholar
- 13.Khan JK, Ahmed F, Uddin Z, Iqbal ST, Jilani SU, Siddiqui AA, Aijaz A (2015) Determination of Weibull parameter by four numerical methods and prediction of wind speed in Jiwani (Balochistan). J Basic Appl Sci 11:62CrossRefGoogle Scholar
- 14.Zhang H, Yu YJ, Liu ZY (2014) Study on the maximum entropy principle applied to the annual wind speed probability distribution: a case study for observations of intertidal zone anemometer towers of Rudong in East China Sea. Appl Energy 114:931–938CrossRefGoogle Scholar
- 15.Sobczyk K, Trcebicki J (1999) Approximate probability distributions for stochastic systems: maximum entropy method. Comput Methods Appl Mech Eng 168(1–4):91–111MathSciNetCrossRefGoogle Scholar
- 16.Chellali F, Khellaf A, Belouchrani A, Khanniche R (2012) A comparison between wind speed distributions derived from the maximum entropy principle and Weibull distribution. Case of study; six regions of Algeria. Renew Sustain Energy Rev 16(1):379–385CrossRefGoogle Scholar
- 17.Gurley KR, Tognarelli MA, Kareem A (1997) Analysis and simulation tools for wind engineering. Probab Eng Mech 12(1):9–31CrossRefGoogle Scholar
- 18.Mohammad-Djafari A (1992) A Matlab program to calculate the maximum entropy distributions. In: Maximum entropy and Bayesian methods (pp. 221–233). Springer, DordrechtGoogle Scholar
- 19.http://projects.knmi.nl/klimatologie/onderzoeksgegevens/potentiele_wind. Accessed July 2018