Information Theory and Statistical Mechanics

Part of the Scientific Computation book series (SCIENTCOMP)


In this chapter, we will discuss some of the elements of the information theory measures. In particular, we will introduce the so-called Shannon and relative entropy of a discrete random process and Markov process. Then, we will discuss the relationship between the entropy using the thermodynamic view and information theory view.


  1. Abarbanel, H.D.I.: Analysis of Observed Chaotic Data. Springer, New York (1996)CrossRefzbMATHGoogle Scholar
  2. Abarbanel, H.D.I., Kennel, M.B.: Local false nearest neighbors and dynamical dimensions from observed chaotic data. Phys. Rev. E 47(5), 3057–3068 (1993)CrossRefADSGoogle Scholar
  3. Arkhipov, A., Yin, Y., Schulten, K.: Four-scale description of membrane sculpting by BARdomains. Biophys. J. 95, 2806 (2008)CrossRefADSGoogle Scholar
  4. Bahar, I., Jernigan, R.L.: Inter-residue potentials in globular proteins and the dominance of highly specific hydrophilic interactions at close separation. J. Mol. Biol. 266, 195–214 (1997)CrossRefGoogle Scholar
  5. Bandt, C., Pompe, B.: Permutation entropy: a natural complexity measure for time series. Phys. Rev. Lett. 88, 174102 (2002)CrossRefADSGoogle Scholar
  6. Bonanno, C., Mega, M.: Toward a dynamical model for prime numbers. Chaos Solitons Fractals 20, 107–118 (2004)CrossRefADSMathSciNetzbMATHGoogle Scholar
  7. Canutescu, A.A., Shelenkov, A.A., Dunbrack, R.L. Jr.: A graph-theory algorithm for rapid protein side-chain prediction. Protein Sci. 12, 2001–2014 (2003)CrossRefGoogle Scholar
  8. Cellucci, C.J., Albano, A.M., Rapp, P.E.: Comparative study of embedding methods. Phys. Rev. E 67, 066210–066213 (2003)CrossRefADSMathSciNetGoogle Scholar
  9. Cellucci, C.J., Albano, A.M., Rapp, P.E.: Statistical validation of mutual information calculations: comparison of alternative numerical algorithms. Phys. Rev. E 71, 066208–066214 (2005)CrossRefADSGoogle Scholar
  10. Dama, J.F., Sinitskiy, A.V., McCullagh, M., Weare, J., Roux, B., Dinner, A.R., Voth, G.A.: J. Chem. Theory Comput. 9, 2466 (2013)CrossRefGoogle Scholar
  11. Gay, J.G., Berne, B.J.: Modification of the overlap potential to mimic a linear site-site potential. J. Chem. Phys. 74, 3316 (1981)CrossRefADSGoogle Scholar
  12. Gohlke, H., Thorpe, M.F.: A natural coarse graining for simulating large biomolecular motion. Biophys. J. 91, 2115–2120 (2006)CrossRefADSGoogle Scholar
  13. Goldstein, H.: Classical Mechanics, 2nd edn. Addison-Wesley, San Francisco (2002)zbMATHGoogle Scholar
  14. Gopal, S.M., Mukherjee, S., Cheng, Y.M., Feig, M.: PRIMO/PRIMONA: A coarse-grained model for proteins and nucleic acids that preserves near-atomistic accuracy. Proteins 78, 1266–1281 (2010)CrossRefGoogle Scholar
  15. Gourévitch, B., Eggermont, J.: Evaluating information transfer between auditory cortical neurons. J. Neurophysiol. 97, 2533–2543 (2007)CrossRefGoogle Scholar
  16. Granger, J.: Investigating causal relations by econometric models and crossspectral methods. Acta Physica Polonica B 37, 424–438 (1969)zbMATHGoogle Scholar
  17. Grassberger, P.: Finite sample corrections to entropy and dimension estimates. Phys. Lett. A 128, 369–373 (1988)CrossRefADSMathSciNetGoogle Scholar
  18. Grassberger, P., Procaccia, I.: Measuring the strangeness of strange attractors. Physica D 9, 189 (1983)CrossRefADSMathSciNetzbMATHGoogle Scholar
  19. Irbäck, A., Sjunnesson, F., Wallin, S.: Hydrogen bonds, hydrophobicity forces and the character of the collapse transition. Proc. Natl. Acad. Sci. U.S.A. 97, 13614 (2000)CrossRefADSGoogle Scholar
  20. Joe, H.: Relative entropy measures of multivariate dependence. J. Am. Statist. Assoc. 84, 157–164 (1989)CrossRefMathSciNetzbMATHGoogle Scholar
  21. Kamberaj, H.: A theoretical model for the collective motion of proteins by means of principal component analysis. Cent. Eur. J. Phys. 9(1), 96–109 (2011)Google Scholar
  22. Kamberaj, H.: Faster protein folding using enhanced conformational sampling of molecular dynamics simulation. J. Mol. Graph. Model. 81, 32–49 (2018)CrossRefGoogle Scholar
  23. Kamberaj, H., van der Vaart, A.: Extracting the causality of correlated motions from molecular dynamics simulations. Biophys. J. 97, 1747–1755 (2009a)CrossRefADSGoogle Scholar
  24. Kamberaj, H., van der Vaart, A.: Extracting the causality of correlated motions from molecular dynamic simulations. Biophys. J. 97, 1747–1755 (2009b)CrossRefADSGoogle Scholar
  25. Kennel, M.B., Brown, R., Abarbanel, H.D.I.: Determining embedding dimension for phase-space reconstruction using a geometrical construction. Phys. Rev. A 45, 3403–3411 (1992)CrossRefADSGoogle Scholar
  26. Kraskov, A., Stögbauer, H., Grassberger, P.: Estimating mutual information. Phys. Rev. E 69(6), 066138 (2004)CrossRefADSMathSciNetGoogle Scholar
  27. Kullback, S.: Information Theory and Statistics. Wiley, New York (1959)zbMATHGoogle Scholar
  28. Kullback, S.: The Kullback-Leibler distance. Am. Stat. 41, 340–341 (1987)Google Scholar
  29. Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Stat. 22, 79–86 (1951)CrossRefMathSciNetzbMATHGoogle Scholar
  30. Lehrman, M., Rechester, A.B., White, R.B.: Symbolic analysis of chaotic signals and turbulent fluctuations. Phys. Rev. Lett 78, 54–57 (1997)CrossRefADSGoogle Scholar
  31. Liang, X.S.: The Liang-Kleeman information flow: theory and applications. Entropy 15, 327–360 (2013)CrossRefADSMathSciNetzbMATHGoogle Scholar
  32. Liang, X.S., Kleeman, R.: Information transfer between dynamical system components. Phys. Rev. Lett. 95, 244101 (2005)CrossRefADSGoogle Scholar
  33. Maciejczyk, M., Spasic, A., Liwo, A., Scheraga, H.A.: Coarse grained model of nucleic acid bases. J. Comput. Chem. 31, 1644 (2010)Google Scholar
  34. McCammon, J.A., Northrup, S.H., Karplus, M., Levy, R.M.: Helixcoil transitions in a simple polypeptide model. Biopolymers 19, 2033–2045 (1980)CrossRefGoogle Scholar
  35. Moon, Y.I., Rajagopalam, B., Lall, U.: Estimation of mutual information using kernel density estimators. Phys. Rev. E 52, 2318 (1995)CrossRefADSGoogle Scholar
  36. Murtola, T., Karttunen, M., Vattulainen, I.: Systematic coarse graining from structure using internal states: application to phospholipid/ cholesterol bilayer. J. Chem. Phys. 131, 055101 (2009)CrossRefADSGoogle Scholar
  37. Noakes, L.: The Takens embedding theorem. Int. J. Bifurcation Chaos Appl. Sci. Eng. 1, 867–872 (1991)CrossRefMathSciNetzbMATHGoogle Scholar
  38. Oldziej, S., Liwo, A., Czaplewski, C., Pillardy, J., Scheraga, H.A.: Optimization of the UNRES force field by hierarchical design of the potential-energy landscape. 2. Off-lattice tests of the method with single proteins. J. Phys. Chem. B 108, 16934–16949 (2004)CrossRefGoogle Scholar
  39. Packard, N.H., Crutchfield, J.P., Farmer, J.D., Shaw, R.S.: Geometry from a time series. Phys. Rev. Lett. 45(9), 712–716 (1980)CrossRefADSGoogle Scholar
  40. Potestio, R., Pontiggia, F., Micheletti, C.: Biophys. J. 96, 4993 (2009)CrossRefADSGoogle Scholar
  41. Prokopenko, M., Lizier, J.T.: Transfer entropy and transient limits of computation. Sci. Rep. 4, 5394 (2014)CrossRefADSGoogle Scholar
  42. Prokopenko, M., Lizier, J.T., Price, D.C.: On thermodynamic interpretation of transfer entropy. Entropy 15: 524–543 (2013)CrossRefADSMathSciNetGoogle Scholar
  43. Rechester, A.B., White, R.B.: Symbolic kinetic equations for a chaotic attractor. Phys. Lett. A 156, 419–424 (1991a)CrossRefADSMathSciNetGoogle Scholar
  44. Rechester, A.B., White, R.B.: Symbolic kinetic analysis of two-dimensional maps. Phys. Lett. A 158, 51–56 (1991b)CrossRefADSMathSciNetGoogle Scholar
  45. Rotkiewicz, P., Skolnick, J.: Fast procedure for reconstruction of full-atom protein models from reduced representations. J. Comput. Chem. 29, 1460–1465 (2008)CrossRefGoogle Scholar
  46. Sauer, T., Yorke, J.A., Casdagli, M.: Embedology. J. Stat. Phys. 65, 579–616 (1991)CrossRefADSGoogle Scholar
  47. Schreiber, T.: Measuring information transfer. Phys. Rev. Lett. 85, 461–464 (2000)CrossRefADSGoogle Scholar
  48. Shannon, C.E., Weaver, W.: The Mathematical Theory of Information. University of Illinois Press, Urbana (1949)zbMATHGoogle Scholar
  49. Shi, Q., Izvekov, S., Voth, G.A.: Mixed atomistic and coarse-grained molecular dynamics: simulation of membrane-bound ion channel. J. Phys. Chem. B 110, 15045–15048 (2006)CrossRefGoogle Scholar
  50. Shih, A.Y., Arkhipov, A., Freddolino, P.L., Schulten, K.: Coarse grained protein-lipid model with application to lipidprotein particles. J. Phys. Chem. B 110, 3674–3684 (2006)CrossRefGoogle Scholar
  51. Sinitskiy, A.V., Saunders, M.G., Voth, G.A.: Optimal number of coarsegrained sites in different components of large biomolecular complexes. J. Phys. Chem. B 116, 8363–8374 (2012)CrossRefGoogle Scholar
  52. Smith, A.V., Hall, C.K.: α-helix formation: discontinuous molecular dynamics on an intermediate-resolution protein model Proteins 44, 344–360 (2001a)Google Scholar
  53. Smith, A.V., Hall, C.K.: Assembly of a tetrameric α-helical bundle: computer simulations on an intermediate-resolution protein model. Proteins 44, 376–391 (2001b)CrossRefGoogle Scholar
  54. Staniek, M., Lehnertz, K.: Symbolic transfer entropy. Phys. Rev. Lett. 100, 158101 (2008)CrossRefADSGoogle Scholar
  55. Stepanova, M.: Dynamics of essential collective motions in proteins: theory. Phys. Rev. E 76(5), 051918 (2007)CrossRefADSMathSciNetGoogle Scholar
  56. Takens, F.: Detecting strange attractors in fluid turbulence, Dynamical Systems and Turbulence. Springer, Berlin (1981)zbMATHGoogle Scholar
  57. Thomas, M.C., Joy, A.T.: Elements of Information Theory. Wiley, Hoboken (2006)zbMATHGoogle Scholar
  58. Tozzini, V.: Coarse-grained models for proteins. Curr. Opin. Struct. Bio. 15, 144–150 (2005)CrossRefGoogle Scholar
  59. Tozzini, V., McCammon, J.: A coarse grained model for the dynamics of the early stages of the binding mechanism of HIV-1 protease. Chem. Phys. Lett. 413, 123–128 (2005)CrossRefADSGoogle Scholar
  60. Tozzini, V., Rocchia, W., McCammon, J.A.: Mapping all-atom models onto one-bead coarse-grained models: general properties and applications to a minimal polypeptide model. J. Chem. Theory Comput. 2, 667–673 (2006)CrossRefGoogle Scholar
  61. Tschöp, W., Kremer, K., Hahn, O., Batoulis, J., Bürger, T.: Simulation of polymer melts. II. From coarse-grained models back to atomistic description. Acta Polym. 49, 75–79 (1998)Google Scholar
  62. Tuckerman, M.E., Liu, Y., Ciccotti, G., Martyna, G.J.: Non-Hamiltonian molecular dynamics: generalizing Hamiltonian phase space principles to non-Hamiltonian systems. J. Chem. Phys. 115(4), 1678–1702 (2001)CrossRefADSGoogle Scholar
  63. Ueda, Y., Taketomi, H., Go, N.: Studies on protein folding, unfolding, and fluctuations by computer simulation. II. A. three-dimensional lattice model of lysozyme. Biopolymers 17, 1531–1548 (1978)Google Scholar
  64. Voth, G.A. (ed.): Coarse-Graining of Condensed Phase and Biomolecular Systems. CRC Press, Boca Raton (2008)Google Scholar
  65. Zhang, J., Muthukumar, M.: Simulations of nucleation and elongation of amyloid fibrils. J. Chem. Phys. 130, 035102 (2009)CrossRefADSGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Computer EngineeringInternational Balkan UniversitySkopjeNorth Macedonia
  2. 2.Advanced Computing Research CenterUniversity of New York TiranaTiranaAlbania

Personalised recommendations