Skip to main content
SpringerLink
Log in

Bridging Density Functional Theory and Big Data Analytics with Applications

  • Chapter
  • First Online:
Handbook of Big Data Analytics

Part of the book series: Springer Handbooks of Computational Statistics ((SHCS))

Abstract

The framework of the density functional theory (DFT) reveals both strong suitability and compatibility for investigating large-scale systems in the Big Data regime. By technically mapping the data space into physically meaningful bases, the chapter provides a simple procedure to formulate global Lagrangian and Hamiltonian density functionals to circumvent the emerging challenges on large-scale data analyses. Then, the informative features of mixed datasets and the corresponding clustering morphologies can be visually elucidated by means of the evaluations of global density functionals. Simulation results of data clustering illustrated that the proposed methodology provides an alternative route for analyzing the data characteristics with abundant physical insights. For the comprehensive demonstration in a high dimensional problem without prior ground truth, the developed density functionals were also applied on the post-process of magnetic resonance imaging (MRI) and better tumor recognitions can be achieved on the T1 post-contrast and T2 modes. It is appealing that the post-processing MRI using the proposed DFT-based algorithm would benefit the scientists in the judgment of clinical pathology. Eventually, successful high dimensional data analyses revealed that the proposed DFT-based algorithm has the potential to be used as a framework for investigations of large-scale complex systems and applications of high dimensional biomedical image processing.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 299.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 379.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 379.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Baer R, Livshits E, Salzner U (2010) Tuned range-separated hybrids in density functional theory. Annu Rev Phys Chem 61:85–109

    Article  Google Scholar 

  • Bas E, Erdogmus D (2010) Piecewise linear cylinder models for 3-dimensional axon segmentation in Brainbow imagery. In: International symposium on biomedical imaging (ISBI), pp 1297–1300

    Google Scholar 

  • Bas E, Erdogmus D, Draft RW, Lichtman JW (2012) Local tracing of curvilinear structures in volumetric color images: application to the Brainbow analysis. J Vis Commun Image R 23:1260–1271

    Article  Google Scholar 

  • Bullmore ET, Bassett DS (2011) Brain graphs: graphical models of the human brain connectome. Annu Rev Clin Psychol 7:113–140

    Article  Google Scholar 

  • Capelle K (2006) A bird’s-eye view of density-functional theory. Braz J Phys 36:1318–1343

    Article  Google Scholar 

  • Casagrande D, Sassano M, Astolfi A (2012) Hamiltonian-based clustering: algorithms for static and dynamic clustering in data mining and image processing. IEEE Control Syst 32:74–91

    Google Scholar 

  • Chen H, Chiang RHL, Storey VC (2012) Business intelligence and analytics: from big data to big impact. MIS Q 36:1165–1188

    Google Scholar 

  • Chothani P, Mehta V, Stepanyants A (2011) Automated tracing of neurites from light microscopy stacks of images. Neuroinformatics 9:263–278

    Article  Google Scholar 

  • Clark MC, Hall LO, Goldgof DB, Velthuizen R, Murtagh FR, Silbiger MS (1998) Automatic tumor segmentation using knowledge-based techniques. IEEE Trans Med Imag 17:187–201

    Article  Google Scholar 

  • Clauset A, Newman MEJ, Moore C (2004) Finding community structure in very large networks. Phys Rev E 70:066111

    Google Scholar 

  • Cramer CJ, Truhlar DG (2009) Density functional theory for transition metals and transition metal chemistry. Phys Chem Chem Phys 11:10757–10816

    Article  Google Scholar 

  • Daw MS, Baskes MI (1983) Semiempirical, quantum mechanical calculation of hydrogen embrittlement in metals. Phys Rev Lett 50:1285–1288

    Article  Google Scholar 

  • Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc B 39:1–38

    Google Scholar 

  • Esquivel AV, Rosvall M (2011) Compression of flow can reveal overlapping-module organization in networks. Phys Rev X 1:021025

    Google Scholar 

  • Finken R, Schmidt M, Löwen H (2001) Freezing transition of hard hyperspheres. Phys Rev E 65:016108

    Google Scholar 

  • Foulkes WMC, Mitas L, Needs RJ, Rajagopal G (2001) Quantum Monte Carlo simulations of solids. Rev Mod Phys 73:33–83

    Article  Google Scholar 

  • Gala R, Chapeton J, Jitesh J, Bhavsar C, Stepanyants A (2014) Active learning of neuron morphology for accurate automated tracing of neurites. Front Neuroanat 8:1–14

    Google Scholar 

  • Girvan M, Newman MEJ (2002) Community structure in social and biological networks. Proc Natl Acad Sci USA 99:7821–7826

    Article  MathSciNet  Google Scholar 

  • Gordillo N, Montseny E, Sobrevilla, P (2013) State of the art survey on MRI brain tumor segmentation. Magn Reson Imaging 31:1426–1438

    Article  Google Scholar 

  • Grimme S, Antony J, Schwabe T, Mück-Lichtenfeld C (2007) Density functional theory with dispersion corrections for supramolecular structures, aggregates, and complexes of (bio)organic molecules. Org Biomol Chem 5:741–758

    Article  Google Scholar 

  • Hampel S, Chung P, McKellar CE, Hall D, Looger LL, Simpson JH (2011) Drosophila Brainbow: a recombinase-based fluorescence labeling technique to subdivide neural expression patterns. Nat Methods 8:253–259

    Article  Google Scholar 

  • Hohenberg P, Kohn W (1964) Inhomogeneous electron gas. Phys Rev 136:B864–B871

    Article  MathSciNet  Google Scholar 

  • Horn D, Gottlieb A (2001) Algorithm for data clustering in pattern recognition problems based on quantum mechanics. Phys Rev Lett 88:018702

    Google Scholar 

  • Hsu Y, Lu HH-S (2013) Brainbow image segmentation using Bayesian sequential partitioning. Int J Comput Electr Autom Control Inf Eng 7:897–902

    Google Scholar 

  • Jacobs A (2009) The pathologies of big data. Commun ACM 52:36–44

    Article  Google Scholar 

  • Kalal Z, Mikolajczyk K, Matas J (2012) Tracking-learning-detection. IEEE Trans Pattern Anal Mach Intell 34:1409–1422

    Article  Google Scholar 

  • Kobiler O, Lipman Y, Therkelsen K, Daubechies I, Enquist LW (2010) Herpesviruses carrying a Brainbow cassette reveal replication and expression of limited numbers of incoming genomes. Nat Commun 1:1–8

    Article  Google Scholar 

  • Kohn W (1999) Nobel lecture: electronic structure of matter-wave functions and density functionals. Rev Mod Phys 71:1253–1266

    Article  Google Scholar 

  • Kohn W, Sham LJ (1965) Self-consistent equations including exchange and correlation effects. Phys Rev 140:A1133–A1138

    Article  MathSciNet  Google Scholar 

  • Kreshuk A, Straehle CN, Sommer C, Korthe U, Knott G, Hamprecht FA (2011) Automated segmentation of synapses in 3D EM data. In: International symposium on biomedical imaging (ISBI), pp 220–223

    Google Scholar 

  • Langreth DC, Perdew JP (1997) Exchange-correlation energy of a metallic surface: wave-vector analysis. Phys Rev B 15:2884–2901

    Article  Google Scholar 

  • Lebègue S, Björkman T, Klintenberg M, Nieminen RM, Eriksson O (2013) Two-dimensional materials from data filtering and Ab initio calculations. Phys Rev X 3:031002

    Google Scholar 

  • Lebowitz JL, Lieb EH (1969) Existence of thermodynamics for real matter with Coulomb forces. Phys Rev Lett 22:631–634

    Article  Google Scholar 

  • Lichtman JW, Livet J, Sanes JR (2008) A technicolour approach to the connectome. Nat Rev Neurosci 9:417–422

    Article  Google Scholar 

  • Livet J, Weissman TA, Kang H, Draft RW, Lu J, Bennis RA, Sanes JR, Kichtman JW (2007) Transgenic strategies for combinatorial expression of fluorescent proteins in the nervous system. Nature 450:56–62

    Article  Google Scholar 

  • Levin A, Rav-Acha A, Lischinski D (2008) Spectral matting. IEEE Trans Pattern Anal Mach Intell 30:1699–1712

    Article  Google Scholar 

  • Levy M (1982) Electron densities in search of Hamiltonians. Phys Rev A 26:1200–1208

    Article  Google Scholar 

  • Levy M (2010) On the simple constrained-search reformulation of the Hohenberg-Kohn theorem to include degeneracies and more (1964–1979). Int J Quant Chem 110:3140–3144

    Article  Google Scholar 

  • Levy M, Perdew JP (1985) Hellmann-Feynman, virial, and scaling requisites for the exact universal density functionals. Shape of the correlation potential and diamagnetic susceptibility for atoms. Phys Rev A 32:2010–2021

    Article  Google Scholar 

  • Lu L, Jiang H, Wong WH (2013) Multivariate density estimation by Bayesian Sequential Partitioning. J Am Stat Assoc Theory Methods 108:1402–1410

    Article  MathSciNet  Google Scholar 

  • March NH (1986) Spatially dependent generalization of Kato’s theorem for atomic closed shells in a bare Coulomb field. Phys Rev A 33:88–89

    Article  MathSciNet  Google Scholar 

  • McAfee A, Brynjolfsson E (2012) Big data: the management revolution. Harv Bus Rev 90:59–68

    Google Scholar 

  • Moon N, Bullitt E, van Leemput K, Gerig G (2002) Automatic brain and tumor segmentation. In: MICCAI proceedings. Lecture notes in computer science, vol 2488, pp 372–379

    Chapter  Google Scholar 

  • Needs RJ, Towler MD, Drummond ND, Ríos PL (2010) Continuum variational and diffusion quantum Monte Carlo calculations. J Phys Condens Matter 22:023201

    Google Scholar 

  • Neese F (2009) Prediction of molecular properties and molecular spectroscopy with density functional theory: from fundamental theory to exchange-coupling. Coord Chem Rev 253:526–563

    Article  Google Scholar 

  • Newman MEJ, Girvan M (2004) Finding and evaluating community structure in networks. Phys Rev E 69:026113

    Google Scholar 

  • Peng H, Long F, Myers G (2011) Automatic 3D neuron tracing using all-path pruning. Bioinformatics 27:i239–i247

    Article  Google Scholar 

  • Prastawa M, Bullitt E, Ho S, Gerig G (2004) A brain tumor segmentation framework based on outlier detection. Med Image Anal 8:275–283

    Article  Google Scholar 

  • Riley KE, Pitoňák M, Jurečka P, Hobza P (2010) Stabilization and structure calculations for noncovalent interactions in extended molecular systems based on wave function and density functional theories. Chem Rev 110:5023–5063

    Article  Google Scholar 

  • Rodriguez A, Ehlenberger DB, Hof PR, Wearne SL (2009) Three-dimensional neuron tracing by voxel scooping. J Neurosci Methods 184:169–175

    Article  Google Scholar 

  • Rozas J, Sánchez-DelBarrio JC, Messeguer X, Rozas R (2003) DnaSP, DNA polymorphism analyses by the coalescent and other methods. Bioinformatics 19:2496–2497

    Article  Google Scholar 

  • Salasnich L, Toigo F (2008) Extended Thomas-Fermi density functional for the unitary Fermi gas. Phys Rev A 78:053626

    Google Scholar 

  • Schadt EE, Linderman MD, Sorenson J, Lee L, Nolan GP (2011) Cloud and heterogeneous computing solutions exist today for the emerging big data problems in biology. Nat Rev Genet 12:224–224

    Article  Google Scholar 

  • Shao H-C, Cheng W-Y, Chen Y-C, Hwang W-L (2012) Colored multi-neuron image processing for segmenting and tracing neural circuits. In: International conference on image processing (ICIP), pp 2025–2028

    Google Scholar 

  • Sporns O (2011) The human connectome: a complex network. Ann NY Acad Sci 1224:109–125

    Article  Google Scholar 

  • Tóth B, Lempérière Y, Deremble C, de Lataillade J, Kockelkoren J, Bouchaud, J-P (2011) Anomalous price impact and the critical nature of liquidity in financial markets. Phys Rev X 1:021006

    Google Scholar 

  • Türetken E, González G, Blum C, Fua P (2011) Automated reconstruction of dendritic and axonal trees by global optimization with geometric priors. Neuroinformatics 9:279–302

    Article  Google Scholar 

  • Türetken E, Benmansour F, Andres B, Pfister H, Fua P (2013) Reconstructing loopy curvilinear structures using integer programming. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1822–1829

    Google Scholar 

  • UCI Machine Learning Repository Iris Dataset (1988) Available at https://archive.ics.uci.edu/ml/datasets/Iris

  • Ullrich CA (2012) Time-dependent density-functional theory. Oxford University Press, New York, pp 21–41

    Google Scholar 

  • Vasilkoski Z, Stepanyants A (2009) Detection of the optimal neuron traces in confocal microscopy images. J Neurosci Methods 178:197–204

    Article  Google Scholar 

  • Voorhis TV, Scuseria GE (1998) A novel form for the exchange-correlation energy functional. J Chem Phys 109:400–410

    Google Scholar 

  • Wang C, Deng F-G, Li Y-S, Liu X-S, Long GL (2005) Quantum secure direct communication with high-dimension quantum superdense coding. Phys Rev A 71:044305

    Google Scholar 

  • Wang Y, Narayanaswamy A, Tsai C-L, Roysam B (2011) A broadly applicable 3-D neuron tracing method based on open-curve snake. Neuroinformatics 9:193–217

    Article  Google Scholar 

  • Wu J (2006) Density functional theory for chemical engineering: from capillarity to soft materials. AIChE J 52:1169–1193

    Article  Google Scholar 

  • Wu T-Y, Juan H-H, Lu HH-S, Chiang A-S (2011) A crosstalk tolerated neural segmentation methodology for Brainbow images. In: Proceedings of the 4th international symposium on applied sciences in biomedical and communication technologies (ISABEL)

    Google Scholar 

  • Wu T-Y, Juan H-H, Lu, HH-S (2012) Improved spectral matting by iterative K-means clustering and the modularity measure. In: IEEE international conference on acoustics, speech, and signal processing (IEEE ICASSP), pp 1165–1168

    Google Scholar 

  • Zhang J, Ma K-K, Er M-H, Chong V (2004) Tumor segmentation from magnetic resonance imaging by learning via one-class support vector machine. In: International Workshop on Advanced Image Technology, IWAIT, pp 207–211

    Google Scholar 

  • Zhang Y, Chen K, Baron M, Teylan MA, Kim Y, Song Z, Greengard P, Wong STC (2010) A neurocomputational method for fully automated 3D dendritic spine detection and segmentation of medium-sized spiny neurons. Neuroimage 50:1472–1484

    Article  Google Scholar 

  • Zupan A, Burke K, Ernzerhof M, Perdew JP (1997) Distributions and averages of electron density parameters: explaining the effects of gradient corrections. J Chem Phys 106:10184–10193

    Article  Google Scholar 

  • Zupan A, Perdew JP, Burke K, Causá M (1997) Density-gradient analysis for density functional theory: application to atoms. Int J Quant Chem 61:835–845

    Article  Google Scholar 

Download references

Acknowledgements

We would like to acknowledge the supports from National Science Council, National Center for Theoretical Sciences, Shing-Tung Yau Center, Center of Mathematical Modeling and Scientific Computing at National Chiao Tung University in Taiwan.

Author information

Authors and Affiliations

  1. Bio-MicroSystems Integration Laboratory, Department of Biomedical Sciences and Engineering, National Central University, Taoyuan City, Taiwan

    Chien-Chang Chen

  2. Shing-Tung Yau Center, National Chiao Tung University, Hsinchu City, Taiwan

    Chien-Chang Chen, Hung-Hui Juan & Henry Horng-Shing Lu

  3. Institute of Statistics, National Chiao Tung University, Hsinchu City, Taiwan

    Meng-Yuan Tsai & Henry Horng-Shing Lu

  4. Big Data Research Center, National Chiao Tung University, Hsinchu City, Taiwan

    Henry Horng-Shing Lu

Authors
  1. Chien-Chang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hung-Hui Juan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meng-Yuan Tsai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Henry Horng-Shing Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Henry Horng-Shing Lu .

Editor information

Editors and Affiliations

  1. Ladislaus von Bortkiewicz Chair of Statistics, C.A.S.E. Center for Applied Statistics & Economics, Humboldt-Universität zu Berlin, Berlin, Germany

    Wolfgang Karl Härdle

  2. Institute of Statistics, National Chiao Tung University, Hsinchu, Taiwan

    Henry Horng-Shing Lu

  3. School of Statistics, University of Minnesota, Minneapolis, USA

    Xiaotong Shen

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Chen, CC., Juan, HH., Tsai, MY., Lu, H.HS. (2018). Bridging Density Functional Theory and Big Data Analytics with Applications. In: Härdle, W., Lu, HS., Shen, X. (eds) Handbook of Big Data Analytics. Springer Handbooks of Computational Statistics. Springer, Cham. https://doi.org/10.1007/978-3-319-18284-1_15

Download citation

Publish with us

Policies and ethics

Search

Navigation