Markov Chains and Monte Carlo Simulation

In this chapter, we present Markov chains that are the most fundamental and long-standing graphical model for modeling dependencies among data entities. We start by introducing various definitions, terminologies, and important properties of Markov chains, followed by describing the stationary distribution and associated theorems. At the end of this chapter, we present the Markov Chain Monte Carlo simulation (MCMC) that is one of the most important applications of Markov chains for probabilistic data sampling and model estimations.