Markov Decision Processes in Practice pp 505-519 | Cite as

# MDP for Query-Based Wireless Sensor Networks

## Abstract

Increased sensors availability and growing interest in sensor monitoring has lead to an significant increase in the number of sensor networks deployed in the last decade. Simultaneously, the amount of sensed data and the number of queries calling this data significantly increased. The challenge is to respond to the queries in a timely manner and with relevant data, without having to resort to hardware updates or duplication. In this chapter we focus on the trade-off between the response time of queries and the freshness of the data provided. Query response time is a significant Quality of Service for sensor networks, especially in the case of real-time applications. Data freshness ensures that queries are answered with relevant data, that closely characterizes the monitored area. To model the trade-off between the two metrics, we propose a continuous-time Markov decision process with a drift, which assigns queries for processing either to a sensor network, where queries *wait* to be processed, or to a central database, which provides stored and possibly *outdated* data. To compute an optimal query assignment policy, a discrete-time discrete-state Markov decision process, shown to be stochastically equivalent to the initial continuous-time process, is formulated. This approach provides a theoretical support for the design and implementation of WSN applications, while ensuring a close-to-optimum performance of the system.

## References

- 1.E.B. Dynkin,
*Markov Processes*, vol. 1 (Academic, New York, 1965)CrossRefGoogle Scholar - 2.I.I. Gikhman, A.V. Skorokhod,
*The Theory of Stochastic Processes: II*, vol. 232 (Springer, New York, 2004)CrossRefGoogle Scholar - 3.A. Hordijk, R. Schassberger, Weak convergence for generalized semi-markov processes. Stoch. Process. Appl.
**12**(3), 271–291 (1982)CrossRefGoogle Scholar - 4.A. Hordijk, F.A. van der Duyn Schouten, Discretization and weak convergence in Markov decision drift processes. Math. Oper. Res.
**9**(1), 112–141 (1984)CrossRefGoogle Scholar - 5.A. Jensen, Markoff chains as an aid in the study of markoff processes. Scand. Actuar. J.
**1953**(sup1), 87–91 (1953)Google Scholar - 6.M. Mitici, M. Onderwater, M. de Graaf, J.-K. van Ommeren, N. van Dijk, J. Goseling, R.J. Boucherie, Optimal query assignment for wireless sensor networks. AEU-Int. J. Electron. Commun.
**69**(8), 1102–1112 (2015)CrossRefGoogle Scholar - 7.A.R. Odoni, On finding the maximal gain for Markov decision processes. Oper. Res.
**17**(5), 857–860 (1969)CrossRefGoogle Scholar - 8.M.L. Puterman,
*Markov Decision Processes: Discrete Stochastic Dynamic Programming*(Wiley, New York, 1994)CrossRefGoogle Scholar - 9.N. van Dijk, On a simple proof of uniformization for continuous and discrete-state continuous-time markov chains. Adv. Appl. Probab.
**22**(3), 749–750 (1990)CrossRefGoogle Scholar - 10.N. van Dijk, A. Hordijk, Time-discretization for controlled Markov processes. I. General approximation results. Kybernetika
**32**(1), 1–16 (1996)Google Scholar