Skip to main content
SpringerLink
Log in

Relations Between Information and Estimation in the Presence of Feedback

  • Chapter
Book cover Information and Control in Networks

Part of the book series: Lecture Notes in Control and Information Sciences ((LNCIS,volume 450))

Abstract

We discuss some of the recent literature on relations between information- and estimation-theoretic quantities. We begin by exploring the connections between mutual information and causal/non-causal, matched/mismatched estimation for the setting of a continuous-time source corrupted by white Gaussian noise. Relations involving causal estimation, in both matched and mismatched cases, and mutual information persist in the presence of feedback. We present a new unified framework, based on Girsanov theory and Itô’s Calculus, to derive these relations. We conclude by deriving some new results using this framework.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Define the filtration \(\mathcal{F}^{Y}_{t} = \sigma\{ Y(B) : B \subseteq\{s: s < t\} \}\). Note that in the setting of Theorem 5.5, the encoder ϕ t is measurable w.r.t. the σ-algebra \(\mathcal{F}^{X}_{T} \vee\mathcal{F}^{Y}_{t}\), and the estimate \(\hat{\phi}_{t}\) is measurable w.r.t. (or adapted to the filtration) \(\mathcal{F}^{Y}_{t}\).

References

  1. Atar, R., Weissman, T.: Mutual information, relative entropy, and estimation in the Poisson channel. IEEE Trans. Inf. Theory 58(3), 1302–1318 (2012)

    Article  MathSciNet  Google Scholar 

  2. Barron, A.R.: Entropy and the central limit theorem. Ann. Probab. 14(1), 336–342 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  3. Brown, L., Dasgupta, A., Haff, L.R., Strawderman, W.E.: The heat equation and Stein’s identity: connections, applications. J. Stat. Plan. Inference 136(7), 2254–2278 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  4. Cover, T.M., Thomas, J.A.: Elements of Information Theory, 2nd edn. Wiley, New York (2006)

    MATH  Google Scholar 

  5. Duncan, T.E.: On the calculation of mutual information. SIAM J. Appl. Math. 19, 215–220 (1970)

    Article  MathSciNet  MATH  Google Scholar 

  6. Girsanov, I.V.: On transforming a certain class of stochastic processes by absolutely continuous substitution of measures. Theory Probab. Appl. 5, 285–301 (1960)

    Article  Google Scholar 

  7. Guo, D., Shamai, S., Verdú, S.: Mutual information and minimum mean-square error in Gaussian channels. IEEE Trans. Inf. Theory IT-51(4), 1261–1283 (2005)

    Article  Google Scholar 

  8. Guo, D., Wu, Y., Shamai (Shitz), S., Verdú, S.: Estimation in Gaussian noise: properties of the minimum mean-square error. IEEE Trans. Inf. Theory 57(4), 2371–2385 (2011)

    Article  Google Scholar 

  9. Kadota, T.T., Zakai, M., Ziv, J.: Mutual information of the white. Gaussian channel with and without feedback. IEEE Trans. Inf. Theory IT-17(4), 368–371 (1971)

    Article  MathSciNet  Google Scholar 

  10. Karatzas, I., Shreve, A.E.: Brownian Motion and Stochastic Calculus, 2nd edn. Springer, New York (1988)

    Book  MATH  Google Scholar 

  11. Merhav, N.: Data processing theorems and the second law of thermodynamics. IEEE Trans. Inf. Theory 57(8), 4926–4939 (2011)

    Article  MathSciNet  Google Scholar 

  12. No, A., Weissman, T.: Minimax filtering regret via relations between information and estimation. In: 2013 IEEE International Symposium on Information Theory Proceedings (ISIT), 7–12 July 2013, pp. 444–448 (2013)

    Chapter  Google Scholar 

  13. Palomar, D., Verdú, S.: Representation of mutual information via input estimates. IEEE Trans. Inf. Theory 53(2), 453–470 (2007)

    Article  Google Scholar 

  14. Palomar, D.P., Verdu, S.: Lautum information. IEEE Trans. Inf. Theory 54(3), 964–975 (2008)

    Article  MathSciNet  Google Scholar 

  15. Polyanskiy, Y., Poor, H.V., Verdú, S.: New channel coding achievability bounds. In: IEEE Int. Symposium on Information Theory 2008, Toronto, Ontario, Canada, 6–11 July 2008

    Google Scholar 

  16. Stam, A.J.: Some inequalities satisfied by the quantities of information of Fisher and Shannon. Inf. Control 2(2), 101–112 (1959)

    Article  MathSciNet  MATH  Google Scholar 

  17. Steele, J.M.: Stochastic Calculus and Financial Applications. Springer, Berlin (2010)

    Google Scholar 

  18. Venkat, K., Weissman, T.: Pointwise relations between information and estimation in Gaussian noise. IEEE Trans. Inf. Theory 58(10), 6264–6281 (2012)

    Article  MathSciNet  Google Scholar 

  19. Verdú, S.: Mismatched estimation and relative entropy. IEEE Trans. Inf. Theory 56(8), 3712–3720 (2010)

    Article  Google Scholar 

  20. Weissman, T.: The relationship between causal and noncausal mismatched estimation in continuous-time AWGN channels. IEEE Trans. Inf. Theory 56(9), 4256–4273 (2010)

    Article  MathSciNet  Google Scholar 

  21. Weissman, T., Kim, Y.-H., Permuter, H.H.: Directed information, causal estimation, and communication in continuous time. IEEE Trans. Inf. Theory 59(3), 1271–1287 (2012)

    Article  MathSciNet  Google Scholar 

  22. Wu, Y., Verdu, S.: Functional properties of MMSE and mutual information. IEEE Trans. Inf. Theory 58(3), 1289–1301 (2012)

    Article  MathSciNet  Google Scholar 

  23. Zakai, M.: On mutual information, likelihood ratios, and estimation error for the additive Gaussian channel. IEEE Trans. Inf. Theory 51(9), 3017–3024 (2005)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgement

This research was supported by LCCC—Linnaeus Grant VR 2007-8646, Swedish Research Council.

Author information

Authors and Affiliations

  1. Electrical Engineering Department, Stanford University, Stanford, CA, USA

    Himanshu Asnani, Kartik Venkat & Tsachy Weissman

Authors
  1. Himanshu Asnani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kartik Venkat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tsachy Weissman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Himanshu Asnani .

Editor information

Editors and Affiliations

  1. Department of Automatic Control, Lund University, Lund, Sweden

    Giacomo Como

  2. Department of Automatic Control, Lund University, Lund, Sweden

    Bo Bernhardsson

  3. Department of Automatic Control, Lund University, Lund, Sweden

    Anders Rantzer

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Asnani, H., Venkat, K., Weissman, T. (2014). Relations Between Information and Estimation in the Presence of Feedback. In: Como, G., Bernhardsson, B., Rantzer, A. (eds) Information and Control in Networks. Lecture Notes in Control and Information Sciences, vol 450. Springer, Cham. https://doi.org/10.1007/978-3-319-02150-8_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-02150-8_5

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-02149-2

  • Online ISBN: 978-3-319-02150-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation