Abstract
Despite long experience with the use of probability for inductive reasoning, propagation of conditional probability using a reduced rather than sufficient data statistic remains to be a challenging puzzle. The paper analyses in detail the main obstacles on the way towards real-time recursive implementation of Bayesian parameter estimation. Three key issues are dealt with — reduction of excessive information, approximation of the ideal Bayesian solution and numerical implementation of the resulting estimator. In all the stages of design of an approximate Bayesian estimator, the uncertainty of the true conditional probability caused by using just limited computer resources is required to be reflected consistently in the posterior uncertainty of the unknown parameters.
Key words
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Abramowitz, M. and I.A. Stegun (Eds) (1964). Handbook of Mathematical Functions with Formulas, Graphs and Mathematical Tables. John Wiley, New York.
Amari, S. (1990). Differential-Geometrical Methods in Statistics,2nd revised edition. Springer-Verlag, New York.
Bahadur, R.R. (1954). Sufficiency and statistical decision functions. Ann. Math. Statist., 25, 423–462.
Barron, A.R. and R.L. Barron (1988). Statistical learning networks: a unifying view. Computing Science and Statistics: Proc. 20th Symp. on the Interface, Alexandria, Va., 192–203.
Barron, A.R. (1989). Statistical properties of artificial neural networks. Proc. 28th IEEE Conf. on Decision and Control,New York.
Barron, A.R. and C.H. Sheu (1991). Approximation of density functions by sequences of exponential families. Ann. Statist., 19 1347–1369.
Bernardo, J.M. (1987). Approximations in statistics from a decision theoretical viewpoint. In R. Viertl (Ed.), Probability and Bayesian Statistics. Plenum Press, New York.
Bertsekas, D. and S.E. Shreve (1978). Stochastic Optimal Control: The Discrete Time Case. Academic Press, New York.
Billings, S.A., H.B. Jamaluddin and S. Chen (1992). Properties of neural networks with applications to modelling non-linear dynamical systems. Int. J. Control, 55, 193–224.
Bohlin, T. (1970). Information pattern for linear discrete-time models with stochastic coefficients. IEEE Trans. Automatic Control, 15, 104–106.
Breiman, L., J.H. Friedman, R.A. Olshen and C.J. Stone (1984). Classification and Regression Trees. Wadsworth, Belmont, Ca.
Brown, L.D. (1987). Fundamentals of Statistical Exponential Families. Institute of Mathematical Statistics, Hayward, Ca.
Byrnes, C.I. and A. Lindquist (Eds) (1986). Theory and Applications of Nonlinear Control Systems. North-Holland, Amsterdam.
Byrnes, C.I., C.F. Martin and R.E. Saeks (Eds) (1988). Analysis and Control of Nonlinear Systems. North-Holland, Amsterdam.
Cencov, N.N. (1982). Statistical Decision Rules and Optimal Inference. American Mathematical Society, Providence, R.I.
Cox, R.T. (1946). Probability, frequency and reasonable expectation. Am. J. Physics, 14, 1–13.
Csiszár, I. (1975). I-divergence geometry of probability distributions and minimization problems. Ann. Probab., 3, 146–158.
Davis, M.H.A. and P.P. Varaiya (1972). Information states for linear stochastic systems. J. Mathematical Analysis and Applications, 37, 384–402.
Davis, M.H.A. and S.I. Marcus (1981). An introduction to nonlinear filtering. In M. Hazewinkel and J.C. Willems (Eds), Stochastic Systems: The Mathematics of Filtering and Identification and Applications. Reidel, Dordrecht., 53–75.
Dawid, A.P. (1979). Conditional independence in statistical theory (with discussion). J. R. Statist. Soc. B, 41, 1–31.
de Boor, C. and J.R. Rice (1979). An adaptive algorithm for multivariate approximation giving optimal convergence rates. J. Approx. Theory,25, 337–359.
de Finetti, B. (1990). Theory of Probability, Vol. 1 and 2, Wiley Classics Library Edition. John Wiley, Chichester.
Dempster, A.P. (1967). Upper and lower probabilities induced by a multivalued mapping. Ann. Math. Statist., 38, 325–339.
Donoho, D.L. and I.M. Johnstone (1989). Projection-based approximation and a duality with kernel methods. Ann. Statist., 17, 435–475.
DiMasi, G.B. and T.J. Taylor (1991). A new approximation method for nonlinear filtering using nilpotent harmonic analysis. Proc. 30th IEEE Conf. on Decision and Control, Brighton, UK., 2750–2751.
Ferguson, T.S. (1973). A Bayesian analysis of some nonparametric problems. Ann. Statist., 1, 209–230.
Fisher, R.A. (1922). On the mathematical foundations of theoretical statistics. Roy. Soc. Phil. Trans. A, 222 309–368.
Florens, J.P., M. Mouchart and J.M. Rolin (1990). Elements of Bayesian Statistics. Marcel Dekker, New York.
Friedman, J.H. and W. Stuetzle (1981). Projection pursuit regression. J. Amer. Statist. Assoc., 76 817–823.
Friedman, J.H. (1991). Multivariate adaptive regression splines (with discussion). Ann. Statist., 19 1–141.
Grandy, W.T. (1985). Incomplete information and generalized inverse problems. In C.R. Smith and W.T. Grandy (Eds), Maximum-Entropy and Bayesian Methods in Inverse Problems. Reidel, Dordrecht., 1–19.
Halmos, P.R. and L.J. Savage (1949). Application of the Radon-Nikodym theorem to the theory of sufficient statistics. Ann. Math. Stat., 20, 225–241.
Hampel, F.R., E.M. Ronchetti, P.J. Rousseeuw and W.A. Stahel (1986). Robust Statistics: The Approach Based on Influence Functions. John Wiley, New York.
Hazewinkel, M. and J.C. Willems (Eds) (1981). Stochastic Systems: The Mathematics of Filtering and Identification and Applications. Reidel, Dordrecht.
Huber, P.J. (1981). Robust Statistics. John Wiley, New York.
Huber, P.J. (1985). Projection pursuit. Ann. Statist., 13, 435–475.
Isidori, A. (1989). Nonlinear Control Systems: An Introduction, 2nd edition. Springer-Verlag, Berlin.
Jacobs, O.L.R. (Ed.) (1980). Analysis and Optimisation of Stochastic Systems. Academic Press, London.
Jaynes, E.T. (1979). Where do we stand on maximum entropy? In R.O. Levine and M. Tribus (Eds), The Maximum Entropy Formalism. MIT Press, Cambridge., 15–118.
Jaynes, E.T. (1984). Prior information and ambiguity in inverse problems. In D.W. McLaughlin (Ed.), Inverse Problems. American Mathematical Society, Providence, R.I.
Kárný, M., A. Halousková, J. Böhm, R. Kulhavý and P. Nedoma (1985). Design of linear quadratic adaptive control: theory and algorithms for practice. Supplement to the journal Kybernetika, 21, No. 1–4.
Kulhavý, R. (1990a). Recursive Bayesian estimation under memory limitation. Kybernetika, 26 1–16.
Kulhavý, R. (1990b). A Bayes-closed approximation of recursive nonlinear estimation. Int. J. Adaptive Control and Signal Processing,4, 271–285.
Kulhavý, R. (1990с). Recursive nonlinear estimation: a geometric approach. Automatica, 26, 545–555.
Kulhavý, R. (1992a). Recursive nonlinear estimation: geometry of a space of posterior densities. Automatica, 28, 313–323.
Kulhavý, R. (1992b). Implementation of Bayesian parameter estimation in adaptive control and signal processing. Int. Conf. on Practical Bayesian Statistics, Nottingham, UK.
Kulhavý, R., I. Nagy and J. Spousta (1992). Towards real-time implementation of Bayesian parameter estimation. IFAC Symp. on Adaptive Systems in Control and Signal Processing, Grenoble, France, 263–268.
Kulhavý, R. and M.B. Zarrop (1992). On a general concept of forgetting. Submitted to Int. J. Control.
Kumar, P.R. (1985). A survey of some results in stochastic adaptive control. SIAM J. Control and Optimization, 23, 329–380.
Kumar, P.R. and P.P. Varaiya (1986). Stochastic Systems: Estimation, Identification and Adaptive Control. Prentice Hall, Englewood Cliffs, N.J.
Kyburg, H.E. and H.E. Smokier (1964). Studies in Subjective Probability. John Wiley, New York.
Lauritzen, S.L. (1988). Extremal Families and Systems of Sufficient Statistics. Springer-Verlag, Berlin.
Light, W.A. and E.W. Cheney (1985). Approximation Theory in Tensor Product Spaces. Springer-Verlag, New York.
Lindley, D.V. (1980). Approximate Bayesian methods. In J.M. Bernardo, M.H. DeGroot, D.V. Lindley and A.F.M. Smith (Eds), Bayesian Statistics 1. Valencia Univ. Press, Valencia, 223–245.
Narendra, K.S. and K. Parthasarathy (1990). Identification and control of dynamical systems using neural networks. IEEE Trans. Neural Networks, 1, 4–27.
Peterka, V. (1981). Bayesian approach to system identification. In P. Eykhoff (Ed.), Trends and Progress in System Identification. Pergamon Press, Oxford. Chap. 8, 239–304.
Poggio, T. and F. Girosi (1989). A theory of networks for approximation and learning. A.I. Memo, No. 1140, Artificial Intelligence Laboratory, Massachusetts Inst. of Technology.
Poggio, T. and F. Girosi (1990). Networks for approximation and learning. Proc. IEEE, 78, 1481–1497.
Rudin, W. (1987). Real and Complex Analysis, 3rd edition. McGraw-Hill, New York.
Sonner, R.M. and J.-J.E. Slotine (1991). Stable adaptive control and recursive identification using radial Gaussian networks. Proc. 30th IEEE Conf. on Decision and Control, Brighton, UK, 2116–2123.
Savage, L.J. (1972). The Foundations of Statistics, 2nd edition. Dover Publications, New York.
Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton Univ. Press, Princeton, N.J.
Shaw, J.E.H. (1988a). Aspects of numerical integration and summarisation. In J.M. Bernardo, M.H. DeGroot, D.V. Lindley and A.F.M. Smith (Eds), Bayesian Statistics 3. Oxford Univ. Press, Oxford, 411–428.
Shaw, J.E.H. (1988b). A quasirandom approach to integration in Bayesian statistics. Ann. Statist., 16, 895–914.
Sorenson, H.W. (1974). Practical realization of nonlinear estimators. In D.G. Lainiotis (Ed.), Estimation Theory. American Elsevier, New York.
Sorenson, H.W. (1988). Recursive estimation for nonlinear dynamic systems. In J.C. Spall (Ed.), Bayesian Analysis of Time Series and Dynamic Models. Dekker, New York.
Striebel, C. (1965). Sufficient statistics in the optimal control of stochastic systems. J. Mathematical Analysis and Applications, 12, 576–592.
Tierney, L. and J.B. Kadane (1986). Accurate approximations for posterior moments and marginal densities. J. Amer. Statist. Assoc., 81, 82–86.
Tierney, L., R.E. Kass and J.B. Kadane (1989). Fully exponential Laplace approximations to expectations and variances of nonpositive functions. J. Amer. Statist. Assoc., 84, 710–716.
van Campenhout, J.M. and T.M. Cover (1981). Maximum entropy and conditional probability. IEEE Trans. Inform. Theory,IT-27, 483–489.
West, M. and J. Harrison (1988). Bayesian Forecasting and Dynamic Models. Springer-Verlag, New York.
White, H. (1989). Learning in artificial neural networks: a statistical perspective. Neural Computation, 1, 425–464.
Wiberg, D.M. and D.G. DeWolf (1991). A convergent approximation of the optimal parameter estimator. Proc. 30th IEEE Conf. on Decision and Control, Brighton, UK, 2017–2023.
Willems, J.C. (1980). Some remarks on the concept of information state. In O.L.R. Jacobs (Ed.), Analysis and Optimization of Stochastic Systems. Academic Press, London, 285–295.
Zadeh, L.A. (1978). Fuzzy sets as a basis for a theory of possibility. Fuzzy Sets and Systems, 1, 3–28.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1993 Springer Science+Business Media New York
About this chapter
Cite this chapter
Kulhavý, R. (1993). On Design of Approximate Finite-Dimensional Estimators: The Bayesian View. In: Kárný, M., Warwick, K. (eds) Mutual Impact of Computing Power and Control Theory. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-2968-2_2
Download citation
DOI: https://doi.org/10.1007/978-1-4615-2968-2_2
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4613-6291-3
Online ISBN: 978-1-4615-2968-2
eBook Packages: Springer Book Archive