Adaptive Filtering pp 79-138 | Cite as

# The Least-Mean-Square (LMS) Algorithm

## Abstract

The least-mean-square (LMS) is a search algorithm in which a simplification of the gradient vector computation is made possible by appropriately modifying the objective function [1]–[2]. The LMS algorithm, as well as others related to it, is widely used in various applications of adaptive filtering due to its computational simplicity [3]–[7]. The convergence characteristics of the LMS algorithm are examined in order to establish a range for the convergence factor that will guarantee stability. The convergence speed of the LMS is shown to be dependent on the eigenvalue spread of the input signal correlation matrix [2]–[6]. In this chapter, several properties of the LMS algorithm are discussed including the misadjustment in stationary and nonstationary environments [2]–[9] and tracking performance [10]–[12]. The analysis results are verified by a large number of simulation examples. Appendix A, section A.1, complements this chapter by analyzing the finite–wordlength effects in LMS algorithms.

## Keywords

Gaussian White Noise Adaptive Filter Convergence Factor Unknown System Convergence Path## Preview

Unable to display preview. Download preview PDF.

## References

- 1.B. Widrow and M. E. Hoff, “Adaptive switching circuits,”
*WES COM Conv. Rec*, pt. 4, pp. 96–140, 1960.Google Scholar - 2.B. Widrow, J. M. McCool, M. G. Larimore, and C. R. Johnson Jr., “Stationary and nonstationary learning characteristics of the LMS adaptive filters,”
*Proceedings of the IEEE*, vol. 64, pp. 1151–1162, Aug. 1976.MathSciNetCrossRefGoogle Scholar - 3.G. Ungerboeck, “Theory on the speed of convergence in adaptive equalizers for digital communication,”
*IBM Journal on Research and Development*, vol. 16, pp. 546–555, Nov. 1972.MATHCrossRefGoogle Scholar - 4.J. E. Mazo, “On the independence theory of equalizer convergence,”
*The Bell System Technical Journal*, vol. 58, pp. 963–993, May 1979.MathSciNetMATHGoogle Scholar - 5.B. Widrow and S. D. Stearns,
*Adaptive Signal Processing*, Prentice Hall, Englewood Cliffs, NJ, 1985.MATHGoogle Scholar - 6.S. Haykin,
*Adaptive Filter Theory*, Prentice Hall, Englewood Cliffs, NJ, 4th edition, 2002.Google Scholar - 7.M. G. Bellanger,
*Adaptive Digital Filters and Signal Analysis*, Marcel Dekker Inc., NY, 1987.Google Scholar - 8.D. C. Farden, “Tracking properties of adaptive signal processing algorithms,”
*IEEE Trans, on Acoust., Speech, and Signal Processing*, vol. ASSP-29, pp. 439–446, June 1981.MathSciNetCrossRefGoogle Scholar - 9.B. Widrow and E. Walach, “On the statistical efficiency of the LMS algorithm with nonstationary inputs,”
*IEEE Trans, on Information Theory*, vol. IT-30, pp. 211–221, March 1984.CrossRefGoogle Scholar - 10.O. Macchi, “Optimization of adaptive identification for time varying filters,”
*IEEE Trans, on Automatic Control*, vol. AC-31, pp. 283–287, March 1986.CrossRefGoogle Scholar - 11.A. Benveniste, “Design of adaptive algorithms for the tracking of time varying systems,”
*Int. J. Adaptive Control and Signal Processing*, vol. 1, pp. 3–29, Jan. 1987.MATHCrossRefGoogle Scholar - 12.W. A. Gardner, “Nonstationary learning characteristics of the LMS algorithm,”
*IEEE Trans, on Circuits and Systems*, vol. CAS-34, pp. 1199–1207, Oct. 1987.CrossRefGoogle Scholar - 13.A. Papoulis,
*Probability, Random Variables, and Stochastic Processes*, McGraw Hill, New York, NY, 3rd edition, 1991.Google Scholar - 14.F. J. Gantmacher,
*The Theory of Matrices*, Chelsea Publishing Company, vol. 2, New York, NY, 1964.Google Scholar - 15.V. Solo, “The limiting behavior of LMS,”
*IEEE Trans, on Acoust., Speech, and Signal Processing*, vol-37, pp. 1909–1922, Dec. 1989.MathSciNetMATHCrossRefGoogle Scholar - 16.N. J. Bershad and O. M. Macchi, “Adaptive recovery of a chirped sinusoid in noise, Part 2: Performance of the LMS algorithm,”
*IEEE Trans, on Signal Processing*, vol. 39, pp. 595–602, March 1991.CrossRefGoogle Scholar - 17.D. G. Manolakis, V. K. Ingle, and S. M. Kogon,
*Statistical and Adaptive Signal Processing*, McGraw Hill, New York, NY, 2000.Google Scholar - 18.V. H. Nascimento and A. H. Sayed, “On the learning mechanism of adaptive filters,”
*IEEE Trans, on Signal Processing*, vol. 48, pp. 1609–1625, June 2000.CrossRefGoogle Scholar - 19.O. Macchi and E. Eweda, “Second–order convergence analysis of stochastic adaptive linear filter,”
*IEEE Trans, on Automatic Control*, vol. AC-28, pp. 76–85, March 1983.MathSciNetCrossRefGoogle Scholar - 20.S. Florian and A. Feuer, “Performance analysis of the LMS algorithm with a tapped delay line (two–dimensional case),”
*IEEE Trans, on Acoust., Speech, and Signal Processing*, vol. ASSP-34, pp. 1542–1549, Dec. 1986.CrossRefGoogle Scholar - 21.O. L. Frost III, “An algorithm for linearly constrained adaptive array processing,”
*Proceedings of the IEEE*, vol. 60, pp. 926–935, Aug. 1972.CrossRefGoogle Scholar - 22.J. A. Apolinario Jr., S. Werner, T. I. Laakso, and P. S. R. Diniz, “Constrained normalized adaptive filtering for CDMA mobile communications,”
*Proc. 1998 EUSIPCO–European Signal Processing Conference*, Rhodes, Greece, pp. 2053–2056, Sept. 1998.Google Scholar - 23.J. A. Apolinario Jr., M. L. R. de Campos, and C. P. Bernal Jr., “The Constrained conjugate–gradient algorithm,”
*IEEE Signal Processing Letters*, vol. 7, pp.351–354, Dec. 2000.CrossRefGoogle Scholar - 24.M. L. R. de Campos, S. Werner, and J. A. Apolinario Jr., “Constrained adaptation algorithms employing Householder transformations”
*IEEE Transactions on Signal Processing*, accepted for publication 2002.Google Scholar - 25.A. Feuer and E. Weinstein, “Convergence analysis of LMS filters with uncorrelated Gaussian data,”
*IEEE Trans, on Acoust., Speech, and Signal Processing*, vol. ASSP-33, pp. 222–230, Feb. 1985.CrossRefGoogle Scholar - 26.D. T. Slock, “On the convergence behavior of the LMS and normalized LMS algorithms,”
*IEEE Trans, on Signal Processing*, vol. 40, pp. 2811–2825, Sept. 1993.CrossRefGoogle Scholar - 27.W. A. Sethares, D. A. Lawrence, C. R. Johnson Jr., and R. R. Bitmead, “Parameter drift in LMS adaptive filters,”
*IEEE Trans, on Acoust., Speech, and Signal Processing*, vol. ASSP-34, pp. 868–878, Aug. 1986.CrossRefGoogle Scholar - 28.S. C. Douglas, “Exact expectation analysis of the LMS adaptive filter,”
*IEEE Trans, on Signal Processing*, vol. 43, pp. 2863–2871, Dec. 1995.CrossRefGoogle Scholar - 29.H. J. Butterweck, “Iterative analysis of the state–space weight fluctuations in LMS–type adaptive filters,”
*IEEE Trans, on Signal Processing*, vol. 47, pp. 2558–2561, Sept. 1999.MATHCrossRefGoogle Scholar - 30.B. Hassibi, A. H. Sayed, and T. Kailath, “H∞optimality of the LMS Algorithm,”
*IEEE Trans, on Signal Processing*, vol. 44, pp. 267–280, Feb. 1996.CrossRefGoogle Scholar - 31.O. J. Tobias, J. C. M. Bermudez, and N. J. Bershad, “Mean weight behavior of the filtered–X LMS algorithm,”
*IEEE Trans, on Signal Processing*, vol. 48, pp. 1061–1075, April 2000.CrossRefGoogle Scholar - 32.V. Solo, “The error variance of LMS with time varying weights,”
*IEEE Trans, on Signal Processing*, vol. 40, pp. 803–813, April 1992.MATHCrossRefGoogle Scholar - 33.S. U. Qureshi, “Adaptive Equalization,”
*Proceedings of the IEEE*, vol-73, pp. 1349–1387, Sept. 1985.CrossRefGoogle Scholar - 34.M. L. Honig, “Echo cancellation of voiceband data signals using recursive least squares and stochastic gradient algorithms,”
*IEEE Trans, on Communications*, vol. COM-33, pp. 65–73, Jan. 1985.CrossRefGoogle Scholar