Computational Optimization and Applications

, Volume 66, Issue 3, pp 533–556 | Cite as

Quasi-Newton smoothed functional algorithms for unconstrained and constrained simulation optimization

  • K. Lakshmanan
  • Shalabh Bhatnagar


We propose a multi-time scale quasi-Newton based smoothed functional (QN-SF) algorithm for stochastic optimization both with and without inequality constraints. The algorithm combines the smoothed functional (SF) scheme for estimating the gradient with the quasi-Newton method to solve the optimization problem. Newton algorithms typically update the Hessian at each instant and subsequently (a) project them to the space of positive definite and symmetric matrices, and (b) invert the projected Hessian. The latter operation is computationally expensive. In order to save computational effort, we propose in this paper a quasi-Newton SF (QN-SF) algorithm based on the Broyden-Fletcher-Goldfarb-Shanno (BFGS) update rule. In Bhatnagar (ACM TModel Comput S. 18(1): 27–62, 2007), a Jacobi variant of Newton SF (JN-SF) was proposed and implemented to save computational effort. We compare our QN-SF algorithm with gradient SF (G-SF) and JN-SF algorithms on two different problems – first on a simple stochastic function minimization problem and the other on a problem of optimal routing in a queueing network. We observe from the experiments that the QN-SF algorithm performs significantly better than both G-SF and JN-SF algorithms on both the problem settings. Next we extend the QN-SF algorithm to the case of constrained optimization. In this case too, the QN-SF algorithm performs much better than the JN-SF algorithm. Finally we present the proof of convergence for the QN-SF algorithm in both unconstrained and constrained settings.


Simulation Stochastic optimization Stochastic approximation algorithms Smoothed functional algorithm Quasi-Newton methods Constrained optimization Multi-stage queueing networks 

Mathematics Subject Classification



  1. 1.
    Andradottir, S.: A scaled stochastic approximation algorithm. Manag. Sci. 42, 475–498 (1996)CrossRefMATHGoogle Scholar
  2. 2.
    Akyildiz, I.F., Su, W., Sankarasubramaniam, Y., Cayirci, E.: Wireless sensor networks: a survey. Comput. Netw. 38(4), 393–422 (2002)CrossRefGoogle Scholar
  3. 3.
    Bhatnagar, S.: Adaptive multivariate three-timescale stochastic approximation algorithms for simulation based optimization. ACM Trans. Model. Comput. Simul. 15(1), 74–107 (2005)CrossRefGoogle Scholar
  4. 4.
    Bhatnagar, S.: Adaptive Newton-based smoothed functional algorithms for simulation optimization. ACM Trans. Model. Comput. Simul. 18(1), 27–62 (2007)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Bhatnagar, S., Borkar, V.S.: A two time scale stochastic approximation scheme for simulation based parametric optimization. Probab. Eng. Inf. Sci. 12, 519–531 (1998)CrossRefMATHGoogle Scholar
  6. 6.
    Bhatnagar, S., Fu, M.C., Marcus, S.I., Fard, P.J.: Optimal structured feedback policies for ABR flow control using two-timescale SPSA. IEEE/ACM Trans. Netw. 9(4), 479–491 (2001)CrossRefGoogle Scholar
  7. 7.
    Bhatnagar, S., Fu, M.C., Marcus, S.I., Bhatnagar, S.: Two timescale algorithms for simulation optimization of hidden Markov models. IIE Trans. 33(3), 245–258 (2001)Google Scholar
  8. 8.
    Bhatnagar, S., Hemachandra, N., Mishra, V.: Stochastic approximation algorithms for constrained optimization via simulation. ACM Trans. Model. Comput. Simul. 21(2), 15:1–15:22 (2011)Google Scholar
  9. 9.
    Bhatnagar, S., Prasad, H.L., Prashanth, L.A.: Stochastic Recursive Algorithms for Optimization: Simultaneous Perturbation Methods. Springer, New York (2013). LNCIS SeriesCrossRefMATHGoogle Scholar
  10. 10.
    Byrd, R.H., Hansen, S.L., Nocedal, J., Singer, Y.: A Stochastic Quasi-Newton Method for Large-Scale Optimization. CoRR arXiv:1401.7020 (2014)
  11. 11.
    Bordes, A., Bottou, L., Gallinari, P.: SGD-QN: careful quasi-Newton stochastic gradient descent. J. Mach. Learn. Res. 10, 1737–1754 (2009)MathSciNetMATHGoogle Scholar
  12. 12.
    Borkar, V.S.: Stochastic Approximation: A Dynamical Systems View point. Cambridge University Press and Hindustan Book Agency, New Delhi (2008)MATHGoogle Scholar
  13. 13.
    Borkar, V.S.: An actor-critic algorithm for constrained Markov decision processes. Syst. Control Lett. 54, 207–213 (2005)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Borkar, V.S., Meyn, S.P.: The O.D.E. method for convergence of stochastic approximation and reinforcement learning. SIAM J. Control Optim. 38(2), 447–469 (2000)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Brandiere, O.: Some pathological traps for stochastic approximation. SIAM J. Control Optim. 36, 1293–1314 (1998)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Cohen, J.E., Kelly, F.P.: A paradox of congestion in a queueing network. J. Appl. Probab. 27, 730–734 (1990)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    Dennis, J.E., Morée, J.J.: Quasi-Newton methods, motivation and theory. SIAM Rev. 19(1), 46–89 (1977)MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Harchol-Balter, M., Crovella, M., Murta, C.: On choosing a task assignment policy for a distributed server system. IEEE J. Parallel Distrib. Comput. 59(2), 204–228 (1999)CrossRefGoogle Scholar
  19. 19.
    Hirsch, M.W.: Convergent activation dynamics in continuous time networks. Neural Netw. 2, 331–349 (1989)CrossRefGoogle Scholar
  20. 20.
    Kao, C., Chen, S.: A stochastic quasi-Newton method for simulation response optimization. Eur. J. Oper. Res. 173, 30–46 (2006)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Katkovnik, V.Y., Kulchitsky, Y.: Convergence of a class of random search algorithms. Autom. Remote Control 8, 1321–1326 (1972)MathSciNetMATHGoogle Scholar
  22. 22.
    Kushner, H.J., Yin, G.G.: Stochastic Approximation and Recursive Algorithms and Applications. Springer, New York (2003)MATHGoogle Scholar
  23. 23.
    Lakshmanan, K., Bhatnagar, S.: Smoothed functional and quasi-Newton algorithms for routing in multi-stage queueing network with constraints. In: International Conference on Distributed Computing and Internet Technology (ICDCIT), vol. 6536, pp. 175–186. LNCS (2011)Google Scholar
  24. 24.
    Pemantle, R.: Nonconvergence to unstable points in urn models and stochastic approximations. Ann. Probab. 18, 698–712 (1990)MathSciNetCrossRefMATHGoogle Scholar
  25. 25.
    Schweitzer, P.J.: Perturbation theory and finite Markov chains. J. Appl. Probab. 5, 401–413 (1968)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Spall, J.C.: Adaptive stochastic approximation by the simultaneous perturbation method. IEEE Trans. Autom. Control 45, 1839–1853 (2000)MathSciNetCrossRefMATHGoogle Scholar
  27. 27.
    Sunehag, P., Trumpf, J., Vishwanathan, S.V.N., Schraudolph, N.N.: Variable metric stochastic approximation theory. In: Proceedings of 12th International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 560–566 (2009)Google Scholar
  28. 28.
    Vazquez-Abad, F.J., Kushner, H.J.: Estimation of the derivative of a stationary measure with respect to a control parameter. J. Appl. Probab. 29, 343–352 (1992)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Xiao, X., Lionel, M.N.: Internet QoS: a big picture. IEEE Netw. 13, 8–18 (1999)CrossRefGoogle Scholar
  30. 30.
    Zhu, X., Spall, J.C.: A modified second-order SPSA optimization algorithm for finite samples. Int. J. Adapt. Control. 16, 397–409 (2002)CrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringAmrita School of EngineeringBangaloreIndia
  2. 2.Department of Computer Science and AutomationIndian Institute of ScienceBangaloreIndia

Personalised recommendations