Reinforcement Learning Aided Robot-Assisted Navigation: A Utility and RRT Two-Stage Approach

  • Luís GarroteEmail author
  • João Paulo
  • Urbano J. Nunes


This work proposes a robot-assisted navigation approach based on user intent adjustment, in the context of robotic walkers. Walkers are prescribed to users with gait disorders so that they can support their body weight on the upper limbs, however, the manipulation of such devices can be cumbersome for some users. Common problems for the users are lack of dexterous upper limb control and visual impairments. These problems can render walkers’ users helpless, making them unable to operate these devices effectively and efficiently. We present a new approach to robot-assisted navigation using a utility decision and safety analysis procedure with user intent adjustments learned by reinforcement learning (RL) and supported on a rapidly-exploring random tree inspired algorithm. The proposed approach offers full control of the assistive platform to the user until obstacles are detected. In dangerous scenarios, corrections are computed in order that the assistive platform avoids collisions and follows social norms, effectively guiding the user through the environment while enforcing safer routes. The experimental validation was carried out in a virtual environment and in a real world scenario using a robotic walker built in our lab (ISR-AIWALKER). Experimental results have shown that the proposed approach provides a reliable solution to the robot-assisted navigation of a robotic walker, in particular the use of utility theory to evaluate candidate motions together with a RL model increases the safety of the user’s navigation.


Assistive robotics Reinforcement learning Robot-assisted navigation Robotic walker Rapidly-exploring random tree Robot operating system 



This work was supported by the Portuguese Foundation for Science and Technology (FCT) under the Ph.D. Grant with reference SFRH/BD/88459/2012 with funds from QREN–POPH and the European Social Fund from the European Union. It was also partially supported by the Project funded by FCT with reference FCT/COMPETE 2020/P2020 through Grants UID/EEA/00048/2013 and MATIS - CENTRO-01-0145-FEDER-000014, Portugal.

Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. 1.
    Balash Y, Hadar-Frumer M, Herman T, Peretz C, Giladi N, Hausdorff J (2007) The effects of reducing fear of falling on locomotion in older adults with a higher level gait disorder. J Neural Trans 114(10):1309–1314CrossRefGoogle Scholar
  2. 2.
    Bateni H, Maki BE (2005) Assistive devices for balance and mobility: benefits, demands, and adverse consequences. Arch Phys Med Rehabil 86(1):134–145CrossRefGoogle Scholar
  3. 3.
    Burns B, Brock O (2007) Single-query motion planning with utility-guided random trees. In: 2007 IEEE international conference on robotics and automation, pp 3307–3312Google Scholar
  4. 4.
    Chuy Jr O, Hirata Y, Wang Z, Kosuge K (2005) Motion control algorithms for a new intelligent robotic walker in emulating ambulatory device function. In: 2005 IEEE international conference mechatronics and automationGoogle Scholar
  5. 5.
    Constantinescu R, Leonard C, Deeley C, Kurlan R (2007) Assistive devices for gait in Parkinson’s disease. Parkinsonism Relat Disord 13(3):133–138CrossRefGoogle Scholar
  6. 6.
    Duan Y, Cui B, Yang H (2008) Robot navigation based on fuzzy RL algorithm. In: International symposium on neural networks. Springer, Berlin, pp 391–399Google Scholar
  7. 7.
    Dubowsky S, Genot F, Godding S, Kozono H, Skwersky A, Yu H, Yu LS (2000) PAMM: a robotic aid to the elderly for mobility assistance and monitoring: a helping-hand for the elderly. In: IEEE International conference on robotics and automationGoogle Scholar
  8. 8.
    Ferrari F, Divan S, Guerrero C, Zenatti F, Guidolin R, Palopoli L, Fontanelli D (2019) Human-robot interaction analysis for a smart walker for elderly: the acanto interactive guidance system. Int J Soc Robot. Google Scholar
  9. 9.
    Garrote L, Paulo J, Perdiz J, Peixoto P, Nunes UJ (2018) Robot-assisted navigation for a robotic walker with aided user intent. In: 2018 27th IEEE international symposium on robot and human interactive communication (RO-MAN), pp 348–355Google Scholar
  10. 10.
    Garrote L, Rosa J, Paulo J, Premebida C, Peixoto P, Nunes U (2017) 3D point cloud downsampling for 2D indoor scene modelling in mobile robotics. In: 2017 IEEE international conference on autonomous robot systems and competitions (ICARSC)Google Scholar
  11. 11.
    Geravand M, Werner C, Hauer K, Peer A (2016) An integrated decision making approach for adaptive shared control of mobility assistance robots. Int J Soc Robot 8(5):631–648CrossRefGoogle Scholar
  12. 12.
    Khriji L, Touati F, Benhmed K, Al-Yahmedi A (2011) Mobile robot navigation based on Q-learning technique. Int J Adv Robot Syst 8(1):4CrossRefGoogle Scholar
  13. 13.
    Kirby R, Simmons R, Forlizzi J (2009) COMPANION: a constraint-optimizing method for person-acceptable navigation. In: IEEE international symposium on robot and human interactive communication (RO-MAN)Google Scholar
  14. 14.
    Kober J, Bagnell JA, Peters J (2013) Reinforcement learning in robotics: a survey. Int J Robot Res 32(11):1238–1274CrossRefGoogle Scholar
  15. 15.
    Lacey G, Namara SM, Dawson-Howe KM (1998) Personal adaptive mobility aid for the infirm and elderly blind. In: Mittal VO, Yanco HA, Aronis J, Simpson R (eds) Assistive technology and artificial intelligence: applications in robotics, user interfaces and natural language processing. Springer, Berlin, Heidelberg, pp 211–220. CrossRefGoogle Scholar
  16. 16.
    Lavalle SM (1998) Rapidly-exploring random trees: a new tool for path planning. Report No. TR 98-11, Computer Science Department, Iowa State University Google Scholar
  17. 17.
    Lee G, Jung EJ, Ohnuma T, Chong NY, Yi BJ (2011) JAIST robotic walker control based on a two-layered Kalman filter. In: 2011 IEEE international conference on robotics and automation (ICRA)Google Scholar
  18. 18.
    Lopes A, Rodrigues J, Perdigao J, Pires G, Nunes U (2016) A new hybrid motion planner: applied in a brain-actuated robotic wheelchair. IEEE Robot Autom Mag 23(4):82–93CrossRefGoogle Scholar
  19. 19.
    Lutz W, Sanderson W, Scherbov S (2008) The coming acceleration of global population ageing. Nature 451(7179):716–719CrossRefGoogle Scholar
  20. 20.
    Makarenko, AA, Williams SB, Bourgault F, Durrant-Whyte HF (2002) An experiment in integrated exploration. In: IEEE/RSJ international conference on intelligent robots and systems, 2002, vol 1, pp 534–539Google Scholar
  21. 21.
    McLachlan S, Arblaster J, Liu D, Miro JV, Chenoweth L (2005) A multi-stage shared control method for an intelligent mobility assistant. In: 9th international conference on rehabilitation robotics, 2005. ICORR 2005, pp 426–429Google Scholar
  22. 22.
    Morris A, Donamukkala R, Kapuria A, Steinfeld A, Matthews JT, Dunbar-Jacob J, Thrun, S (2003) A robotic walker that provides guidance. In: 2003 IEEE international conference on robotics and automation, vol 1, pp 25–30Google Scholar
  23. 23.
    Mou WH, Chang MF, Liao CK, Hsu YH, Tseng SH, Fu LC (2012) Context-aware assisted interactive robotic walker for Parkinson’s disease patients. In: 2012 IEEE/RSJ international conference on intelligent robots and systems (IROS)Google Scholar
  24. 24.
    Moustris GP, Geravand M, Tzafestas C, Peer A (2016) User-adaptive shared control in a mobility assistance robot based on human-centered intention reading and decision making scheme. In: IEEE ICRA workshop: human–robot interfaces for enhanced physical interactionsGoogle Scholar
  25. 25.
    Paulo J, Peixoto P, Nunes U (2017) ISR-AIWALKER: robotic walker for intuitive and safe mobility assistance and gait analysis. IEEE Trans Hum Mach Syst 47:1110–1122CrossRefGoogle Scholar
  26. 26.
    Pérez-Higueras N, Caballero F, Merino L (2018) Teaching robot navigation behaviors to optimal RRT planners. Int J Soc Robot 10(2):235–249CrossRefGoogle Scholar
  27. 27.
    Ramrattan RS, Wolfs RC, Panda-Jonas S, Jonas JB, Bakker D, Pols HA, Hofman A, de Jong PT (2001) Prevalence and causes of visual field loss in the elderly and associations with impairment in daily functioning: the Rotterdam study. Arch Ophthalmol 119(12):1788–1794CrossRefGoogle Scholar
  28. 28.
    Russell S, Norvig P (2009) Artificial intelligence: a modern approach, 3rd edn. Prentice Hall Press, Upper Saddle RiverzbMATHGoogle Scholar
  29. 29.
    Simpson R, Boninger ML (2008) Clinical evaluation of guido robotic walker. J Rehabil Res Dev 45(9):1281CrossRefGoogle Scholar
  30. 30.
    Simpson R, LoPresti E, Cooper R (2008) How many people would benefit from a smart wheelchair? J Rehabil Res Dev 45(1):53–72CrossRefGoogle Scholar
  31. 31.
    Truong XT, Ngo TD (2016) Dynamic social zone based mobile robot navigation for human comfortable safety in social environments. Int J Soc Robot 8(5):663–684CrossRefGoogle Scholar
  32. 32.
    Wachaja A, Agarwal P, Zink M, Adame MR, Möller K, Burgard W (2017) Navigating blind people with walking impairments using a smart walker. Auton Robots 41(3):555–573CrossRefGoogle Scholar
  33. 33.
    Xu W, Huang J, Yan Q (2015) Multi-sensor based human motion intention recognition algorithm for walking-aid robot. In: 2015 IEEE international conference on robotics and biomimetics (ROBIO), pp 2041–2046Google Scholar
  34. 34.
    Yu KT, Lam CP, Chang MF, Mou WH, Tseng SH, Fu LC (2010) An interactive robotic walker for assisting elderly mobility in senior care unit. In: 2010 IEEE workshop on advanced robotics and its social impacts (ARSO)Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Department of Electrical and Computer Engineering, Institute of Systems and RoboticsUniversity of CoimbraCoimbraPortugal

Personalised recommendations