Development of Fast Parallel Algorithms Based on Visual and Audio Information in Motion Control Systems of Mobile Robots

Part of the Intelligent Systems Reference Library book series (ISRL, volume 136)


Decision making for movement is one of the essential activities in motion control systems of mobile robots. It is based on methods and algorithms of data processing obtained from the mobile robot sensors, usually video and audio sensors, like video cameras and microphone arrays. After image processing, information about the objects and persons including their current positions in area of mobile robot observation can be obtained. The aim of methods and algorithms is to achieve the appropriate precision and effectiveness of mobile robot’s visual perception, as well as the detection and tracking of objects and persons applying the mobile robot motion path planning. The precision in special cases of visual speaking person’s detection and tracking can be augmented adding the information of sound arrival in order to receive and execute the voice commands. There exist algorithms using only visual perception and attention or also the joined audio perception and attention. These algorithms are usually tested in the most cases as simulations and cannot provide a real time tracking objects and people. Therefore, the goal in this chapter is to develop and test the fast parallel algorithms for decision making in the motion control systems of mobile robots. The depth analysis of the existing methods and algorithms was conducted, which provided the main ways to increase the speed of an algorithm, such as the optimization, simplification of calculations, applying high level programming languages, special libraries for image and audio signal processing based on the hybrid hardware and software implementations, using processors like Digital Signal Processor (DSP) and Field-Programmable Gate Array (FPGA). The high speed proposed algorithms were implemented in the parallel computing multiprocessor hardware structure and software platform using the well known NVIDIA GPU processor and GUDA platform, respectively. The experimental results with different parallel structures confirm the real time execution of algorithms for the objects and speaking person’s detection and tracking using the given mobile robot construction.


Visual and audio decision making Mobile robot Motion control system Visual and audio perception and attention Parallel algorithm GPU CUDA 


  1. 1.
    Siegwart, R., Nourbakhsh, I.R.: Introduction to Autonomous Mobile Robots. The MIT Press (2004)Google Scholar
  2. 2.
    Erdmann, M., Hsu, D., Overmars, M., Van der Stappen, A.F. (eds.): Algorithmic Foundations of Robotics VI. Springer, Heidelberg (2005)Google Scholar
  3. 3.
    Siciliano, Br., Khatib, O. (eds.): Springer Handbook of Robotics. Springer, Heidelberg (2008)Google Scholar
  4. 4.
    Dehkharghani, S.S., Bekiarski, A., Pleshkova, S.: Application of probabilistic methods in mobile robots audio visual motion control combined with laser range finder distance measurements. In: 11th WSEAS International Conference on Circuits, Systems, Electronics, Control & Signal Processing (CSECS’2012), pp. 91–98 (2012)Google Scholar
  5. 5.
    Dehkharghani, S.S., Bekiarski, A., Pleshkova, S.: Method and algorithm for precise estimation of joined audio visual robot control. In: Iran’s 3rd International Conference on Industrial Automation (2013)Google Scholar
  6. 6.
    Venkov, P., Bekiarski, Al., Dehkharghani, S.S., Pleshkova, Sn.: Search and tracking of targets with mobile robot by using audio-visual information. In: International Conference on Automation and Informatics (CAI’2010), pp. 463–469 (2010)Google Scholar
  7. 7.
    Al, Bekiarski: Visual mobile robots perception for motion control. In: Kountchev, R., Nakamatsu, K. (eds.) Advances in Reasoning-Based Image Processing Intelligent Systems, ISRL, vol. 29, pp. 173–209. Springer, Berlin (2012)CrossRefGoogle Scholar
  8. 8.
    Pleshkova, Sn., Bekiarski, Al.: Audio visual attention models in the mobile robots navigation. In: Kountchev, R., Nakamatsu, K. (eds.) New Approaches in Intelligent Image Analysis, ISRL, vol. 108, pp. 253–294. Springer International Publishing, Switzerland (2016)Google Scholar
  9. 9.
    Dehkharghani, S.S., Pleshkova, S.: Geometric thermal infrared camera calibration for target tracking by a mobile robot. Comptes rendus de l’Academie bulgare des Sciences 67(1), 109–114 (2014)Google Scholar
  10. 10.
    Bekiarski, Al., Pleshkova, Sn.: Microphone array beamforming for mobile robot. In: 8th WSEAS International Conference on Circuits, Systems, Electronics, Control & Signal Processing (CSECS’2009), pp. 146–149 (2009)Google Scholar
  11. 11.
    Dehkharghani, S.S.: Development of methods and algorithms for audio-visual mobile robot motion control. Doctoral Thesis, Technical University, Sofia, Bulgaria (2013)Google Scholar
  12. 12.
    Cebenoyan, C.: GPU Computing for Games. Available from: Accessed 6 May 2017
  13. 13.
    NVIDIA CUDA Compute Unified Device Architecture. Programming Guide. Version 1.0. Available from: Accessed 6 May 2017
  14. 14.
    Nukada, A., Matsuoka, S.: Auto-tuning 3-D FFT library for CUDA GPUs. In: Conference for High Performance Computing Networking, Storage and Analysis, pp. 30.1–30.10 (2009)Google Scholar
  15. 15.
    Kijsipongse, E., U-ruekolan, S., Ngamphiw, C., Tongsima, S.: Efficient large Pearson correlation matrix computing using hybrid MPI/CUDA. In: Computer Science and Software Engineering (JCSSE’2011), pp. 237–241 (2011)Google Scholar
  16. 16.
    Podlozhnyuk, V.: Image Convolution with CUDA. NVIDIA. Available from: Accessed 6 May 2017
  17. 17.
  18. 18.
    Cypher, R., Sanz, J.L.C.: The SIMD Model of Parallel Computation. Springer Publishing Company, Incorporated (2011)Google Scholar
  19. 19.
    NVIDIA CUDA Architecture. Introduction & Overview. Version 1.1 Available from: Accessed 6 May 2017
  20. 20.
    Sinha, R.S., Singh, S., Singh, S., Banga, V.K.: Accelerating genetic algorithm using general purpose GPU and CUDA. International Journal of Computer Graphics 7(1), 17–30 (2016)CrossRefGoogle Scholar
  21. 21.
    NVIDIA Developer CUDA Toolkit Documentation. Available from: Accessed 6 May 2017
  22. 22.
    Ding, C., Karlsson, C., Liu, H., Davies, T., Chen, Z.: Matrix multiplication on GPUs with on-line fault tolerance. In: 9th IEEE Int Symposium Parallel and Distributed Processing with Applications, IEEE Compute Society, pp. 311–317 (2011)Google Scholar
  23. 23.
    Restocchi, V.: Matrix-matrix multiplication on the GPU with NVIDIA CUDA. Available from: (2015). Accessed 6 May 2017
  24. 24.
    Fatahalian, K., Sugerman, J., Hanrahan, P.: Understanding the Efficiency of GPU algorithms for matrix-matrix multiplication. In: Akenine-Möller, T., McCool, M. (eds.) Graphics Hardware: The Eurographics Association, Stanford University, pp. 1–5 (2004)Google Scholar
  25. 25.
    CUDA—Matrix Multiplication. Available from: Accessed 6 May 2017
  26. 26.
    Ramey, W.: Languages, APIs and Development Tools for GPU Computing. NVIDIA Corporation. Available from: Accessed 6 May 2017
  27. 27.
    Java bindings for CUDA. Available from: Accessed 6 May 2017
  28. 28.
  29. 29.
    NVIDIA Developer CUDA Toolkit Documentation. Accessed 6 May 2017
  30. 30.
    Gembris, D., Neeb, M., Gipp, M., Kugel, A., Männer, R.: Correlation analysis on GPU systems using NVIDIA’s CUDA. J. Real-Time Image Proc. 6(4), 275–280 (2011)CrossRefGoogle Scholar
  31. 31.
    Papamakariosa, G., Rizosa, G., Pitsianisab, N.P., Sunb, X.: Fast computation of local correlation coefficients on graphics processing units. In: Luk, F.T., Schmalz, M.S., Ritter, G.X., Barrera, J., Astola, J.T. (eds.) Mathematics for Signal and Information Processing. Proceedings of SPIE, vol. 7444, pp. 12.1–12.8 (2010)Google Scholar
  32. 32.
    Podlozhnyuk V. Image Convolution with CUDA. Available from: Accessed 6 May 2017
  33. 33.
    Iandola, F., Sheffield, D., Anderson, M., Phothilimthana, P.M., Keutzer, K.: Communication-minimizing 2D convolution in GPU registers. In: IEEE International Conference on Image Processing (ICIP’2013), pp. 2116–2120 (2013)Google Scholar
  34. 34.
    Hizook (Robotics news for Academics and professionals). Available from: Accessed 6 May 2017
  35. 35.
    Hokuyo. Available from: Accessed 6 May 2017
  36. 36.
    Pomerleau, D.A. Neural Nertwork perception for mobile robot guidance. Springer Science+Business Media LLC (1993)Google Scholar
  37. 37.
    Wang, Zh., Dissanayake, G., Huang, Sh.: Simultaneous Localization and Mapping: Exactly Sparse Information Filters. World Scientific Publishing Co. Pte. Ltd (2011)Google Scholar
  38. 38.
    Liénard, J.S.: Variability, ambiguity and attention: a perception model based on analog induction. In: Cantoni, V., di Gesù, V., Setti, A., Tegolo, D. (eds.) Attention in Human and Machine Perception 2: Emergence, Attention, and Creativity. Springer Science+Business Media LLC, pp. 87–99 (1999)Google Scholar
  39. 39.
    Levorato, R., Pagello, E.: DOA acoustic source localization in mobile robot sensor networks. IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC’2015), pp. 71–76 (2015)Google Scholar
  40. 40.
    Chou, Y., Liu, J.: A robotic indoor 3D mapping system using a 2D laser range finder mounted on a rotating four-bar linkage of a mobile platform. Int. J. Adv. Rob. Syst. 10, 257–271 (2013)CrossRefGoogle Scholar
  41. 41.
    Martínez-Gómez, J., Fernández-Caballero, A., García-Varea, I., Rodríguez, L., Romero-González, Cr.: A taxonomy of vision systems for ground mobile robots. Int. J. Adv. Robot. Syst. 11.1–11.11 (2014)Google Scholar
  42. 42.
    Naminski, M.R.: An analysis of simultaneous localization and mapping (SLAM) algorithms. Macalester College, Honors Projects Mathematics, Statistics, and Computer Science (2013)Google Scholar
  43. 43.
    Thrun, S., Burgar, W., Fox, D.: Probabilistic Robotics. Massachusetts Institute of Technology (2006)Google Scholar
  44. 44.
    Michiel, H.: Jacobian. Encyclopedia of Mathematics. Springer, Berlin (2001)Google Scholar
  45. 45.
    Dmochowski, Benesty, J.: Direction of arrival estimation using the parameterized spatial correlation matrix. IEEE Trans Audio Speech Lang. Process. 15(4), 1327–1339 (2007)CrossRefGoogle Scholar
  46. 46.
    Chen, J., Huang, Y., Benesty, J.: Time delay estimation. Audio signal processing for next-generation multimedia communication systems. In: Huang, Y., Benesty, J. (eds.) Audio Signal Processing for Next-Generation Multimedia Communication Systems, pp. 197–227. Kluwer Academic Publishers (2004)Google Scholar
  47. 47.
    Dvorkind, T., Gannot, S.: The time difference of arrival estimation of speech source in a noisy and reverberant environment. Sig. Process. 85(1), 177–204 (2005)CrossRefMATHGoogle Scholar
  48. 48.
    Vonderschen, K., Wagner, H.: Detecting interaural time differences and remodeling their representation. Trends Neurosci. 37(5), 289–300 (2014)CrossRefGoogle Scholar
  49. 49.
    Tashev, I.: Sound Capture and Processing. Wiley (2009)Google Scholar
  50. 50.
    Dhull, S., Arya, S., Sahu, O.P.: Comparison of time-delay estimation techniques in acoustic environment. Int. J. Comput. Appl. 8(9), 29–31 (2010)Google Scholar
  51. 51.
    Dmochowski, J., Benesty, J.: A generalized steered response power method for computationally viable source localization. Trans. Audio Speech Lang. Process. 15(8), 2510–2526 (2007)CrossRefGoogle Scholar
  52. 52.
    DiBiase, J.H.: A high-accuracy, low-latency technique for talker localization in reverberant environments using microphone arrays. Ph.D. thesis, Brown University (2000)Google Scholar
  53. 53.
    Surveyor SRV-1 Blackfin Robot Review. Available from: Accessed 6 May 2017
  54. 54.
    Surveyor SRV-1 Blackfin Camera Board. Available from: Accessed 6 May 2017
  55. 55.
  56. 56.
    Scanning range finder URG-04LX-UG01. Available from: Accessed 6 May 2017
  57. 57.
    Geforce. Available from: Accessed 6 May 2017
  58. 58.
    NVIDIA GeForce 920 M. Available from: 6 May 2017
  59. 59.
    CUDA Toolkit 8.0. Available from: Accessed 6 May 2017
  60. 60.
    Microsoft Visual Studio Professional Version 2015. Available from: Accessed 6 May 2017
  61. 61.
    Matlab Release 2016b. Available from: Accessed 6 May 2017
  62. 62.
    CUDA C Programming Guide. Available from: Accessed 6 May 2017
  63. 63.
    MATLAB GPU Computing Support for NVIDIA CUDA-Enabled GPUs. Available from: Accessed 6 May 2017
  64. 64.
    Parallel Computing Toolbox. Available from: Accessed 6 May 2017

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Technical UniversitySofiaBulgaria

Personalised recommendations