Robust Tracking Occluded Human in Group by Perception Sensors Network System

  • Anh Vu Le
  • JongSuk Choi


Tracking people even being partially or fully occluded in the group situation is studied using a Perception Sensor Network (PSN) system which is composed of multiple Kinects used to detect human 3D locations, and pan tilt zoom (PTZ) cameras used to identify human faces. A method is proposed to fuse multiple detection of human in the PSN system. After associating detected human with corresponding names, the novel grouping and ungrouping algorithms are proposed. When a group of multiple human staying close together is formed, viewpoint and illumination invariant features of group members including human 3D location, height, color and binary robust invariant scalable keypoint (BRISK), retrieved from region of interest (ROI) of both depth and color images, are then stored and updated into the group database. Based on the distance between a group location at previous frame and each member location in the group at current frame, the PSN system decides whether to keep the members in the group or to ungroup them then reassign the right name among the group database by minimizing multiple criterions. The experimental results demonstrate the outperforming of the proposed method on tracking people in group than conventional methods.


Sensors network Sensor fusion Group tracking Human tracking Features matching 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



The research was partly supported by R&D programs of MOTIE (10041629 [SimonPiC] and 10077468 [DeepTasK]) and by ICT R&D programs of IITP (2015-0-00197 [LISTEN] and 2017-0-00432 [BCI]).


  1. 1.
    Goodrich, M.A., Schultz, A.C.: Human-robot interaction: a survey. Found Trends Hum-Comput Interact 1, 203–275 (2007)CrossRefzbMATHGoogle Scholar
  2. 2.
    Steinfeld, A., Fong, T., Kaber, D., Lewis, M., Scholtz, J., Schultz, A., Goodrich, M.: Common metrics for human-robot interaction. In: Proceedings of the 1St ACM SIGCHI/SIGART conference on human-robot interaction, pp. 33–40. ACM, New York (2006)Google Scholar
  3. 3.
    Ong, K.S., Hsu, Y.H., Fu, L.C.: Sensor fusion based human detection and tracking system for human-robot interaction. In: 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 4835–4840 (2012)Google Scholar
  4. 4.
    Zhang, Z.: Microsoft kinect sensor and its effect. IEEE Multimed 19, 4–10 (2012)CrossRefGoogle Scholar
  5. 5.
    Xia, L., Chen, C.C., Aggarwal, J.K.: Human detection using Depth information by Kinect. In: CVPR 2011 WORKSHOPS, pp. 15–22 (2011)Google Scholar
  6. 6.
    ROS OpenNI Kinect: ROS OpenNI open source project: (Accessed November 10, 2016) (2016)
  7. 7.
    Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.: ROS: an open-source Robot Operating SystemGoogle Scholar
  8. 8.
    Schwarz, L.A., Mkhitaryan, A., Mateus, D., Navab, N.: Human skeleton tracking from depth data using geodesic distances and optical flow. Image Vis. Comput. 30, 217–226 (2012)CrossRefGoogle Scholar
  9. 9.
    Zhao, T., Nevatia, R.: Tracking multiple humans in crowded environment. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004, vol. 2, pp. II-406-II-413 (2004)Google Scholar
  10. 10.
    Yun, X., Bachmann, E.R.: Design, implementation, and experimental results of a Quaternion-Based kalman filter for human body motion tracking. IEEE Trans. Robot. 22, 1216–1227 (2006)CrossRefGoogle Scholar
  11. 11.
    Choi, W., Pantofaru, C., Savarese, S.: Detecting and tracking people using an RGB-D camera via multiple detector fusion. In: 2011 IEEE international conference on computer vision workshops (ICCV workshops), pp. 1076–1083 (2011)Google Scholar
  12. 12.
    Munaro, M., Basso, F., Menegatti, E.: Tracking people within groups with RGB-D data. In: 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 2101–2107 (2012)Google Scholar
  13. 13.
    Benfold, B., Reid, I.: Stable multi-target tracking in real-time surveillance video. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp. 3457–3464 (2011)Google Scholar
  14. 14.
    Yamaguchi, K., Berg, A.C., Ortiz, L.E., Berg, T.L.: Who are You with and Where are You Going?. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp. 1345–1352 (2011)Google Scholar
  15. 15.
    Pellegrini, S., Ess, A., Gool, L.V.: Improving data association by joint modeling of pedestrian trajectories and groupings. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) Computer vision – ECCV 2010, pp. 452–465. Springer, Berlin (2010)Google Scholar
  16. 16.
    Zhang, C., Hamid, R., Zhang, Z.: Taylor expansion based classifier adaptation: application to person detection. In: IEEE conference on computer vision and pattern recognition, 2008. CVPR 2008, pp. 1–8 (2008)Google Scholar
  17. 17.
    Leal-Taixé, L., Pons-Moll, G., Rosenhahn, B.: Everybody needs somebody: modeling social and grouping behavior on a linear programming multiple people tracker. In: 2011 IEEE international conference on computer vision workshops (ICCV workshops), pp. 120–127 (2011)Google Scholar
  18. 18.
    Izadinia, H., Saleemi, I., Li, W., Shah, M.: (MP)2T: multiple people multiple parts tracker. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) Computer vision – ECCV 2012, pp. 100–114. Springer, Berlin (2012)Google Scholar
  19. 19.
    Breitenstein, M.D., Reichlin, F., Leibe, B., Koller-Meier, E., Gool, L.V.: Online multiperson tracking-by-detection from a single, uncalibrated camera. IEEE Trans. Pattern Anal. Mach. Intell. 33, 1820–1833 (2011)CrossRefGoogle Scholar
  20. 20.
    Berclaz, J., Fleuret, F., Turetken, E., Fua, P.: Multiple object tracking using K-Shortest paths optimization. IEEE Trans. Pattern Anal. Mach. Intell. 33, 1806–1819 (2011)CrossRefGoogle Scholar
  21. 21.
    Conte, D., Foggia, P., Percannella, G., Vento, M.: Performance evaluation of a people tracking system on PETS2009 database. In: 2010 Seventh IEEE international conference on advanced video and signal based surveillance (AVSS), pp. 119–126 (2010)Google Scholar
  22. 22.
    Berclaz, J., Fleuret, F., Fua, P.: Multiple object tracking using flow linear programming. In: 2009 Twelfth IEEE international workshop on performance evaluation of tracking and surveillance (PETS-Winter), pp. 1–8 (2009)Google Scholar
  23. 23.
    Alahi, A., Jacques, L., Boursier, Y., Vandergheynst, P.: Sparsity-Driven People Localization Algorithm: Evaluation in Crowded Scenes Environments. In: 2009 Twelfth IEEE international workshop on performance evaluation of tracking and surveillance (PETS-Winter), pp. 1–8 (2009)Google Scholar
  24. 24.
    An, K., Park, J., Hoang, M.D., Choi, J.: Dispensing materials of mobile robot cooperating with perception sensor network. In: 2014 11Th international conference on ubiquitous robots and ambient intelligence (URAI), pp. 496–499 (2014)Google Scholar
  25. 25.
    Park, J., An, K., Choi, J.: Realistic 3D simulation of multiple human recognition over perception sensor network. In: The 23Rd IEEE international symposium on robot and human interactive communication, pp. 507–512 (2014)Google Scholar
  26. 26.
    Leutenegger, S., Chli, M., Siegwart, R.Y.: BRISK: binary robust invariant scalable keypoints. In: 2011 international conference on computer vision, pp. 2548–2555 (2011)Google Scholar
  27. 27.
    Lowe, D.G.: Distinctive image features from Scale-Invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)CrossRefGoogle Scholar
  28. 28.
    Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-Up Robust features (SURF). Comput. Vis. Image Underst. 110, 346–359 (2008)CrossRefGoogle Scholar
  29. 29.
    Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: The CLEAR MOT metrics. EURASIP J. Image Video Process. 2008, 246309 (2008)CrossRefGoogle Scholar
  30. 30.
    Ferryman, J., Shahrokni, A.: An Overview of the PETS 2009 Challenge. In: Presented at the Eleventh IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, Miami (2009)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2017

Authors and Affiliations

  1. 1.Optoelectronics Research Group, Faculty of Electrical and Electronics EngineeringTon Duc Thang UniversityHo Chi Minh CityVietnam
  2. 2.Center for Robotics ResearchKorea Institute of Science and TechnologySeoulKorea

Personalised recommendations