Detection, localisation and tracking of pallets using machine learning techniques and 2D range data

Abstract

The problem of autonomous transportation in industrial scenarios is receiving a renewed interest due to the way it can revolutionise internal logistics, especially in unstructured environments. This paper presents a novel architecture allowing a robot to detect, localise, and track (possibly multiple) pallets using machine learning techniques based on an on-board 2D laser rangefinder only. The architecture is composed of two main components: the first stage is a pallet detector employing a Faster Region-Based Convolutional Neural Network (Faster R-CNN) detector cascaded with a CNN-based classifier; the second stage is a Kalman filter for localising and tracking detected pallets, which we also use to defer commitment to a pallet detected in the first stage until sufficient confidence has been acquired via a sequential data acquisition process. For fine-tuning the CNNs, the architecture has been systematically evaluated using a real-world dataset containing 340 labelled 2D scans, which have been made freely available in an online repository. Detection performance has been assessed on the basis of the average accuracy over k-fold cross-validation, and it scored 99.58% in our tests. Concerning pallet localisation and tracking, experiments have been performed in a scenario where the robot is approaching the pallet to fork. Although data have been originally acquired by considering only one pallet as per specification of the use case we consider, artificial data have been generated as well to mimic the presence of multiple pallets in the robot workspace. Our experimental results confirm that the system is capable of identifying, localising and tracking pallets with a high success rate while being robust to false positives.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Notes

  1. 1.

    www.ottomotors.com.

  2. 2.

    www.fetchrobotics.com/.

  3. 3.

    www.peerlessresearch.com/.

  4. 4.

    https://github.com/EMAROLab/PDT.

  5. 5.

    http://cs-chan.com/source/FADL/Online_Paper_Summary_Table.pdf.

References

  1. 1.

    Aref MM, Ghabcheloo R, Kolu A, Hyvonen M, Huhtala K, Mattila J (2013) Position-based visual servoing for pallet picking by an articulated-frame-steering hydraulic mobile machine. In: Proceedings of the 2013 IEEE conference on robotics, automation and mechatronics (RAM). Manila, Philippines

  2. 2.

    Aref MM, Ghabcheloo R, Kolu A, Mattila J (2016) A multistage controller with smooth switching for autonomous pallet picking. In: Proceedings of the 2016 IEEE international conference on robotics and automation (ICRA). Stockholm, Sweden

  3. 3.

    Aref MM, Ghabcheloo R, Mattila J (2014) A macro-micro controller for pallet picking by an articulated-frame-steering hydraulic mobile machine. In: Proceedings of the 2014 IEEE international conference on robotics and automation (ICRA). Hong Kong, China

  4. 4.

    Arlot S, Celisse A (2010) A survey of cross-validation procedures for model selection. Stat Surv 4:40–79

    MathSciNet  Article  Google Scholar 

  5. 5.

    Asvadi A, Garrote L, Premebida C, Peixoto P, Nunes UJ (2017) Depthcn: vehicle detection using 3D-LIDAR and ConvNet. In: IEEE 20th international conference on intelligent transportation systems (ITSC), 2017. IEEE, pp 1–6

  6. 6.

    Baglivo L, Bellomo N, Miori G, Marcuzzi E, Pertile M, De Cecco M (2008) An object localization and reaching method for wheeled mobile robots using laser rangefinder. In: Proceedings of the 2008 international IEEE conference intelligent systems (IS). Varna, Bulgaria

  7. 7.

    Baglivo L, Biasi N, Biral F, Bellomo N, Bertolazzi E, Da Lio M, De Cecco M (2011) Autonomous pallet localization and picking for industrial forklifts: a robust range and look method. Meas Sci Technol 22(8):085502

    Article  Google Scholar 

  8. 8.

    Beder C, Bartczak B, Koch R (2007) A comparison of PMD-cameras and stereo-vision for the task of surface reconstruction using patchlets. In: Proceedings of the 2007 IEEE conference on computer vision and pattern recognition (CVPR). Minneapolis, MN, USA

  9. 9.

    Bostelman R, Hong T, Chang T (2006) Visualization of pallets. In: Proceedings of SPIE, the international society for optical engineering. Boston, MA, USA

  10. 10.

    Brust CA, Sickert S, Simon M, Rodner E, Denzler J (2015) Convolutional patch networks with spatial prior for road detection and urban scene understanding. arXiv preprint arXiv:1502.06344

  11. 11.

    Byun S, Kim M (2008) Real-time positioning and orienting of pallets based on monocular vision. In: Proceedings of the 2008 IEEE international conference on tools with artificial intelligence (ICTAI). Daytona, OH, USA

  12. 12.

    Capezio F, Mastrogiovanni F, Scalmato A, Sgorbissa A, Vernazza P, Vernazza T, Zaccaria R (2011) Mobile robots in hospital environments: an installation case study. In: Proceedings of the 2011 European conference on mobile robotics (ECMR). Örebro, Sweden

  13. 13.

    Chen G, Peng R, Wang Z, Zhao W (2012) Pallet recognition and localization method for vision guided forklift. In: Proceedings of the 2012 international conference on wireless communications, networking and mobile computing (WiCOM). Shanghai, China

  14. 14.

    Cucchiara R, Piccardi M, Prati A (2000) Focus based feature extraction for pallets recognition. In: Proceedings of the 11th British machine vision conference (BMVC). Bristol, UK

  15. 15.

    Cuevas EV, Zaldivar D, Rojas R (2005) Kalman filter for vision tracking. Technical report, Freie Universität Berlin, Inst. Informatik, Berlin, Germany

  16. 16.

    Cui GZ, Lu LS, He ZD, Yao LN, Yang CX, Huang BY, Hu ZH (2010) A robust autonomous mobile forklift pallet recognition. In: Proceedings of the 2010 international Asia conference on informatics in control, automation and robotics (CAR). Wuhan, China

  17. 17.

    D’Andrea R (2012) A revolution in the warehouse: a retrospective on Kiva systems and the grand challenges ahead. IEEE Trans Autom Sci Eng 4(9):638–639

    Article  Google Scholar 

  18. 18.

    Darvish K, Wanderlingh F, Bruno B, Simetti E, Mastrogiovanni F, Casalino G (2018) Flexible human–robot cooperation models for assisted shop-floor tasks. arXiv preprint arXiv:1707.02591

  19. 19.

    Feng D, Rosenbaum L, Dietmayer K (2018) Towards safe autonomous driving: capture uncertainty in the deep neural network for lidar 3D vehicle detection. arXiv preprint arXiv:1804.05132

  20. 20.

    Garibotto G, Masciangelo S, Ilic M, Bassino P (1996) Robolift: a vision guided autonomous fork-lift for pallet handling. In: Proceedings of the 1996 IEEE/RSJ international conference on intelligent robots and systems (IROS). Osaka, Japan

  21. 21.

    Garibotto G, Masciangelo S, Ilic M, Bassino P (1997) Service robotics in logistic automation: Robolift: vision based autonomous navigation of a conventional fork-lift for pallet handling. In: Proceedings of the 1997 international conference on advanced robotics (ICAR). Monterey, CA, USA

  22. 22.

    Ghosh B (1970) Sequential tests of statistical hypotheses. Addison-Wesley, Boston

    Google Scholar 

  23. 23.

    Girshick R (2015) Fast R-CNN. In: Proceedings of the (2015) IEEE international conference on computer vision (ICCV). Santiago, Chile

  24. 24.

    Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the (2014) IEEE conference on computer vision and pattern recognition (CVPR). Washington, DC, USA

  25. 25.

    He Z, Wang X, Liu J, Sun J, Cui G (2010) Feature-to-feature based laser scan matching for pallet recognition. In: Proceedings of the (2010) international conference on measuring technology and mechatronics automation (ICMTMA). Changsha City, China

  26. 26.

    Hebert M (1986) Outdoor scene analysis using range data. In: Proceedings of the (1986) IEEE international conference on robotics and automation (ICRA). CA, USA, San Francisco

  27. 27.

    Heyer C (2010) Human--robot interaction and future industrial robotics applications. In: Proceedings of the (2010) IEEE/RSJ international conference on intelligent robots and systems (IROS). Taipei, Taiwan

  28. 28.

    Hoffman R, Jain AK (1987) Segmentation and classification of range images. IEEE Trans. Pattern Anal Mach Intell 5:608–620

    Article  Google Scholar 

  29. 29.

    Holz D, Behnke S (2016) Fast edge-based detection and localization of transport boxes and pallets in RGB-D images for mobile robot bin picking. In: Proceedings of the 2016 international symposium on robotics (ISR). Munich, Germany

  30. 30.

    Kim W, Helmick D, Kelly A (2001) Model based object pose refinement for terrestrial and space autonomy. In: Proceedings of the 6th international symposium on artificial intelligence, robotics and automation in space. Montreal, Canada

  31. 31.

    Krüger J, Lien T, Verl A (2009) Cooperation of humans and machines in the assembly lines. CIRP Ann Manuf Technol 58(2):628–646

    Article  Google Scholar 

  32. 32.

    Kuhn HW (1955) The hungarian method for the assignment problem. Nav Res Logist (NRL) 2(1–2):83–97

    MathSciNet  Article  Google Scholar 

  33. 33.

    Lecking D, Wulf O, Wagner B (2006) Variable pallet pick-up for automatic guided vehicles in industrial environments. In: Proceedings of the 2006 IEEE conference on emerging technologies and factory automation (ETFA). Prague, Czech Republic

  34. 34.

    LeCun Y, Bengio YE, Hinton G (2015) Deep learning. Nature 521:436–444

    Article  Google Scholar 

  35. 35.

    Liang M, Yang B, Wang S, Urtasun R (2018) Deep continuous fusion for multi-sensor 3D object detection. In: Proceedings of the European conference on computer vision (ECCV), pp 641–656

  36. 36.

    Mastrogiovanni F, Sgorbissa A, Zaccaria R (2004) From autonomous robots to artificial ecosystems. In: Nakashima H, Aghajan H, Augusto JC (eds) Handbook of ambient intelligence and smart environments. Springer, Boston, pp 635–668

    Google Scholar 

  37. 37.

    Mastrogiovanni F, Sgorbissa A, Zaccaria R (2004) A system for hierarchical planning in service mobile robotics. In: Proceedings of the 8th international conference on intelligent autonomous systems (IAS-8). Amsterdam, The Netherlands

  38. 38.

    Mastrogiovanni F, Sgorbissa A, Zaccaria R (2007) The more the better? A discuss about line features for self-localisation. In: Proceedings of the 2007 IEEE/RSJ international conference on intelligent robots and systems (IROS). San Diego, CA, USA

  39. 39.

    Mastrogiovanni F, Sgorbissa A, Zaccaria R (2008) Learning to extract line features: beyond split & merge. In: Proceedings of the 2008 international conference on intelligent autonomous systems (IAS-10). Baden-Baden, Germany

  40. 40.

    Mastrogiovanni F, Sgorbissa A, Zaccaria R (2009) Context assessment strategies for ubiquitous robots. In: Proceedings of the (2009) IEEE international conference on robotics and automation (ICRA). Atlanta, GA, USA

  41. 41.

    Mastrogiovanni F, Sgorbissa A, Zaccaria R (2009) Robust navigation in an unknown environment with minimal sensing and representation. IEEE Trans Syst Man Cybern Part B Cybern 39(1):212–229

    Article  Google Scholar 

  42. 42.

    Mastrogiovanni F, Sgorbissa A, Zaccaria R (2013) How the location of the range sensor affects EKF-based localisation. J Intell Robot Syst 68(2):121–145

    Article  Google Scholar 

  43. 43.

    Matti D, Ekenel HK, Thiran JP (2017) Combining LiDAR space clustering and convolutional neural networks for pedestrian detection. arXiv preprint arXiv:1710.06160

  44. 44.

    Mohamed IS (2017) Detection and tracking of pallets using a laser rangefinder and machine learning techniques. Master’s thesis, European Master on Advanced Robotics Plus (EMARO+), University of Genova

  45. 45.

    Mohamed IS, Capitanelli A, Mastrogiovanni F, Rovetta S, Zaccaria R (2019) A 2D laser rangefinder scans dataset of standard EUR pallets. Data Brief 24:103837

    Article  Google Scholar 

  46. 46.

    Newman TS, Flynn PJ, Jain AK (1993) Model-based classification of quadric surfaces. CVGIP Image Underst 58(2):235–249

    Article  Google Scholar 

  47. 47.

    Novíkov A (2001) Uniform asymptotic expansion of likelihood ratio for Markov dependent observations. Ann Inst Stat Math 53(4):799–809

    MathSciNet  Article  Google Scholar 

  48. 48.

    Nygards J, Hogstrom T, Wernersson A (2000) Docking to pallets with feedback from a sheet-of-light range camera. In: Proceedings of the 2000 IEEE/RSJ international conference on intelligent robots and systems (IROS). Takamatsu, Japan

  49. 49.

    Oh JY, Choi HS, Jung SH, Kim HS, Shin HY (2014) Development of pallet recognition system using Kkinect camera. Int J Multimed Ubiquitous Eng 9(4):227–232

    Article  Google Scholar 

  50. 50.

    Pages J, Armangué X, Salvi J, Freixenet J, Martí J (2001) A computer vision system for autonomous forklift vehicles in industrial environments. In: Proceedings of the 2001 Mediterranean conference on control and automation (MEDS). Dubrovnik, Croatia

  51. 51.

    Pfister ST, Roumeliotis SI, Burdick JW (2003) Weighted line fitting algorithms for mobile robot map building and efficient data representation. In: Proceedings of the 2003 IEEE international conference on robotics and automation ICRA. Taipei, Taiwan

  52. 52.

    Premebida C, Nunes U (2005) Segmentation and geometric primitives extraction from 2D laser range data for mobile robot applications. Robotica 2005:17–25

    Google Scholar 

  53. 53.

    Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). Las Vegas, NV, USA, pp 779–788

  54. 54.

    Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  55. 55.

    Schulenburg E, Weigel T, Kleiner A (2003) Self-localization in dynamic environments based on laser and vision data. In: Proceedings of the 2003 IEEE/RSJ international conference on intelligent robots and systems (IROS). Las Vegas, NV, USA

  56. 56.

    Seelinger M, Yoder JD (2005) Automatic pallet engagment by a vision guided forklift. In: Proceedings of the 2005 IEEE international conference on robotics and automation (ICRA). Barcelona, Spain

  57. 57.

    Syu JL, Li HT, Chiang JS, Hsia CH, Wu PH, Hsieh CF, Li SA (2017) A computer vision assisted system for autonomous forklift vehicles in real factory environment. Multimed Tools Appl 76(4):18387–18407

    Article  Google Scholar 

  58. 58.

    Teng Z, Kim JH, Kang DJ (2010) Real-time lane detection by using multiple cues. In: Proceedings of the 2010 international conference on control automation and systems (ICCAS). Gyeonggi-do, South Korea

  59. 59.

    Uijlings JR, Van De Sande KE, Gevers T, Smeulders AW (2013) Selective search for object recognition. Int J Comput Vis 104(2):154–171

    Article  Google Scholar 

  60. 60.

    Varga R, Costea A, Nedevschi S (2015) Improved autonomous load handling with stereo cameras. In: Proceedings of the 2015 IEEE international conference on intelligent computer communication and processing (ICCP). Cluj-Napoca, Romania

  61. 61.

    Varga R, Nedevschi S (2014) Vision-based autonomous load handling for automated guided vehicles. In: Proceedings of the 2014 IEEE international conference on intelligent computer communication and processing (ICCP). Cluj-Napoca, Romania, pp 239–244

  62. 62.

    Varga R, Nedevschi S (2016) Robust pallet detection for automated logistics operations. In: Proceedings of the 11th joint conference on computer vision, imaging and computer graphics theory and applications (VISIGRAPH), volume 4: VISAPP. Rome, Italy, pp. 470–477

  63. 63.

    Wald A (1945) Sequential tests of statistical hypotheses. Ann Math Stat 16(2):117–186

    MathSciNet  Article  Google Scholar 

  64. 64.

    Walter MR, Antone M, Chuangsuwanich E, Correa A, Davis R, Fletcher L, Frazzoli E, Friedman Y, Glass J, How JP (2015) A situationally aware voice-commandable robotic forklift working alongside people in unstructured outdoor environments. J Field Robot 32(4):590–628

    Article  Google Scholar 

  65. 65.

    Walter MR, Karaman S, Frazzoli E, Teller S (2010) Closed-loop pallet manipulation in unstructured environments. In: Proceedings of the 2010 IEEE/RSJ international conference on intelligent robots and systems (IROS). Taipei, Taiwan, pp 5119–5126

  66. 66.

    Wang S, Ye A, Guo H, Gu J, Wang X, Yuan K (2016) Autonomous pallet localization and picking for industrial forklifts based on the line structured light. In: Proceedings of the 2016 IEEE international conference on mechatronics and automation (ICMA). Harbin, China

  67. 67.

    Weichert F, Skibinski S, Stenzel J, Prasse C, Kamagaew A, Rudak B, Ten Hompel M (2013) Automated detection of euro pallet loads by interpreting pmd camera depth images. Logist Res 6(2–3):99–118

    Article  Google Scholar 

  68. 68.

    Zhou Y, Tuzel O (2017) Voxelnet: end-to-end learning for point cloud based 3D object detection. arXiv preprint arXiv:1711.06396

Download references

Acknowledgements

The research leading to these results has received funding from the POR/FESR Liguria regional funding scheme, under Grant Agreement Number 56 (AIRONE).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Ihab S. Mohamed.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mohamed, I.S., Capitanelli, A., Mastrogiovanni, F. et al. Detection, localisation and tracking of pallets using machine learning techniques and 2D range data. Neural Comput & Applic 32, 8811–8828 (2020). https://doi.org/10.1007/s00521-019-04352-0

Download citation

Keywords

  • Pallet detection
  • Automated guided vehicle
  • 2D laser rangefinder
  • Faster R-CNN
  • Computer vision