Robotic Grasping and Manipulation Competition: Future Tasks to Support the Development of Assembly Robotics

  • Karl Van WykEmail author
  • Joe Falco
  • Elena Messina
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 816)


The Robot Grasping and Manipulation Competition, held during the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) in Daejeon, South Korea was sponsored by the IEEE Robotic and Automation Society (RAS) Technical Committee (TC) on Robotic Hands Grasping and Manipulation (RHGM) [1]. This competition was the first of a planned series of grasping and manipulation-themed events of increasing difficulty that are intended to spur technological developments and advance test methods and benchmarks so that they can be formalized for use by the community. The coupling of standardized performance testing with robot competitions will promote the use of unbiased evaluation methods to assess how well a robot system performs in a particular application space. A strategy is presented for a series of grasping and manipulation competitions that facilitate objective performance benchmarking of robotic assembly solutions. This strategy is based on test methods that can be used for more rigorous assessments and comparison of systems and components outside of the competition regime. While competitions have proven to be useful mechanisms for assessing the relative performance of robotic systems with measures of success, they often lack a methodical measurement science foundation. Consequently, scientifically sound and statistically significant metrics, measurement, and evaluation methods to quantify performance are missing. Using performance measurement methods in a condensed format will accommodate competition time limits while introducing the methods to the community as tools for benchmarking performance in the developmental and deployment phases of a robot system. The particular evaluation methods presented here are focused on the mechanical assembly process, an application space that is expected to accelerate with the new robot technologies coming to market.


Robot Grasping Manipulation Competition Benchmarks Performance measures Manufacturing 


  1. 1.
    Robotic Hands Grasping and Manipulation RAS TC. Accessed 30 June 2017
  2. 2.
    Robocup. Accessed 30 June 2017
  3. 3.
    DARPA Robotics Challenge. Accessed 30 June 2017
  4. 4.
    Amazon Picking Challenge. Accessed 25 Aug 2017
  5. 5.
    Amigoni, F., Bonarini, A., Fontana, G., Matteucci, M., Schiaffonati, V.: To what extent are competitions experiments? A critical view. Accessed 30 June 2017
  6. 6.
    Anderson, M., Jenkins, O.C., Osentoski, S.: Recasting robotics challenges as experiments. IEEE Robot. Autom. Mag. 18(2), 10–11 (2011)Google Scholar
  7. 7.
    Jacoff, A., et al.: Using competitions to advance the development of standard test methods for response robots. Association of Computing Machinery (ACM) (2012). Accessed 30 June 2017
  8. 8.
    Marvel, J., Hong, T., Messina, E.: 2011 solutions in perception challenge performance metrics and results. In: Proceedings of the Workshop on Performance Metrics for Intelligent Systems, pp. 59–63 (2012)Google Scholar
  9. 9.
    Falco, J., Van Wyk, K.: Grasping the performance: facilitating replicable performance measures via benchmarking and standardized methodologies. Robot. Autom. Mag. 22(4), 125–136 (2015)CrossRefGoogle Scholar
  10. 10.
    Performance metrics and benchmarks to advance the state of robotic grasping web portal. Accessed 30 June 2017
  11. 11.
    Downs, A., Harrison, W., Schlenoff, C.: Test methods for robot agility in manufacturing. Ind. Robot Int. J. 43(5), 563–572 (2016)CrossRefGoogle Scholar
  12. 12.
    Agile robotics for industrial automation competition (ARIAC). Accessed 30 June 2017
  13. 13.
    Shneier, S., Messina, E., Schlenoff, C., Proctor, F., Kramer, T., Falco, J.: Measuring and representing the performance of manufacturing assembly robots, NIST Internal Report 8090, November 2015Google Scholar
  14. 14.
    International federation of robotics, World Robotics Report (2016). Accessed 30 June 2017
  15. 15.
    Price Waterhouse, The new hire: How a new generation of robots is transforming manufacturing. Accessed 30 June 2017
  16. 16.
    Bostelman, R., Falco, J.: Survey of industrial manipulation technologies for autonomous assembly applications, National Institute of Standards and Technology Report, NISTIR 7844 (2012)Google Scholar
  17. 17.
    Tobe, F.: Why co-bots will be a huge innovation and growth driver for robotics industry. IEEE Spectrum, December 2015. Accessed 30 June 2017
  18. 18.
    Boothroyd, G., Dewhurst, P., Knight, W.: Product Design for Manufacture and Assembly, 3rd edn. CRC Press, Boca Raton (2011)Google Scholar

Copyright information

© This is a U.S. government work and its text is not subject to copyright protection in the United States; however, its text may be subject to foreign copyright protection 2018

Authors and Affiliations

  1. 1.National Institute of Standards and Technology (NIST)GaithersburgUSA

Personalised recommendations