Journal of Intelligent & Robotic Systems

, Volume 95, Issue 1, pp 149–164 | Cite as

Evaluation of approach strategies for harvesting robots: Case study of sweet pepper harvesting

Category: (5)
  • Ola RingdahlEmail author
  • Polina KurtserEmail author
  • Yael Edan
Open Access


Robotic harvesters that use visual servoing must choose the best direction from which to approach the fruit to minimize occlusion and avoid obstacles that might interfere with the detection along the approach. This work proposes different approach strategies, compares them in terms of cycle times, and presents a failure analysis methodology of the different approach strategies. The different approach strategies are: in-field assessment by human observers, evaluation based on an overview image using advanced algorithms or remote human observers, or attempting multiple approach directions until the fruit is successfully reached. In the latter approach, each attempt costs time, which is a major bottleneck in bringing harvesting robots into the market. Alternatively, a single approach strategy that only attempts one direction can be applied if the best approach direction is known a-priori. The different approach strategies were evaluated for a case study of sweet pepper harvesting in laboratorial and greenhouse conditions. The first experiment, conducted in a commercial greenhouse, revealed that the fruit approach cycle time increased 8% and 116% for reachable and unreachable fruits respectively when the multiple approach strategy was applied, compared to the single approach strategy. The second experiment measured human observers’ ability to provide insights to approach directions based on overview images taken in both greenhouse and laboratorial conditions. Results revealed that human observers are accurate in detecting unapproachable directions while they tend to miss approachable directions. By detecting fruits that are unreachable (via automatic algorithms or human operators), harvesting cycle times can be significantly shortened leading to improved commercial feasibility of harvesting robots.


Agricultural robotics Robotic harvesting Fruit approach Human-robot collaboration 



This research was supported by the European Commission (SWEEPER GA no. 644313) and by Ben-Gurion University of the Negev through the Helmsley Charitable Trust, the Agricultural, Biological and Cognitive Robotics Initiative, the Marcus Endowment Fund and the Rabbi W. Gunther Plaut Chair in Manufacturing Engineering.


  1. 1.
    Bac, C.W.: Improving Obstacle Awareness for Robotic Harvesting of Sweet-Pepper. PhD thesis, Wageningen University (2015)Google Scholar
  2. 2.
    Bac, C.W., Hemming, J., van Tuijl, B.A.J., Barth, R., Wais, E., van Henten, E.J.: Performance evaluation of a harvesting robot for sweet pepper. J. Field Rob. 34(6), 1123–1139 (2017). CrossRefGoogle Scholar
  3. 3.
    Bac, C.W., Hemming, J., Van Henten, E.J.: Stem localization of sweet-pepper plants using the support wire as a visual cue. Comput. Electron. Agric. 105, 111–120 (2014). CrossRefGoogle Scholar
  4. 4.
    Bac, C.W., Henten, E.J., Hemming, J., Edan, Y.: Harvesting robots for high-value crops: state-of-the-art review and challenges ahead. J. Field Rob. 31(6), 888–911 (2014)CrossRefGoogle Scholar
  5. 5.
    Bac, C.W., Roorda, T., Reshef, R., Berman, S., Hemming, J., van Henten, E.J.: Analysis of a motion planning problem for sweet-pepper harvesting in a dense obstacle environment. Biosyst. Eng. 146, 85–97 (2016)CrossRefGoogle Scholar
  6. 6.
    Barth, R., Hemming, J., van Henten, E.J.: Design of an eye-in-hand sensing and servo control framework for harvesting robotics in dense vegetation. Biosyst. Eng. 146, 71–84 (2016). CrossRefGoogle Scholar
  7. 7.
    Bechar, A., Edan, Y.: Human-robot collaboration for improved target recognition of agricultural robots. Ind. Robot Int. J. 30(5), 432–436 (2003)CrossRefGoogle Scholar
  8. 8.
    Berenstein, R., Edan, Y.: Human-Robot Cooperative precision spraying: Collaboration levels and optimization function. IFAC Proceedings 45(22), 799–804 (2012)CrossRefGoogle Scholar
  9. 9.
    Bontsema, J., Hemming, J., Pekkeriet, E., Saeys, W., Edan, Y., Shapiro, A., Hočevar, M., Oberti, R., Armada, M., Ulbrich, H., et al.: CROPS: Clever robots for crops. Eng. Technol. Ref. 1(1), 11 (2015). Google Scholar
  10. 10.
    Chang, W.C.: Precise positioning of binocular eye-to-hand robotic manipulators. J. Intell. Robot. Syst. 49(3), 219–236 (2007)CrossRefGoogle Scholar
  11. 11.
    Comba, L., Gay, P., Piccarolo, P., Ricauda Aimonino, D.: Robotics and automation for crop management : trends and perspective. In: International Conference Ragusa SHWA2010, pp. 471–478 (2010)Google Scholar
  12. 12.
    Edan, Y., Engel, B.A., Miles, G.E.: Intelligent control system simulation of an agricultural robot. J. Intell. Robot. Syst. 8(2), 267–284 (1993)CrossRefGoogle Scholar
  13. 13.
    Edan, Y., Flash, T., Peiper, U.M., Shmulevich, I., Sarig, Y.: Near-minimum-time task planning for fruit-picking robots. IEEE Trans. Robot. Autom. 7(1), 48–56 (1991)CrossRefGoogle Scholar
  14. 14.
    Eizicovits, D., Berman, S.: Efficient sensory-grounded grasp pose quality mapping for gripper design and online grasp planning. Robot. Auton. Syst. 62(8), 1208–1219 (2014). CrossRefGoogle Scholar
  15. 15.
    Eizicovits, D., van Tuijl, B., Berman, S., Edan, Y.: Integration of perception capabilities in gripper design using graspability maps. Biosyst. Eng. 146, 98–113 (2016). CrossRefGoogle Scholar
  16. 16.
    Fleiss, J.L., Cohen, J.: The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ. Psychol. Meas. 33(3), 613–619 (1973)CrossRefGoogle Scholar
  17. 17.
    Font, D., Pallejà, T., Tresanchez, M., Teixidó, M., Martinez, D., Moreno, J., Palacín, J.: Counting red grapes in vineyards by detecting specular spherical reflection peaks in RGB images obtained at night with artificial illumination. Comput. Electron. Agric. 108, 105–111 (2014). CrossRefGoogle Scholar
  18. 18.
    Gongal, A., Amatya, S., Karkee, M., Zhang, Q., Lewis, K.: Sensors and systems for fruit detection and localization: a review. Comput. Electron. Agric. 116, 8–19 (2015)CrossRefGoogle Scholar
  19. 19.
    Han, K.S., Kim, S.C., Lee, Y.B., Kim, S.C., Im, D.H., Choi, H.K., Hwang, H.: Strawberry harvesting robot for bench-type cultivation. J. Biosyst. Eng. 37(1), 65–74 (2012)CrossRefGoogle Scholar
  20. 20.
    Harel, B., Kurtser, P., Van Herck, L., Parmet, Y., Edan, Y.: Sweet pepper maturity evaluation via multiple viewpoints color analyses. In: Agen (2016)Google Scholar
  21. 21.
    Hellström, T., Ringdahl, O.: A software framework for agricultural and forestry robots. Ind. Robot Int. J. 40(1), 20–26 (2013). CrossRefGoogle Scholar
  22. 22.
    Hemming, J., Ruizendaal, J., Hofstee, J.W., van Henten, E.J.: Fruit detectability analysis for different camera positions in sweet-pepper. Sensors 14(4), 6032–6044 (2014)CrossRefGoogle Scholar
  23. 23.
    Hosmer, D.W. Jr, Lemeshow, S., Sturdivant, R.X.: Applied Logistic Regression, vol. 398. Wiley, Hoboken (2013)CrossRefzbMATHGoogle Scholar
  24. 24.
    Kapach, K., Barnea, E., Mairon, R., Edan, Y., Ben-Shahar, O.: Computer vision for fruit harvesting robots–state of the art and challenges ahead. Int. J. Comput. Vision Robot. 3(1-2), 4–34 (2012)CrossRefGoogle Scholar
  25. 25.
    Kitamura, S., Oka, K.: Recognition and cutting system of sweet pepper for picking robot in greenhouse horticulture. In: IEEE International Conference Mechatronics and Automation, 2005, vol. 4, pp. 1807–1812. (2005)
  26. 26.
    Kurtser, P., Arad, B., Ben Shahar, O., van Bree, M., Moonen, J., van Tujil, B., Edan, Y.: Robotic data acquisition of sweet pepper images for research and development. In: Israeli Conference on Robotics (2016)Google Scholar
  27. 27.
    Landis, J.R., Koch, G.G.: The Measurement of Observer Agreement for Categorical Data. Biometrics pp. 159–174 (1977)Google Scholar
  28. 28.
    Lass, L.W., Prather, T.S.: Detecting the locations of brazilian pepper trees in the everglades with a hyperspectral sensor. Weed Technol. 18(2), 437–442 (2004)CrossRefGoogle Scholar
  29. 29.
    Lehnert, C., Sa, I., McCool, C., Upcroft, B., Perez, T.: Sweet pepper pose detection and grasping for automated crop harvesting. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 2428–2434. IEEE (2016)Google Scholar
  30. 30.
    Mann, M.P., Zion, B., Shmulevich, I., Rubinstein, D., Linker, R.: Combinatorial optimization and performance analysis of a multi-arm cartesian robotic fruit harvester-extensions of graph coloring. J. Intell. Robot. Syst. 82(3-4), 399–411 (2016)CrossRefGoogle Scholar
  31. 31.
    McCool, C., Sa, I., Dayoub, F., Lehnert, C., Perez, T., Upcroft, B.: Visual detection of occluded crop: for automated harvesting. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 2506–2512 (2016)Google Scholar
  32. 32.
    Ringdahl, O., Kurtser, P., Barth, R., Edan, Y.: Operational flow of an autonomous sweetpepper harvesting robot. In: The 5th Israeli Conference on Robotics 2016, 13-14 April 2016. Air Force Conference Center Hertzilya, Israel (2016)Google Scholar
  33. 33.
    Ringdahl, O., Kurtser, P., Edan, Y.: Strategies for selecting best approach direction for a sweet-pepper harvesting robot. In: Towards Autonomous Robotic Systems : 18th annual conference, pp. 516–525. Springer, Guildford (2017)Google Scholar
  34. 34.
    Rosenbaum, D.A., Cohen, R.G., Meulenbroek, R.G.J., Vaughan, J.: Plans for grasping objects. In: Motor Control and Learning, pp. 9–25. Kluwer Academic Publishers, Boston. (2006)
  35. 35.
    Sa, I., Ge, Z., Dayoub, F., Upcroft, B., Perez, T., McCool, C.: Deepfruits: a fruit detection system using deep neural networks. Sensors 16(8), 1222 (2016)CrossRefGoogle Scholar
  36. 36.
    Sakia, R.M.: The Box-Cox transformation technique: a review. Journal of the Royal Statistical Society. Series D (The Statistician) 41(2), 169–178 (1992). Google Scholar
  37. 37.
    Suh, H.K., Hofstee, J.W., van Henten, E.J.: Improved vegetation segmentation with ground shadow removal using an HDR camera. Precis. Agric. 19(2), 218–237 (2018). CrossRefGoogle Scholar
  38. 38.
    Tadesse, T., Hewett, E.W., Nichols, M.A., Fisher, K.J.: Changes in physicochemical attributes of sweet pepper cv. Domino during fruit growth and development. Sci. Hortic. 93(2), 91–103 (2002). CrossRefGoogle Scholar
  39. 39.
    Tang, J.L., Chen, X.Q., Miao, R.H., Wang, D.: Weed detection using image processing under different illumination for site-specific areas spraying. Comput. Electron. Agric. 122, 103–111 (2016). CrossRefGoogle Scholar
  40. 40.
    Urrea, C., Muṅoz, J.: Path tracking of mobile robot in crops. J. Intell. Robot. Syst. 80(2), 193–205 (2015)CrossRefGoogle Scholar
  41. 41.
    Vitzrabin, E., Edan, Y.: Adaptive thresholding with fusion using a RGBD sensor for red sweet-pepper detection. Biosyst. Eng. 146, 45–56 (2016)CrossRefGoogle Scholar
  42. 42.
    Yoshida, J., Okuyama, S., Suzuki, H.: Fruit Harvesting Apparatus with Television Camera and Monitor. US Patent 4,519,193 (1985)Google Scholar
  43. 43.
    Zemmour, E., Kurtser, P., Edan, Y.: Dynamic thresholding algorithm for robotic apple detection. In: 2017 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), pp. 240–246 (2017)Google Scholar

Copyright information

© The Author(s) 2018

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Department of Computing ScienceUmeå UniversityUmeåSweden
  2. 2.Department of Industrial Engineering and ManagementBen-Gurion University of the NegevBeer-ShevaIsrael

Personalised recommendations