3D Plant Modeling: Localization, Mapping and Segmentation for Plant Phenotyping Using a Single Hand-held Camera

  • Thiago Teixeira SantosEmail author
  • Luciano Vieira Koenigkan
  • Jayme Garcia Arnal Barbedo
  • Gustavo Costa Rodrigues
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8928)


Functional-structural modeling and high-throughput phenomics demand tools for 3D measurements of plants. In this work, structure from motion is employed to estimate the position of a hand-held camera, moving around plants, and to recover a sparse 3D point cloud sampling the plants’ surfaces. Multiple-view stereo is employed to extend the sparse model to a dense 3D point cloud. The model is automatically segmented by spectral clustering, properly separating the plant’s leaves whose surfaces are estimated by fitting trimmed B-splines to their 3D points. These models are accurate snapshots for the aerial part of the plants at the image acquisition moment and allow the measurement of different features of the specimen phenotype. Such state-of-the-art computer vision techniques are able to produce accurate 3D models for plants using data from a single free moving camera, properly handling occlusions and diversity in size and structure for specimens presenting sparse canopies. A data set formed by the input images and the resulting camera poses and 3D points clouds is available, including data for sunflower and soybean specimens.


Plant phenotyping SLAM Structure from motion Segmentation 

Supplementary material (8.2 mb)
Supplementary material (ZIP 8,410 KB)


  1. 1.
    Alenya, G., Dellen, B., Torras, C.: 3D modelling of leaves from color and ToF data for robotized plant measuring. In: 2011 IEEE International Conference on Robotics and Automation, pp. 3408–3414, May 2011Google Scholar
  2. 2.
    Alenya, G., Dellen, B., Foix, S., Torras, C.: Robotized Plant Probing: Leaf Segmentation Utilizing Time-of-Flight Data. IEEE Robotics & Automation Magazine 20(3), 50–59 (2013)CrossRefGoogle Scholar
  3. 3.
    Baumberg, A., Lyons, A., Taylor, R.: 3D S.O.M.A commercial software solution to 3D scanning. Graphical Models 67(6), 476–495 (2005)CrossRefGoogle Scholar
  4. 4.
    Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-Up Robust Features (SURF). Computer Vision and Image Understanding 110(3), 346–359 (2008)CrossRefGoogle Scholar
  5. 5.
    Bellasio, C., Olejníčková, J., Tesa, R., Sebela, D., Nedbal, L.: Computer reconstruction of plant growth and chlorophyll fluorescence emission in three spatial dimensions. Sensors (Basel, Switzerland) 12(1), 1052–1071 (2012)CrossRefGoogle Scholar
  6. 6.
    Biskup, B., Scharr, H., Schurr, U., Rascher, U.: A stereo imaging system for measuring structural parameters of plant canopies. Plant, Cell & Environment 30(10), 1299–1308 (2007)CrossRefGoogle Scholar
  7. 7.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(11), 1222–1239 (2001)CrossRefGoogle Scholar
  8. 8.
    Chéné, Y., Rousseau, D., Lucidarme, P., Bertheloot, J., Caffier, V., Morel, P., Belin, E., Chapeau-Blondeau, F.: On the use of depth camera for 3D phenotyping of entire plants. Computers and Electronics in Agriculture 82, 122–127 (2012)CrossRefGoogle Scholar
  9. 9.
    Dassot, M., Colin, A., Santenoise, P., Fournier, M., Constant, T.: Terrestrial laser scanning for measuring the solid wood volume, including branches, of adult standing trees in the forest environment. Computers and Electronics in Agriculture 89, 86–93 (2012)CrossRefGoogle Scholar
  10. 10.
    Delagrange, S., Rochon, P.: Reconstruction and analysis of a deciduous sapling using digital photographs or terrestrial-LiDAR technology. Annals of Botany 108(6), 991–1000 (2011)CrossRefGoogle Scholar
  11. 11.
    Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: KDD, vol. 96, pp. 226–231 (1996)Google Scholar
  12. 12.
    Fiorani, F., Schurr, U.: Future scenarios for plant phenotyping. Annual Review of Plant Biology 64, 267–291 (2013)CrossRefGoogle Scholar
  13. 13.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24(6), 381–395 (1981)CrossRefMathSciNetGoogle Scholar
  14. 14.
    Flöry, S., Hofer, M.: Constrained curve fitting on manifolds. Computer-Aided Design 40(1), 25–34 (2008)CrossRefGoogle Scholar
  15. 15.
    Foix, S., Alenya, G., Torras, C.: Lock-in Time-of-Flight (ToF) Cameras: A Survey. IEEE Sensors Journal 11(9), 1917–1926 (2011)CrossRefGoogle Scholar
  16. 16.
    Freedman, B., Shpunt, A., Machline, M., Arieli, Y.: Depth mapping using projected patterns. US Patent 8,493,496, July 23, 2013Google Scholar
  17. 17.
    Furbank, R.T., Tester, M.: Phenomics-technologies to relieve the phenotyping bottleneck. Trends in Plant Science 16(12), 635–644 (2011)CrossRefGoogle Scholar
  18. 18.
    Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(8), 1362–1376 (2010)CrossRefGoogle Scholar
  19. 19.
    Godin, C., Costes, E., Sinoquet, H.: A Method for Describing Plant Architecture which Integrates Topology and Geometry. Annals of Botany 84(3), 343–357 (1999)CrossRefGoogle Scholar
  20. 20.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, April 2004Google Scholar
  21. 21.
    van der Heijden, G., Song, Y., Horgan, G., Polder, G., Dieleman, A., Bink, M., Palloix, A., van Eeuwijk, F., Glasbey, C.: SPICY: towards automated phenotyping of large pepper plants in the greenhouse. Functional Plant Biology 39(11), 870 (2012)CrossRefGoogle Scholar
  22. 22.
    Henry, P., Krainin, M., Herbst, E., Ren, X., Fox, D.: RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments. The International Journal of Robotics Research 31(5), 647–663 (2012)CrossRefGoogle Scholar
  23. 23.
    Ivanov, N., Boissard, P., Chapron, M., Andrieu, B.: Computer stereo plotting for 3-D reconstruction of a maize canopy. Agricultural and Forest Meteorology 75(1–3), 85–102 (1995)CrossRefGoogle Scholar
  24. 24.
    Kaminuma, E., Heida, N., Tsumoto, Y., Yamamoto, N., Goto, N., Okamoto, N., Konagaya, A., Matsui, M., Toyoda, T.: Automatic quantification of morphological traits via three-dimensional measurement of Arabidopsis. The Plant Journal 38(2), 358–365 (2004)CrossRefGoogle Scholar
  25. 25.
    Lang, A.: Leaf orientation of a cotton plant. Agricultural Meteorology 11(c), 37–51 (1973)CrossRefGoogle Scholar
  26. 26.
    Leung, C., Appleton, B., Buckley, M., Sun, C.: Embedded voxel colouring with adaptive threshold selection using globally minimal surfaces. International Journal of Computer Vision 99(2), 215–231 (2012)CrossRefMathSciNetGoogle Scholar
  27. 27.
    Lourakis, M.I.A., Argyros, A.A.: SBA: A software package for generic sparse bundle adjustment. ACM Transactions on Mathematical Software 36(1), 1–30 (2009)CrossRefMathSciNetGoogle Scholar
  28. 28.
    Lowe, D.G.: Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  29. 29.
    Luxburg, U.: A tutorial on spectral clustering. Statistics and Computing 17(4), 395–416 (2007)CrossRefMathSciNetGoogle Scholar
  30. 30.
    Mörwald, T.: Fitting trimmed B-splines to unordered point clouds.
  31. 31.
    Newcombe, R.a., Davison, A.J., Izadi, S., Kohli, P., Hilliges, O., Shotton, J., Molyneaux, D., Hodges, S., Kim, D., Fitzgibbon, A.: KinectFusion: Real-time dense surface mapping and tracking. In: 2011 10th IEEE International Symposium on Mixed and Augmented Reality, pp. 127–136. IEEE, October 2011Google Scholar
  32. 32.
    Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: DTAM: dense tracking and mapping in real-time. In: 2011 International Conference on Computer Vision, pp. 2320–2327, November 2011Google Scholar
  33. 33.
    Ng, A., Jordan, M., Weiss, Y.: On spectral clustering: Analysis and an algorithm. Advances in neural information ... (2002)Google Scholar
  34. 34.
    Paproki, A., Fripp, J., Salvado, O., Sirault, X., Berry, S., Furbank, R.: Automated 3D segmentation and analysis of cotton plants. In: 2011 International Conference on Digital Image Computing: Techniques and Applications, pp. 555–560, December 2011Google Scholar
  35. 35.
    Paproki, A., Sirault, X., Berry, S., Furbank, R., Fripp, J.: A novel mesh processing based technique for 3D plant analysis. BMC Plant Biology 12, 63 (2012)CrossRefGoogle Scholar
  36. 36.
    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011)zbMATHGoogle Scholar
  37. 37.
    Piegl, L., Tiller, W.: The NURBS Book. Monographs in Visual Communication, U.S. Government Printing Office (1997)Google Scholar
  38. 38.
    Rakocevic, M., Sinoquet, H., Christophe, A., Varlet-Grancher, C.: Assessing the Geometric Structure of a White Clover (Trifolium repens L.) Canopy using3-D Digitising. Annals of Botany 86(3), 519–526 (2000)CrossRefGoogle Scholar
  39. 39.
    Rascher, U., Blossfeld, S., Fiorani, F., Jahnke, S., Jansen, M., Kuhn, A.J., Matsubara, S., Märtin, L.L.A., Merchant, A., Metzner, R., Müller-Linow, M., Nagel, K.A., Pieruschka, R., Pinto, F., Schreiber, C.M., Temperton, V.M., Thorpe, M.R., Dusschoten, D.V., Van Volkenburgh, E., Windt, C.W., Schurr, U.: Non-invasive approaches for phenotyping of enhanced performance traits in bean. Functional Plant Biology 38(12), 968 (2011)CrossRefGoogle Scholar
  40. 40.
    Reuzeau, C., Frankard, V., Hatzfeld, Y., Sanz, A., Camp, W.V., Lejeune, P., Wilde, C.D., Lievens, K., de Wolf, J., Vranken, E., Peerbolte, R., Broekaert, W.: Traitmill™: a functional genomics platform for the phenotypic analysis of cereals. Plant Genetic Resources 4(01), 20–24 (2006)CrossRefGoogle Scholar
  41. 41.
    Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proceedings Third International Conference on 3-D Digital Imaging and Modeling, pp. 145–152 (2001)Google Scholar
  42. 42.
    Rusu, R.B., Cousins, S.: 3D is here: point cloud library (PCL). In: IEEE International Conference on Robotics and Automation (ICRA), Shanghai, China, May 9–13, 2011Google Scholar
  43. 43.
    Santos, T., Ueda, J.: Automatic 3D plant reconstruction from photographies, segmentation and classification of leaves and internodes using clustering 1. In: Sievänen, R., Nikinmaa, E., Godin, C., Anna Lintunen, P.N. (eds.) Proceedings of the 7th International Conference on Functional-Structural Plant Models, Saariselkä, Finland, pp. 95–97 (2013)Google Scholar
  44. 44.
    Santos, T.T., de Oliveira, A.A.: Image-based 3D digitizing for plant architecture analysis and phenotyping. In: Saúde, A.V., Guimarães, S.J.F. (eds.) Workshop on Industry Applications (WGARI) in SIBGRAPI 2012 (XXV Conference on Graphics, Patterns and Images). Ouro Preto (2012)Google Scholar
  45. 45.
    Scaramuzza, D., Fraundorfer, F.: Visual Odometry [Tutorial]. IEEE Robotics & Automation Magazine 18(4), 80–92 (2011)CrossRefGoogle Scholar
  46. 46.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8), 888–905 (2000)CrossRefGoogle Scholar
  47. 47.
    Sinoquet, H., Moulia, B., Bonhomme, R.: Estimating the three-dimensional geometry of a maize crop as an input of radiation models: comparison between three-dimensional digitizing and plant profiles. Agricultural and Forest Meteorology 55(3–4), 233–249 (1991)CrossRefGoogle Scholar
  48. 48.
    Sirault, X., Fripp, J., Paproki, A., Kuffner, P., Nguyen, C., Li, R., Daily, H., Guo, J., Furbank, R.: PlantScan: a three-dimensional phenotyping platform for capturing the structural dynamic of plant development and growth. In: Proceedings of the 7th International Conference on FunctionalStructural Plant Models, Saariselkä, Finland, pp. 45–48 (2013)Google Scholar
  49. 49.
    Snavely, N., Seitz, S., Szeliski, R.: Modeling the World from Internet Photo Collections. International Journal of Computer Vision 80(2), 189–210 (2008)CrossRefGoogle Scholar
  50. 50.
    Song, Y., Glasbey, C.A., van der Heijden, G.W.A.M., Polder, G., Dieleman, J.A.: Combining stereo and time-of-flight images with application to automatic plant phenotyping. In: Heyden, A., Kahl, F. (eds.) SCIA 2011. LNCS, vol. 6688, pp. 467–478. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  51. 51.
    Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment – a modern synthesis. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) ICCV-WS 1999. LNCS, vol. 1883, p. 298. Springer, Heidelberg (2000) CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Thiago Teixeira Santos
    • 1
    Email author
  • Luciano Vieira Koenigkan
    • 1
  • Jayme Garcia Arnal Barbedo
    • 1
  • Gustavo Costa Rodrigues
    • 1
  1. 1.Embrapa Agricultural Informatics CampinasCampinasBrazil

Personalised recommendations