A shadow detection and removal method for fruit recognition in natural environments

Abstract

Effective shadow detection and shadow removal can improve the performance of fruit recognition in natural environments and provide technical support for agricultural intelligence. In this study, a superpixel segmentation method was used to divide an image into multiple small regions. Based on the superpixel segmentation results, the shadow regions and the shadowless regions of the orchard images under natural light were compared and studied. Seven shadow saliency features (SSF) were explored and analyzed for shadow detection. The SSF were used to enhance the shadow characteristics. Then, the genetic algorithm (GA) was used to optimize the parameters, and support vector machine recursive feature elimination (SVM-RFE) was used to determine the best feature combination for shadow detection. According to the best feature combination, the support vector machine (SVM) algorithm was used to determine whether each segment of the superpixel segmentation results belonged to the shadow region. Shadow removal was carried out on each detected shadow region, and a natural light image after shadow removal was obtained. Finally, the accuracy of shadow detection was tested. The experimental results showed that the average accuracy of the shadow detection algorithm in this study was 91.91%. As a result, the precision and recall for fruits recognition after shadow removal generally improved.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

References

  1. Adankon, M. M., & Cheriet, M. (2009). Support vector machine. In S. Z. Li & A. Jain (Eds.), Encyclopedia of biometrics (pp. 1303–1308). Boston, MA, USA: Springer.

    Chapter  Google Scholar 

  2. Baba, M., Mukunoki, M., & Asada, N. (2004). Shadow removal from a real image based on shadow density. In Proceedings of the ACM SIGGRAPH (pp. 8–12). New York, USA: ACM.

  3. Chang, C.-C., & Lin, C.-J. (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology,2(3), 1–27. https://doi.org/10.1145/1961189.1961199.

    Article  Google Scholar 

  4. Comaniciu, D., & Meer, P. (2002). Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence,24(5), 603–619. https://doi.org/10.1109/34.1000236.

    Article  Google Scholar 

  5. Finlayson, G. D., Drew, M. S., & Lu, C. (2004). Intrinsic images by entropy minimization. In T. Pajdla & J. Matas (Eds.), Computer vision (ECCV) (pp. 582–595). Berlin, Heidelberg, Germany: Springer.

    Google Scholar 

  6. Finlayson, G. D., Drew, M. S., & Lu, C. (2009). Entropy minimization for shadow removal. International Journal of Computer Vision,85(1), 35–57. https://doi.org/10.1007/s11263-009-0243-z.

    Article  Google Scholar 

  7. Finlayson, G., Fredembach, C., & Drew, M. S. (2007). Detecting illumination in images. In 2007 IEEE 11th international conference on computer vision (ICCV) (pp. 1–8). Los Alamitos, CA, USA: IEEE.

  8. Finlayson, G. D., Hordley, S. D., & Drew, M. S. (2002). Removing shadows from images. In A. Heyden, G. Sparr, M. Nielsen, & P. Johansen (Eds.), Computer vision (ECCV) (pp. 823–836). Berlin, Heidelberg, Germany: Springer.

    Google Scholar 

  9. Finlayson, G. D., Hordley, S. D., Lu, C., & Drew, M. S. (2006). On the removal of shadows from images. IEEE Transactions on Pattern Analysis and Machine Intelligence,28(1), 59–68. https://doi.org/10.1109/tpami.2006.18.

    Article  PubMed  Google Scholar 

  10. Freeman, W. T., Pasztor, E. C., & Carmichael, O. T. (2000). Learning low-level vision. International Journal of Computer Vision,40(1), 25–47. https://doi.org/10.1023/A:1026501619075.

    Article  Google Scholar 

  11. Goldberg, D. E. (1989). Genetic algorithms in search, optimization and machine learning. Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc.

    Google Scholar 

  12. Gong, H., & Cosker, D. (2017). User-assisted image shadow removal. Image and Vision Computing,62, 19–27. https://doi.org/10.1016/j.imavis.2017.04.001.

    Article  Google Scholar 

  13. Gonzalez, R. C., & Woods, R. E. (2007). Digital image processing (3rd ed.). Upper Saddle River, NJ, USA: Prentice-Hall Inc.

    Google Scholar 

  14. Gonzalez, R. C., Woods, R. E., & Eddins, S. L. (2003). Digital image processing using MATLAB. Upper Saddle River, NJ, USA: Prentice-Hall Inc.

    Google Scholar 

  15. Guo, R. Q., Dai, Q. Y., & Hoiem, D. (2013). Paired regions for shadow detection and removal. IEEE Transactions on Pattern Analysis and Machine Intelligence,35(12), 2956–2967. https://doi.org/10.1109/tpami.2012.214.

    Article  PubMed  Google Scholar 

  16. Guyon, I., Weston, J., Barnhill, S., & Vapnik, V. (2002). Gene selection for cancer classification using support vector machines. Machine Learning,46(1–3), 389–422. https://doi.org/10.1023/a:1012487302797.

    Article  Google Scholar 

  17. Hertog, W., Llenas, A., & Carreras, J. (2015). Optimizing indoor illumination quality and energy efficiency using a spectrally tunable lighting system to augment natural daylight. Optics Express,23(24), 1564–1574. https://doi.org/10.1364/oe.23.01564.

    CAS  Article  Google Scholar 

  18. Jobson, D. J., Rahman, Z., & Woodell, G. A. (1997). A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Transactions on Image Processing,6(7), 965–976. https://doi.org/10.1109/83.597272.

    CAS  Article  PubMed  Google Scholar 

  19. Khan, S. H., Bennamoun, M., Sohel, F., & Togneri, R. (2016). Automatic shadow detection and removal from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence,38(3), 431–446. https://doi.org/10.1109/tpami.2015.2462355.

    Article  PubMed  Google Scholar 

  20. Lalonde, J. F., Efros, A. A., & Narasimhan, S. G. (2010). Detecting ground shadows in outdoor consumer photographs. In K. Daniilidis, P. Maragos, & N. Paragios (Eds.), Computer vision (ECCV) (Vol. 6312, Lecture Notes in Computer Science, pp. 322–335). Berlin, Heidelberg, Germany: Springer.

  21. Lei, T., Jia, X. H., Zhang, Y. N., He, L. F., Meng, H. Y., & Nandi, A. K. (2018). Significantly fast and robust fuzzy c-means clustering algorithm based on morphological reconstruction and membership filtering. IEEE Transactions on Fuzzy Systems,26(5), 3027–3041. https://doi.org/10.1109/tfuzz.2018.2796074.

    Article  Google Scholar 

  22. Levine, M. D., & Bhattacharyya, J. (2005). Removing shadows. Pattern Recognition Letters,26(3), 251–265. https://doi.org/10.1016/j.patrec.2004.10.021.

    Article  Google Scholar 

  23. MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth berkeley symposium on mathematical statistics and probability (pp. 281–297). Berkeley, CA, USA: University of California Press.

  24. Matsushita, Y., Nishino, K., Ikeuchi, K., & Sakauchi, M. (2004). Illumination normalization with time-dependent intrinsic images for video surveillance. IEEE Transactions on Pattern Analysis and Machine Intelligence,26(10), 1336–1347. https://doi.org/10.1109/tpami.2004.86.

    Article  PubMed  Google Scholar 

  25. Mo, N., Zhu, R. X., Yan, L., & Zhao, Z. (2018). Deshadowing of urban airborne imagery based on object-oriented automatic shadow detection and regional matching compensation. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing,11(2), 585–605. https://doi.org/10.1109/jstars.2017.2787116.

    Article  Google Scholar 

  26. Otsu, N. (1979). Threshold selection method from gray-level histograms. IEEE Transactions on Systems Man and Cybernetics,9(1), 62–66. https://doi.org/10.1109/tsmc.1979.4310076.

    Article  Google Scholar 

  27. Phong, B. T. (1975). Illumination for computer generated pictures. Communications of the ACM,18(6), 311–317. https://doi.org/10.1145/360825.360839.

    Article  Google Scholar 

  28. Shen, L., & Yeo, C. (2011). Intrinsic images decomposition using a local and global sparse representation of reflectance. In 2011 IEEE conference on computer vision and pattern recognition (CVPR) (pp. 697–704). Los Alamitos, CA, USA: IEEE.

  29. Shen, L., Yeo, C., & Hua, B.-S. (2013). Intrinsic image decomposition using a sparse representation of reflectance. IEEE Transactions on Pattern Analysis and Machine Intelligence,35(12), 2904–2915. https://doi.org/10.1109/tpami.2013.136.

    Article  PubMed  Google Scholar 

  30. Suh, H. K., Hofstee, J. W., & van Henten, E. J. (2018). Improved vegetation segmentation with ground shadow removal using an HDR camera. Precision Agriculture,19(2), 218–237. https://doi.org/10.1007/s11119-017-9511-z.

    Article  Google Scholar 

  31. Sun, J., Tian, J. D., Du, Y. K., & Tang, Y. D. (2009). Retinex theory-based shadow detection and removal in single outdoor image. Industrial Robot: An International Journal,36(3), 263–269. https://doi.org/10.1108/01439910910950531.

    Article  Google Scholar 

  32. Wang, G., Wei, Y., & Qiao, S. (2018). Equation solving generalized inverses. In G. Wang, Y. Wei, & S. Qiao (Eds.), Generalized inverses: Theory and computations (pp. 1–64). Singapore: Springer.

    Chapter  Google Scholar 

  33. Wang, S.-T., Yuan, Y.-Y., Zhu, C.-Y., Kong, D.-M., & Wang, Y.-T. (2019). Discrimination of polycyclic aromatic hydrocarbons based on fluorescence spectrometry coupled with CS-SVM. Measurement,139, 475–481. https://doi.org/10.1016/j.measurement.2019.01.087.

    Article  Google Scholar 

  34. Weiss, Y. (2001). Deriving intrinsic images from image sequences. In Proceedings eighth IEEE international conference on computer vision (ICCV 2001) (pp. 68–75). Los Alamitos, CA, USA: IEEE. https://doi.org/10.1109/iccv.2001.937606.

  35. Wu, T. P., & Tang, C. K. (2005). A Bayesian approach for shadow extraction from a single image. In Tenth IEEE international conference on computer vision (ICCV) (pp. 480–487). Los Alamitos, CA, USA: IEEE.

  36. Zhu, J. J., Samuel, K. G. G., Masood, S. Z., & Tappen, M. F. (2010). Learning to recognize shadows in monochromatic natural images. In 2010 IEEE conference on computer vision and pattern recognition (pp. 223–230). Los Alamitos, CA, USA: IEEE.

Download references

Acknowledgements

This work is supported by the Natural Science Foundation of Guangdong (No. 2018A030313330), the Special Funds for the Cultivation of Guangdong College Students’ Scientific and Technological Innovation (“Climbing Program” Special Funds.) (pdjh2019a0073), the National Natural Science Foundation of China (Nos. 31201135, 31571568) and the Science and Technology Plan Project of Guangzhou (201802020032). The authors wish to thank the useful comments of the anonymous reviewers to this paper.

Author information

Affiliations

Authors

Contributions

BR and XJ conceived and designed the experiments; BR designed the algorithm; BR, CS and ZZ performed the experiments and analyzed the data; BR, XJ, CS, ZZ, GW, YZ and LX wrote the paper.

Corresponding author

Correspondence to Juntao Xiong.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Bu, R., Xiong, J., Chen, S. et al. A shadow detection and removal method for fruit recognition in natural environments. Precision Agric 21, 782–801 (2020). https://doi.org/10.1007/s11119-019-09695-1

Download citation

Keywords

  • Shadow detection
  • Fruit detection
  • Feature extraction
  • Shadow removal
  • Support vector machine