The Objective Evaluation of Image Object Segmentation Quality

  • Ran Shi
  • King Ngi Ngan
  • Songnan Li
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8192)


In this paper, a novel objective quality metric is proposed for individual object segmentation in images. We analyze four types of segmentation errors, and verify experimentally that besides quantity, area and contour, the distortion of object content is another useful segmentation quality index. Our metric evaluates the similarity between ideal result and segmentation result by measuring these distortions. The metric has been tested on our subjectively-rated image segmentation database and demonstrated a good performance in matching subjective ratings.


Object segmentation Objective metric Distortions 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Powers, D.: Evaluation: From Precision, Recall and F-Factor to ROC, Informedne, Markedness & Correlation. Journal of Machine Learning Technologies 2(1), 37–63 (2011)MathSciNetGoogle Scholar
  2. 2.
    Ge, F., Wang, S., Liu, T.: New benchmark for image segmentation evaluation. Journal of Electronic Imaging 16(3), 33011 (2007)CrossRefGoogle Scholar
  3. 3.
    Villegas, P., Marichal, X.: Perceptually-weighted evaluation criteria for segmentation masks in video sequences. IEEE Trans. Image Process 13(8), 1092–1103 (2004)CrossRefGoogle Scholar
  4. 4.
    Erdem, C., Sankur, B.: Performance evaluation metrics for objectbased video segmentation. In: Proc. X Eur. Signal Process Conf., Tampere, Finland, vol. 2, pp. 917–920 (2000)Google Scholar
  5. 5.
    Strasters, K., Gebrands, J.: Three-dimensional image segmentation using a split, merge and group approach. Pattern Recognit. Lett. 12(5), 307–325 (1991)CrossRefGoogle Scholar
  6. 6.
    McGuinness, K., O’Connor, N.: A comparative evaluation of interactive segmentation algorithms. Pattern Recognition 43(2), 434–444 (2010)CrossRefzbMATHGoogle Scholar
  7. 7.
    Gelasca, E.D.: Full-reference objective quality metrics for video watermarking, video segmentation and 3D model watermarking. In: Ph.D. dissertation, EPFL, Lausanne, Switzerland (2005)Google Scholar
  8. 8.
    Correia, P., Pereira, F.: Objective evaluation of video segmentation quality. IEEE Trans. Image Process 12(2), 186–200 (2003)CrossRefGoogle Scholar
  9. 9.
    Zhang, Y.J.: A Survey on Evaluation Methods for Image Segmentation. Pattern Recognition 29(8), 1335–1346 (1996)CrossRefGoogle Scholar
  10. 10.
    Li, S., Mak, L.C.-M., Ngan, K.N.: Visual Quality Evaluation for Images and Videos. In: Lin, W., Tao, D., Kacprzyk, J., Li, Z., Izquierdo, E., Wang, H. (eds.) Multimedia Analysis, Processing and Communications. SCI, vol. 346, pp. 497–544. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  11. 11.
    Achanta, R., Hemami, S., Estrada, F., Susstrunk, S., Frequency-tuned, M.S.: salient region detection. In: Proc. IEEE CVPR, Miami, USA, pp. 1597–1604 (2009)Google Scholar
  12. 12.
    Rahtu, E., Kannala, J., Salo, M., Heikkilä, J.: Segmenting salient objects from images and videos. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 366–379. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  13. 13.
    Internal Telecommunication Union Radio communication Sector, C.: ITU-R Recommendation BT.500-13, Methodology for the Subjective Assessment of the Quality of Television Pictures (2012)Google Scholar
  14. 14.
    Video Quality Expert Group (VQEG) S.: Final Report From the Video Quality Experts Group on the Validation of Objective Models of Video Quality Assessment I (2010)Google Scholar
  15. 15.
    Zadeh, L.: Fuzzy sets and systems. Information and Control 8(3), 338–353 (1965)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Li, S., Zhang, F., Ma, L., Ngan, K.N.: Image quality assessment by separately evaluating detail losses and additive impairments. IEEE Trans. Multimedia 13(5), 935–949 (2011)CrossRefGoogle Scholar
  17. 17.
    Huynh-Thu, Q., Garcia, N.N., Speranza, F., Corriveau, P., Raake, A.: Study of rating scales for subjective quality assessment of high-definition video. IEEE Trans. Broadcasting 57(1), 1–14 (2011)CrossRefGoogle Scholar
  18. 18.
    Gelasca, E.D., Karaman, M., Ebrahimi, T., Sikora, T.: A Framework for Evaluating Video Object Segmentation Algorithms. In: Proc. IEEE CVPR Workshop, New York, USA, pp. 198–198 (2006)Google Scholar
  19. 19.
    Internal Telecommunication Union Telecommunication Standardization Sector, C.: ITU-T Recommendation P.910, Subjective video quality assessment methods for multimedia applications (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Ran Shi
    • 1
  • King Ngi Ngan
    • 1
  • Songnan Li
    • 1
  1. 1.Department of Electronic EngineeringThe Chinese University of Hong KongHong Kong

Personalised recommendations