Ranking Domain-Specific Highlights by Analyzing Edited Videos

  • Min Sun
  • Ali Farhadi
  • Steve Seitz
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8689)


We present a fully automatic system for ranking domain-specific highlights in unconstrained personal videos by analyzing online edited videos. A novel latent linear ranking model is proposed to handle noisy training data harvested online. Specifically, given a search query (domain) such as “surfing”, our system mines the Youtube database to find pairs of raw and corresponding edited videos. Leveraging the assumption that edited video is more likely to contain highlights than the trimmed parts of the raw video, we obtain pair-wise ranking constraints to train our model. The learning task is challenging due to the amount of noise and variation in the mined data. Hence, a latent loss function is incorporated to robustly deal with the noise. We efficiently learn the latent model on a large number of videos (about 700 minutes in all) using a novel EM-like self-paced model selection procedure. Our latent ranking model outperforms its classification counterpart, a motion analysis baseline [15], and a fully-supervised ranking system that requires labels from Amazon Mechanical Turk. Finally, we show that impressive highlights can be retrieved without additional human supervision for domains like skating, surfing, skiing, gymnastics, parkour, and dog activity in unconstrained personal videos.


Video highlight detection latent ranking 


  1. 1.
    Borgo, R., Chen, M., Daubney, B., Grundy, E., Heidemann, G., Hoferlin, B., Hoferlin, M., Janicke, H., Weiskopf, D., Xie, X.: A survey on video-based graphics and video visualization. In: EUROGRAPHICS (2011)Google Scholar
  2. 2.
    Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: LIBLINEAR: A library for large linear classification. Journal of Machine Learning Research 9, 1871–1874 (2008)zbMATHGoogle Scholar
  3. 3.
    Gong, Y., Liu, X.: Video summarization using singular value decomposition. In: CVPR (2000)Google Scholar
  4. 4.
    Hanjalic, A.: Adaptive extraction of highlights from a sport video based on excitement modeling. IEEE Transactions on Multimedia (2005)Google Scholar
  5. 5.
    Hannon, J., McCarthy, K., Lynch, J., Smyth, B.: Personalized and automatic social summarization of events in video. In: IUI (2011)Google Scholar
  6. 6.
    Hu, Y., Li, M., Yu, N.: Multiple-instance ranking: Learning to rank images for image retrieval. In: CVPR (2008)Google Scholar
  7. 7.
    Jacobs, C.E., Finkelstein, A., Salesin, D.H.: Fast multiresolution image querying. In: SIGGRAPH (1995)Google Scholar
  8. 8.
    Khosla, A., Hamid, R., Lin, C.J., Sundaresan, N.: Large-scale video summarization using web-image priors. In: CVPR (2013)Google Scholar
  9. 9.
    Khurram Soomro, A.R.Z., Shah, M.: Ucf101: A dataset of 101 human action classes from videos in the wild. CRCV-TR (2013)Google Scholar
  10. 10.
    Kolekar, M., Sengupta, S.: Event-importance based customized and automatic cricket highlight generation. In: ICME (2006)Google Scholar
  11. 11.
    Lee, C.P., Lin, C.J.: Large-scale linear ranksvm. Neural Computation (2013)Google Scholar
  12. 12.
    Lee, Y.J., Ghosh, J., Grauman, K.: Discovering important people and objects for egocentric video summarization. In: CVPR (2012)Google Scholar
  13. 13.
    Liu, D., Hua, G., Chen, T.: A hierarchical visual model for video object summarization. TPAMI (2010)Google Scholar
  14. 14.
    Lu, Z., Grauman, K.: Story-driven summarization for egocentric video. In: CVPR (2013)Google Scholar
  15. 15.
    Mendi, E., Clemente, H.B., Bayrak, C.: Sports video summarization based on motion analysis. Computers and Electrical Engineering 39(3), 790–796 (2013)CrossRefGoogle Scholar
  16. 16.
    Nepal, S., Srinivasan, U., Reynolds, G.: Automatic detection of goal segments in basketball videos. ACM Multimedia (2001)Google Scholar
  17. 17.
    Ngo, C., Ma, Y., Zhan, H.: Video summarization and scene detection by graph modeling. In: CSVT (2005)Google Scholar
  18. 18.
    Olsen, D.R., Moon, B.: Video summarization based on user interaction. In: EuroITV (2011)Google Scholar
  19. 19.
    Parikh, D., Grauman, K.: Relative attributes. In: ICCV (2011)Google Scholar
  20. 20.
    Rui, Y., Gupta, A., Acero, A.: Automatically extracting highlights for tv baseball programs. ACM Multimedia (2000)Google Scholar
  21. 21.
    Siddiquie, B., Feris, R., Davis, L.: Image ranking and retrieval based on multi-attribute queries. In: CVPR (2011)Google Scholar
  22. 22.
    Sun, M., Farhadi, A., Seitz, S.: Technical report of ranking domain-specific highlights,
  23. 23.
    Tang, H., Kwatra, V., Sargin, M., Gargi, U.: Detecting highlights in sports videos: Cricket as a test case. ICME (2011)Google Scholar
  24. 24.
    Wang, H., Kläser, A., Schmid, C., Liu, C.L.: Action Recognition by Dense Trajectories. In: CVPR (2011)Google Scholar
  25. 25.
    Xiong, Z., Radhakrishnan, R., Divakaran, A., Huang, T.: Highlights extraction from sports video based on an audio-visual marker detection framework. ICME (2005)Google Scholar
  26. 26.
    Wang, J., Xu, C., Chang, E., Tian, Q.: Sports highlight detection from keyword sequences using hmm. In: ICME (2004)Google Scholar
  27. 27.
    Yow, D., Yeo, B., Yeung, M., Liu, B.: Analysis and presentation of soccer highlights from digital video. In: ACCV (1995)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Min Sun
    • 1
  • Ali Farhadi
    • 1
  • Steve Seitz
    • 1
  1. 1.University of WashingtonSeattleUSA

Personalised recommendations