Advertisement

Task-Driven Webpage Saliency

  • Quanlong Zheng
  • Jianbo Jiao
  • Ying CaoEmail author
  • Rynson W. H. Lau
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11218)

Abstract

In this paper, we present an end-to-end learning framework for predicting task-driven visual saliency on webpages. Given a webpage, we propose a convolutional neural network to predict where people look at it under different task conditions. Inspired by the observation that given a specific task, human attention is strongly correlated with certain semantic components on a webpage (e.g., images, buttons and input boxes), our network explicitly disentangles saliency prediction into two independent sub-tasks: task-specific attention shift prediction and task-free saliency prediction. The task-specific branch estimates task-driven attention shift over a webpage from its semantic components, while the task-free branch infers visual saliency induced by visual features of the webpage. The outputs of the two branches are combined to produce the final prediction. Such a task decomposition framework allows us to efficiently learn our model from a small-scale task-driven saliency dataset with sparse labels (captured under a single task condition). Experimental results show that our method outperforms the baselines and prior works, achieving state-of-the-art performance on a newly collected benchmark dataset for task-driven webpage saliency detection.

Keywords

Webpage analysis Saliency detection Task-specific saliency 

Notes

Acknowledgement

We thank the anonymous reviewers for their insightful comments. We also thank NVIDIA for donation of a Titan X Pascal GPU card.

Supplementary material

474202_1_En_18_MOESM1_ESM.pdf (8.3 mb)
Supplementary material 1 (pdf 8518 KB)

References

  1. 1.
    Borji, A., Cheng, M., Jiang, H., Li, J.: Salient object detection: a survey. arXiv:1411.5878 (2014)
  2. 2.
    Borji, A., Itti, L.: State-of-the-art in visual attention modeling. TPAMI (2013)Google Scholar
  3. 3.
    Bylinskii, Z., Judd, T., Oliva, A., Torralba, A., Durand, F.: What do different evaluation metrics tell us about saliency models? arXiv:1604.03605 (2016)
  4. 4.
    Bylinskii, Z., et al.: Learning visual importance for graphic designs and data visualizations. In: UIST (2017)Google Scholar
  5. 5.
    EYEQUANT (2018). http://www.eyequant.com/
  6. 6.
    Guanbin Li, Yuan Xie, L., Yu, Y.: Instance-level salient object segmentation. In: CVPR (2017)Google Scholar
  7. 7.
    He, S., Jiao, J., Zhang, X., Han, G., Lau, R.: Delving into salient object subitizing and detection. In: ICCV (2017)Google Scholar
  8. 8.
    He, S., Lau, R.: Exemplar-driven top-down saliency detection via deep association. In: CVPR (2016)Google Scholar
  9. 9.
    Hou, Q., Cheng, M., Hu, X.W., Borji, A., Tu, Z., Torr, P.: Deeply supervised salient object detection with short connections. In: CVPR (2017)Google Scholar
  10. 10.
    Huang, X., Shen, C., Boix, X., Zhao, Q.: SALICON: reducing the semantic gap in saliency prediction by adapting deep neural networks. In: ICCV (2015)Google Scholar
  11. 11.
    Jiang, M., Huang, S., Duan, J., Zhao, Q.: SALICON: saliency in context. In: CVPR (2015)Google Scholar
  12. 12.
    Judd, T., Durand, F., Torralba, A.: A benchmark of computational models of saliency to predict human fixations. MIT Technical report (2012)Google Scholar
  13. 13.
    Judd, T., Ehinger, K., Durand, F., Torralba, A.: Learning to predict where humans look. In: ICCV, pp. 2106–2113. IEEE (2009)Google Scholar
  14. 14.
    Kruthiventi, S., Gudisa, V., Dholakiya, J., Venkatesh Babu, R.: Saliency unified: a deep architecture for simultaneous eye fixation prediction and salient object segmentation. In: CVPR (2016)Google Scholar
  15. 15.
    Kuen, J., Wang, Z., Wang, G.: Recurrent attentional networks for saliency detection. In: CVPR (2016)Google Scholar
  16. 16.
    Kümmerer, M., Theis, L., Bethge, M.: Deep gaze I: Boosting saliency prediction with feature maps trained on imagenet. In: ICLR (2015)Google Scholar
  17. 17.
    Lee, G., Tai, Y., Kim, J.: Deep saliency with encoded low level distance map and high level features. In: CVPR (2016)Google Scholar
  18. 18.
    Li, G., Yu, Y.: Visual saliency based on multiscale deep features. In: CVPR (2015)Google Scholar
  19. 19.
    Li, G., Yu, Y.: Deep contrast learning for salient object detection. In: CVPR (2016)Google Scholar
  20. 20.
    Liu, N., Han, J.: DHSNet: deep hierarchical saliency network for salient object detection. In: CVPR (2016)Google Scholar
  21. 21.
    Navalpakkam, V., Itti, L.: Modeling the influence of task on attention. Vis. Res. (2005)Google Scholar
  22. 22.
    Liu, N., Han, J.: A deep spatial contextual long-term recurrent convolutional network for saliency detection. IEEE TIP (2018)Google Scholar
  23. 23.
    Pan, J., Sayrol, E., Giro-i Nieto, X., McGuinness, K., O’Connor, N.: Shallow and deep convolutional networks for saliency prediction. In: CVPR (2016)Google Scholar
  24. 24.
    Pang, X., Cao, Y., Lau, R., Chan, A.: Directing user attention via visual flow on web designs. In: SIGGRAPH Asia (2016)Google Scholar
  25. 25.
    Selvaraju, et al.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: ICCV (2017)Google Scholar
  26. 26.
    Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic segmentation. TPAMI (2017)Google Scholar
  27. 27.
    Shen, C., Huang, X., Zhao, Q.: Predicting eye fixations on webpage with an ensemble of early features and high-level representations from deep network. IEEE Trans. Multimed. (2015)Google Scholar
  28. 28.
    Shen, C., Zhao, Q.: Webpage saliency. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 33–46. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10584-0_3CrossRefGoogle Scholar
  29. 29.
    Simonyan, et al.: Deep inside convolutional networks: visualising image classification models and saliency maps. In: ICLR Workshop (2014)Google Scholar
  30. 30.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)
  31. 31.
    Tang, Y., Wu, X.: Saliency detection via combining region-level and pixel-level predictions with CNNs. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 809–825. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46484-8_49CrossRefGoogle Scholar
  32. 32.
    Wang, L., Lu, H., Ruan, X., Yang, M.: Deep networks for saliency detection via local estimation and global search. In: CVPR (2015)Google Scholar
  33. 33.
    Wang, L., Wang, L., Lu, H., Zhang, P., Ruan, X.: Saliency detection with recurrent fully convolutional networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 825–841. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_50CrossRefGoogle Scholar
  34. 34.
    Xiao, H., Feng, J., Wei, Y., Zhang, M., Yan, S.: Deep salient object detection with dense connections and distraction diagnosis. IEEE Trans. Multimed. (2018)Google Scholar
  35. 35.
    Xu, Y., Wu, J., Li, N., Gao, S., Yu, J.: Personalized saliency and its prediction. IJCAI (2017)Google Scholar
  36. 36.
    Yarbus, A.: Eye movements during perception of complex objects. In: Eye Movements and Vision. Springer, Boston (1967).  https://doi.org/10.1007/978-1-4899-5379-7_8CrossRefGoogle Scholar
  37. 37.
    Zhang, J., Lin, Z., Brandt, J., Shen, X., Sclaroff, S.: Top-down neural attention by excitation backprop. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 543–559. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_33CrossRefGoogle Scholar
  38. 38.
    Zhang, P., Wang, D., Lu, H., Wang, H., Yin, B.: Learning uncertain convolutional features for accurate saliency detection. In: ICCV (2017)Google Scholar
  39. 39.
    Zhang, X., Wei, Y., Feng, J., Yang, Y., Huang, T.: Adversarial complementary learning for weakly supervised object localization. In: CVPR (2018)Google Scholar
  40. 40.
    Zhao, R., Ouyang, W., Li, H., Wang, X.: Saliency detection by multi-context deep learning. In: CVPR (2015)Google Scholar
  41. 41.
    Zhou, et al.: Learning deep features for discriminative localization. In: CVPR (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Department of Computer ScienceCity University of Hong KongHong KongHong Kong
  2. 2.University of Illinois at Urbana-ChampaignUrbanaUSA

Personalised recommendations