Saliency Detection with Flash and No-flash Image Pairs
Abstract
In this paper, we propose a new saliency detection method using a pair of flash and no-flash images. Our approach is inspired by two observations. First, only the foreground objects are significantly brightened by the flash as they are relatively nearer to the camera than the background. Second, the brightness variations introduced by the flash provide hints to surface orientation changes. Accordingly, the first observation is explored to form the background prior to eliminate background distraction. The second observation provides a new orientation cue to compute surface orientation contrast. These photometric cues from the two observations are independent of visual attributes like color, and they provide new and robust distinctiveness to support salient object detection. The second observation further leads to the introduction of new spatial priors to constrain the regions rendered salient to be compact both in the image plane and in 3D space. We have constructed a new flash/no-flash image dataset. Experiments on this dataset show that the proposed method successfully identifies salient objects from various challenging scenes that the state-of-the-art methods usually fail.
Keywords
Saliency detection Flash photography Background elimination Surface orientationSupplementary material
References
- 1.Achanta, R., Hemami, S., Estrada, F., Susstrunk, S.: Frequency-tuned salient region detection. In: CVPR, pp. 1597–1604 (2009)Google Scholar
- 2.Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE TPAMI, 2274–2282 (2012)Google Scholar
- 3.Agrawal, A., Raskar, R., Nayar, S., Li, Y.: Removing photography artifacts using gradient projection and flash-exposure sampling. ACM TOG 24(3), 828–835 (2005)CrossRefGoogle Scholar
- 4.Borji, A., Sihite, D.N., Itti, L.: Salient object detection: A benchmark. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part II. LNCS, vol. 7573, pp. 414–429. Springer, Heidelberg (2012)CrossRefGoogle Scholar
- 5.Chang, K., Liu, T., Chen, H., Lai, S.: Fusing generic objectness and visual saliency for salient object detection. In: ICCV (2011)Google Scholar
- 6.Chen, T., Cheng, M., Tan, P., Shamir, A., Hu, S.: Sketch2photo: Internet image montage. ACM TOG 28(5), 124:1–124:10 (2009)Google Scholar
- 7.Cheng, M., Zhang, G., Mitra, N., Huang, X., Hu, S.: Global contrast based salient region detection. In: CVPR, pp. 409–416 (2011)Google Scholar
- 8.Eisemann, E., Durand, F.: Flash photography enhancement via intrinsic relighting. ACM TOG 23(3), 673–678 (2004)CrossRefGoogle Scholar
- 9.Enns, J., Rensink, R.: Sensitivity to three-dimensional orientation in visual search. Psychological Science 1, 323–326 (1990)CrossRefGoogle Scholar
- 10.Goferman, S., Zelnik-Manor, L., Tal, A.: Context-aware saliency detection. In: CVPR (2010)Google Scholar
- 11.Harel, J., Koch, C., Perona, P.: Graph-based visual saliency. In: NIPS, pp. 545–552 (2007)Google Scholar
- 12.He, Z., Nakayama, K.: Visual attention to surfaces in 3-d space. Proc. National Academy of Sciences 92, 11155–11159 (1995)CrossRefGoogle Scholar
- 13.Itti, L., Koch, C.: Computational modelling of visual attention. Nature Reviews Neuroscience 2(3), 194–203 (2001)CrossRefGoogle Scholar
- 14.Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE TPAMI 20(11), 1254–1259 (1998)CrossRefGoogle Scholar
- 15.Jiang, H., Wang, J., Yuan, Z., Liu, T., Zheng, N.: Automatic salient object segmentation based on context and shape prior. In: BMVC (2011)Google Scholar
- 16.Koch, C., Ullman, S.: Shifts in Selective Visual Attention: Towards the Underlying Neural Circuitry. Human Neurobiology 4, 219–227 (1985)Google Scholar
- 17.Krishnan, D., Fergus, R.: Dark flash photography. ACM TOG 28(3), 96:1–96:11 (2009)Google Scholar
- 18.Liao, M., Wang, L., Yang, R., Gong, M.: Light fall-off stereo. In: CVPR, pp. 1–8 (2007)Google Scholar
- 19.Liu, T., Yuan, Z., Sun, J., Wang, J., Zheng, N., Tang, X., Shum, H.: Learning to detect a salient object. IEEE TPAMI 33(2), 353–367 (2011)CrossRefGoogle Scholar
- 20.Ma, Y., Zhang, H.: Contrast-based image attention analysis by using fuzzy growing. ACM Multimedia, 374–381 (2003)Google Scholar
- 21.Margolin, R., Tal, A., Zelnik-Manor, L.: What makes a patch distinct? In: CVPR (2013)Google Scholar
- 22.Niu, Y., Geng, Y., Li, X., Liu, F.: Leveraging stereopsis for saliency analysis. In: CVPR (2012)Google Scholar
- 23.Niu, Y., Liu, F., Li, X., Gleicher, M.: Warp propagation for video resizing. In: CVPR (2010)Google Scholar
- 24.Nothdurft, H.: Salience from feature contrast: additivity across dimensions. Vision Research 40(10), 1183–1201 (2000)CrossRefGoogle Scholar
- 25.Parkhurst, D., Law, K., Niebur, E.: Modeling the role of salience in the allocation of overt visual attention. Vision Research 42(1), 107–123 (2002)CrossRefGoogle Scholar
- 26.Perazzi, F., Krähenbühl, P., Pritch, Y., Hornung, A.: Saliency filters: Contrast based filtering for salient region detection. In: CVPR, pp. 733–740 (2012)Google Scholar
- 27.Petschnigg, G., Szeliski, R., Agrawala, M., Cohen, M., Hoppe, H., Toyama, K.: Digital photography with flash and no-flash image pairs. ACM TOG 23(3), 664–672 (2004)CrossRefGoogle Scholar
- 28.Raskar, R., Tan, K., Feris, R., Yu, J., Turk, M.: Non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging. ACM TOG 23(3), 679–688 (2004)CrossRefGoogle Scholar
- 29.Reinagel, P., Zador, A.: Natural scene statistics at the centre of gaze. In: Network: Computation in Neural Systems, pp. 341–350 (1999)Google Scholar
- 30.Reynolds, J., Desimone, R.: Interacting roles of attention and visual salience in V4. Neuron 37(5), 853–863 (2003)CrossRefGoogle Scholar
- 31.Sharma, G., Jurie, F., Schmid, C.: Discriminative spatial saliency for image classification. In: CVPR (2012)Google Scholar
- 32.Shi, K., Wang, K., Lu, J., Lin, L.: PISA: Pixelwise image saliency by aggregating complementary appearance contrast measures with spatial priors. In: CVPR (2013)Google Scholar
- 33.Sun, J., Li, Y., Kang, S., Shum, H.: Flash matting. ACM TOG 25(3), 772–778 (2006)CrossRefGoogle Scholar
- 34.Sun, J., Sun, J., Kang, S., Xu, Z., Tang, X., Shum, H.: Flash cut: Foreground extraction with flash and no-flash image pairs. In: CVPR (2007)Google Scholar
- 35.Toet, A.: Computational versus psychophysical bottom-up image saliency: A comparative evaluation study. IEEE TPAMI 33(11), 2131–2146 (2011)CrossRefGoogle Scholar
- 36.Valenti, R., Sebe, N., Gevers, T.: Image saliency by isocentric curvedness and color. In: ICCV (2009)Google Scholar
- 37.Wei, Y., Wen, F., Zhu, W., Sun, J.: Geodesic saliency using background priors. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part III. LNCS, vol. 7574, pp. 29–42. Springer, Heidelberg (2012)CrossRefGoogle Scholar
- 38.Wexler, M., Ouarti, N.: Depth affects where we look. Current Biology 18(23), 1872–1876 (2008)CrossRefGoogle Scholar
- 39.Wolfe, J., Horowitz, T.: What attributes guide the deployment of visual attention and how do they do it? Nature Reviews Neuroscience 5(6), 495–501 (2004)CrossRefGoogle Scholar
- 40.Yan, Q., Xu, L., Shi, J., Jia, J.: Hierachical saliency detection. In: CVPR (2013)Google Scholar
- 41.Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M.H.: Saliency detection via graph-based manifold ranking. In: CVPR (2013)Google Scholar
- 42.Zhou, C., Troccoli, A., Pulli, K.: Robust stereo with flash and no-flash image pairs. In: CVPR, pp. 342–349 (2012)Google Scholar