Abstract
Videos contain complex spatially-varying motion blur due to the combination of object motion, camera motion, and depth variation with finite shutter speeds. Existing methods to estimate optical flow, deblur the images, and segment the scene fail in such cases. In particular, boundaries between differently moving objects cause problems, because here the blurred images are a combination of the blurred appearances of multiple surfaces. We address this with a novel layered model of scenes in motion. From a motion-blurred video sequence, we jointly estimate the layer segmentation and each layer’s appearance and motion. Since the blur is a function of the layer motion and segmentation, it is completely determined by our generative model. Given a video, we formulate the optimization problem as minimizing the pixel error between the blurred frames and images synthesized from the model, and solve it using gradient descent. We demonstrate our approach on synthetic and real sequences.
Chapter PDF
Similar content being viewed by others
References
Baker, S., Kanade, T.: Super-resolution optical flow. Tech. Rep. CMU-RI-TR-99-36, Carnegie Mellon University, The Robotics Institute (1999)
Bar, L., Berkels, B., Rumpf, M., Sapiro, G.: A variational framework for simultaneous motion estimation and restoration of motion-blurred video. In: IEEE Int. Conf. on Computer Vision (ICCV), pp. 1–8 (2007)
Bascle, B., Blake, A., Zisserman, A.: Motion deblurring and super-resolution from an image sequence. In: Buxton, B.F., Cipolla, R. (eds.) ECCV 1996. LNCS, vol. 1065, pp. 573–582. Springer, Heidelberg (1996), http://dl.acm.org/citation.cfm?id=645310.649034
Chakrabarti, A., Zickler, T., Freeman, W.: Analyzing spatially-varying blur. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2512–2519 (June 2010)
Charbonnier, P., Blanc-Feraud, L., Aubert, G., Barlaud, M.: Two deterministic half-quadratic regularization algorithms for computed imaging. In: IEEE Int. Conf. Image Proc. (ICIP), vol. 2, pp. 168–172 (1994)
Chen, W.G., Nandhakumar, N., Martin, W.: Image motion estimation from motion smear-a new computational model. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI) 18(4), 412–425 (1996)
Cho, S., Lee, S.: Fast motion deblurring. In: ACM Trans. Graphics (TOG) – Proc. SIGGRAPH Asia, pp. 145:1–145:8. ACM, New York (2009), http://doi.acm.org/10.1145/1661412.1618491
Cho, S., Wang, J., Lee, S.: Video deblurring for hand-held cameras using patch-based synthesis. ACM Trans. Graph. 31(4), 64:1–64:9 (2012), http://doi.acm.org/10.1145/2185520.2185560
Chunhe, S., Hai, Z., Wei, J.: Motion deblurring from a single image using multi-layer statistics priors. In: 2011 IEEE International Conference on Consumer Electronics (ICCE), pp. 481–482 (January 2011)
Cremers, D., Soatto, S.: Variational space-time motion segmentation. In: Triggs, B., Zisserman, A. (eds.) Int. Conf. on Computer Vision (ICCV), Nice, vol. 2, pp. 886–892 (October 2003)
Dai, S., Wu, Y.: Removing partial blur in a single image. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2544–2551 (2009)
Dai, S., Wu, Y.: Motion from blur. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (June 2008)
Fergus, R., Singh, B., Hertzmann, A., Roweis, S.T., Freeman, W.T.: Removing camera shake from a single photograph. ACM Trans. Graphics (TOG) – Proc. SIGGRAPH 25(3), 787–794 (2006), http://doi.acm.org/10.1145/1141911.1141956
Fish, D.A., Brinicombe, A.M., Pike, E.R., Walker, J.G.: Blind deconvolution by means of the richardson–lucy algorithm. J. Opt. Soc. Am. A 12(1), 58–65 (1995), http://josaa.osa.org/abstract.cfm?URI=josaa-12-1-58
Frey, B., Jojic, N., Kannan, A.: Learning appearance and transparency manifolds of occluded objects in layers. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 45–52 (2003)
Grundmann, M., Kwatra, V., Castro, D., Essa, I.: Calibration-free rolling shutter removal. In: IEEE International Conference Computational Photography (ICCP), pp. 1–8 (2012)
He, X., Luo, T., Yuk, S.C., Chow, K., Wong, K., Chung, R.H.Y.: Motion estimation method for blurred videos and application of deblurring with spatially varying blur kernels. In: Int. Conf. on Computer Sciences and Convergence Information Technology (ICCIT), pp. 355–359 (2010)
Jackson, J., Yezzi, A., Soatto, S.: Dynamic shape and apperance modeling via moving and deforming layers. International Journal of Computer Vision (IJCV) 79(1), 71–84 (2008)
Jepson, A.D., Fleet, D.J., Black, M.J.: A layered motion representation with occlusion and compact spatial support. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part I. LNCS, vol. 2350, pp. 692–706. Springer, Heidelberg (2002), http://dx.doi.org/10.1007/3-540-47969-4_46
Jin, H., Favaro, P., Cipolla, R.: Visual tracking in the presence of motion blur. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 2, pp. 18–25 (2005)
Jojic, N., Frey, B.: Learning flexible sprites in video layers. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. I-199–I-206 (2001)
Kolmogorov, V., Zabin, R.: What energy functions can be minimized via graph cuts? IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) 26(2), 147–159 (2004)
Lee, S.H., Moon, N.S., Lee, C.W.: Recovery of blurred video signals using iterative image restoration combined with motion estimation. In: Int. Conf. on Image Processing (ICIP), vol. 1, pp. 755–758 (1997)
Levin, A.: Blind motion deblurring using image statistics. In: Schölkopf, B., Platt, J., Hoffman, T. (eds.) Advances in Neural Information Processing Systems, vol. 19, pp. 841–848. MIT Press, Cambridge (2007)
Li, Y., Kang, S.B., Joshi, N., Seitz, S., Huttenlocher, D.: Generating sharp panoramas from motion-blurred videos. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 2424–2431 (2010)
Lin, H.T., Tai, Y.W., Brown, M.: Motion regularization for matting motion blurred objects. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) 33(11), 2329–2336 (2011)
Nir, T., Kimel, R., Bruckstein, A.: Variational approach for joint optic-flow computation and video restoration. Tech. Rep. CIS-2005-03, Technion Israel Institute of Technology (2005)
Paramanand, C., Rajagopalan, A.: Non-uniform motion deblurring for bilayer scenes. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1115–1122 (2013)
Pawan Kumar, M., Torr, P., Zisserman, A.: Learning layered motion segmentations of video. International Journal of Computer Vision (IJCV) 76(3), 301–319 (2008), http://dx.doi.org/10.1007/s11263-007-0064-x
Portz, T., Zhang, L., Jiang, H.: Optical flow in the presence of spatially-varying motion blur. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1752–1759 (2012)
Rav-Acha, A., Kohli, P., Rother, C., Fitzgibbon, A.: Unwrap mosaics: A new representation for video editing. ACM Transactions on Graphics (TOG) - Proc. SIGGRAPH 27(3), 17:1–17:11 (2008)
Schoenemann, T., Cremers, D.: A coding cost framework for super-resolution motion layer decomposition. IEEE Trans. Im. Proc. 23(3), 1097–1110 (2012)
Seitz, S., Baker, S.: Filter flow. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 143–150 (2009)
Sellent, A., Eisemann, M., Goldlucke, B., Cremers, D., Magnor, M.: Motion field estimation from alternate exposure images. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) 33(8), 1577–1589 (2011)
Shan, Q., Xiong, W., Jia, J.: Rotational motion deblurring of a rigid object from a single image. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1–8 (2007)
Sun, D., Sudderth, E., Black, M.J.: Layered segmentation and optical flow estimation over time. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1768–1775 (2012)
Sun, D., Wulff, J., Sudderth, E., Pfister, H., Black, M.: A fully-connected layered model of foreground and background flow. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Portland, OR, pp. 2451–2458 (June 2013)
Takeda, H., Milanfar, P.: Removing motion blur with space-time processing. IEEE Transactions on Image Processing 20(10), 2990–3000 (2011)
Torr, P.H.S., Szeliski, R., Anandan, P.: An integrated bayesian approach to layer extraction from image sequences. In: IEEE Int. Conf. on Computer Vision (ICCV), vol. 2, pp. 983–990 (1999)
Wang, J.Y.A., Adelson, E.H.: System for encoding image data into multiple layers representing regions of coherent motion and associated motion parameters. US Pat. 5557684 (1996)
Wang, J., Adelson, E.: Representing moving images with layers. IEEE Trans. Image Processing 3(5), 625–638 (1994)
Weiss, Y.: Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In: IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Washington, DC, USA, pp. 520–526 (1997), http://dl.acm.org/citation.cfm?id=794189.794437
Whyte, O., Sivic, J., Zisserman, A., Ponce, J.: Non-uniform deblurring for shaken images. International Journal of Computer Vision (IJCV) 98(2), 168–186 (2012)
Xu, L., Jia, J., Matsushita, Y.: Motion detail preserving optical flow estimation. IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI) 34(9), 1744–1757 (2012)
Yalcin, H., Black, M.J., Fablet, R.: The dense estimation of motion and appearance in layers. In: IEEE Workshop on Image and Video Registration (June 2004)
Yamaguchi, T., Fukuda, H., Furukawa, R., Kawasaki, H., Sturm, P.: Video deblurring and super-resolution technique for multiple moving objects. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part IV. LNCS, vol. 6495, pp. 127–140. Springer, Heidelberg (2011), http://dl.acm.org/citation.cfm?id=1966111.1966123
Zhou, Y., Tao, H.: A background layer model for object tracking through occlusion. In: IEEE Int. Conf. on Computer Vision (CVPR), vol. 2, pp. 1079–1085 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
1 Electronic Supplementary Material
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Wulff, J., Black, M.J. (2014). Modeling Blurred Video with Layers. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds) Computer Vision – ECCV 2014. ECCV 2014. Lecture Notes in Computer Science, vol 8694. Springer, Cham. https://doi.org/10.1007/978-3-319-10599-4_16
Download citation
DOI: https://doi.org/10.1007/978-3-319-10599-4_16
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-10598-7
Online ISBN: 978-3-319-10599-4
eBook Packages: Computer ScienceComputer Science (R0)