Video-to-Video Dynamic Super-Resolution for Grayscale and Color Sequences

  • Sina Farsiu
  • Michael Elad
  • Peyman Milanfar
Open Access
Research Article
Part of the following topical collections:
  1. Super-Resolution Imaging: Analysis, Algorithms, and Applications


We address the dynamic super-resolution (SR) problem of reconstructing a high-quality set of monochromatic or color super-resolved images from low-quality monochromatic, color, or mosaiced frames. Our approach includes a joint method for simultaneous SR, deblurring, and demosaicing, this way taking into account practical color measurements encountered in video sequences. For the case of translational motion and common space-invariant blur, the proposed method is based on a very fast and memory efficient approximation of the Kalman filter (KF). Experimental results on both simulated and real data are supplied, demonstrating the presented algorithms, and their strength.


Color Information Technology Real Data Deblurring Quantum Information 


  1. 1.
    Huang TS, Tsai RY: Multi-frame image restoration and registration. In Advances in Computer Vision and Image Processing. Volume 1. JAI Press, Greenwich, Conn, USA; 1984:317–339. chapter 7Google Scholar
  2. 2.
    Nguyen N, Milanfar P, Golub GH: A computationally efficient super-resolution image reconstruction algorithm. Transactions on Image Processing 2001, 10(4):573–583. 10.1109/83.913592CrossRefGoogle Scholar
  3. 3.
    Irani M, Peleg S: Improving resolution by image registration. CVGIP: Graphical Models and Image Processing 1991, 53(3):231–239. 10.1016/1049-9652(91)90045-LGoogle Scholar
  4. 4.
    Elad M, Feuer A: Restoration of a single super-resolution image from several blurred, noisy, and undersampled measured images. Transactions on Image Processing 1997, 6(12):1646–1658. 10.1109/83.650118CrossRefGoogle Scholar
  5. 5.
    Zomet A, Peleg S: Efficient super-resolution and applications to mosaics. Proceedings of IEEE 15th International Conference on Pattern Recognition (ICPR '00), September 2000, Barcelona, Spain 1: 579–583.CrossRefGoogle Scholar
  6. 6.
    Ng MK, Bose NK: Mathematical analysis of super-resolution methodology. IEEE Signal Processing Magazine 2003, 20(3):62–74. 10.1109/MSP.2003.1203210CrossRefGoogle Scholar
  7. 7.
    Altunbasak Y, Patti AJ, Mersereau RM: Super-resolution still and video reconstruction from MPEG-coded video. IEEE Transactions on Circuits and Systems for Video Technology 2002, 12(4):217–226. 10.1109/76.999200CrossRefGoogle Scholar
  8. 8.
    Farsiu S, Robinson MD, Elad M, Milanfar P: Fast and robust multi-frame super-resolution. IEEE Transactions on Image Processing 2004, 13(10):1327–1344. 10.1109/TIP.2004.834669CrossRefGoogle Scholar
  9. 9.
    Segall CA, Katsaggelos AK, Molina R, Mateos J: Bayesian resolution enhancement of compressed video. IEEE Transactions on Image Processing 2004, 13(7):898–911. 10.1109/TIP.2004.827230CrossRefGoogle Scholar
  10. 10.
    Borman S, Stevenson RL: Super-resolution from image sequences—a review. Proceedings of Midwest Symposium on Circuits and Systems (MWSCAS '98), August 1998, Notre Dame, Ind, USA 374–378.Google Scholar
  11. 11.
    Park SC, Park MK, Kang MG: Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine 2003, 20(3):21–36. 10.1109/MSP.2003.1203207CrossRefGoogle Scholar
  12. 12.
    Farsiu S, Robinson MD, Elad M, Milanfar P: Advances and challenges in super-resolution. International Journal of Imaging Systems and Technology 2004, 14(2):47–57. 10.1002/ima.20007CrossRefGoogle Scholar
  13. 13.
    Elad M, Feuer A: Super-resolution restoration of an image sequence: adaptive filtering approach. IEEE Transactions on Image Processing 1999, 8(3):387–395. 10.1109/83.748893CrossRefGoogle Scholar
  14. 14.
    Elad M, Feuer A: Super-resolution reconstruction of image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence 1999, 21(9):817–834. 10.1109/34.790425CrossRefGoogle Scholar
  15. 15.
    Elad M, Hel-Or Y: A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Transactions on Image Processing 2001, 10(8):1187–1193. 10.1109/83.935034CrossRefGoogle Scholar
  16. 16.
    Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution from undersampled color images. Computational Imaging II, January 2004, San Jose, Calif, USA, Proceedings of SPIE 5299: 222–233.CrossRefGoogle Scholar
  17. 17.
    Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing 2006, 15(1):141–159.CrossRefGoogle Scholar
  18. 18.
    Kay SM: Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory. Prentice-Hall, Englewood Cliffs, NJ, USA; 1993.zbMATHGoogle Scholar
  19. 19.
    Jazwinski AH: Stochastic Processes and Filtering Theory. Academic Press, New York, NY, USA; 1970.zbMATHGoogle Scholar
  20. 20.
    Elad M: Super-resolution reconstruction of continuous image sequence, Ph.D. dissertation. Technion-Israel Institute of Technology, Haifa, Israel; 1997.Google Scholar
  21. 21.
    Rauch HE, Striebel CT, Tung F: Maximum likelihood estimates of dynamic linear systems. American Institute of Aeronautics and Astronautics 1965, 3(8):1445–1450.CrossRefGoogle Scholar
  22. 22.
    Harvey AC: Forecasting, Structural Time Series Models and the Kalman Filter. Cambridge University Press, Cambridge, UK; 1990.CrossRefGoogle Scholar
  23. 23.
    Bovik A: Handbook of Image and Video Processing. Academic Press, New York, NY, USA; 2000.zbMATHGoogle Scholar
  24. 24.
    Rudin LI, Osher S, Fatemi E: Nonlinear total variation based noise removal algorithms. Physica D 1992, 60(1–4):259–268.MathSciNetCrossRefGoogle Scholar
  25. 25.
    Li Y, Santosa F: A computational algorithm for minimizing total variation in image restoration. IEEE Transactions on Image Processing 1996, 5(6):987–995. 10.1109/83.503914CrossRefGoogle Scholar
  26. 26.
    Chan TF, Osher S, Shen J: The digital TV filter and nonlinear denoising. IEEE Transactions on Image Processing 2001, 10(2):231–241. 10.1109/83.902288CrossRefGoogle Scholar
  27. 27.
    Tomasi C, Manduchi R: Bilateral filtering for gray and color images. Proceedings of IEEE 6th International Conference on Computer Vision (ICCV '98), January 1998, Bombay, India 839–846.Google Scholar
  28. 28.
    Elad M: On the origin of the bilateral filter and ways to improve it. IEEE Transactions on Image Processing 2002, 11(10):1141–1151. 10.1109/TIP.2002.801126MathSciNetCrossRefGoogle Scholar
  29. 29.
    Laroche C, Prescott M: Apparatus and method for adaptive for adaptively interpolating a full color image utilizing chrominance gradients. United States Patent 5,373,322, 1994Google Scholar
  30. 30.
    Kimmel R: Demosaicing: image reconstruction from color CCD samples. IEEE Transactions on Image Processing 1999, 8(9):1221–1228. 10.1109/83.784434CrossRefGoogle Scholar
  31. 31.
    Keren D, Osadchy M: Restoring subsampled color images. Machine Vision and Applications 1999, 11(4):197–202. 10.1007/s001380050102CrossRefGoogle Scholar
  32. 32.
    Hel-Or Y, Keren D: Demosaicing of color images using steerable wavelets. In Tech. Rep. HPL-2002-206R1 20020830. HP Laboratories Israel, Haifa, Israel; 2002. Online, available: Scholar
  33. 33.
    Muresan DD, Parks TW: Optimal recovery demosaicing. Proceedings of IASTED International Conference on Signal and Image Processing (SIP '02), August 2002, Kauai, Hawaii, USAGoogle Scholar
  34. 34.
    Gunturk BK, Altunbasak Y, Mersereau RM: Color plane interpolation using alternating projections. IEEE Transactions on Image Processing 2002, 11(9):997–1013. 10.1109/TIP.2002.801121CrossRefGoogle Scholar
  35. 35.
    Alleysson D, Süsstrunk S, Hérault J: Color demosaicing by estimating luminance and opponent chromatic signals in the Fourier domain. Proceedings of IS&T/SID 10th Color Imaging Conference, November 2002, Scottsdale, Ariz, USA 331–336.Google Scholar
  36. 36.
    Ramanath R, Snyder WE, Bilbro GL, Sander WA: Demosaicking methods for the Bayer color arrays. Journal of Electronic Imaging 2002, 11(3):306–315. 10.1117/1.1484495CrossRefGoogle Scholar
  37. 37.
    Pei S-C, Tam I-K: Effective color interpolation in CCD color filter arrays using signal correlation. IEEE Transactions on Circuits and Systems for Video Technology 2003, 13(6):503–513. 10.1109/TCSVT.2003.813422CrossRefGoogle Scholar
  38. 38.
    Zomet A, Peleg S: Multi-sensor super-resolution. Proceedings of IEEE 6th Workshop on Applications of Computer Vision (WACV '02), December 2002, Orlando, Fla, USA 27–31.Google Scholar
  39. 39.
    Gotoh T, Okutomi M: Direct super-resolution and registration using raw CFA images. Proceedings of IEEE Computer Society Conference on Computer Vision and Patern Recognition (CVPR '04), June–July 2004, Washington, DC, USA 2: 600–607.Google Scholar
  40. 40.
    Shah NR, Zakhor A: Resolution enhancement of color video sequences. IEEE Transactions on Image Processing 1999, 8(6):879–885. 10.1109/83.766865CrossRefGoogle Scholar
  41. 41.
    Tom BC, Katsaggelos AK: Resolution enhancement of monochrome and color video using motion compensation. IEEE Transactions on Image Processing 2001, 10(2):278–287. 10.1109/83.902292CrossRefGoogle Scholar
  42. 42.
    Pratt WK: Digital Image Processing. 3rd edition. John Wiley & Sons, New York, NY, USA; 2001.CrossRefGoogle Scholar
  43. 43.
    Golland P, Bruckstein AM: Motion from color. Computer Vision and Image Understanding 1997, 68(3):346–362. 10.1006/cviu.1997.0553CrossRefGoogle Scholar
  44. 44.
    Bergen JR, Anandan P, Hanna KJ, Hingorani R: Hierarchical model-based motion estimation. Proceedings of European Conference on Computer Vision (ECCV '92), May 1992, Santa Margherita Ligure, Italy 237–252.Google Scholar

Copyright information

© Farsiu et al. 2006

Authors and Affiliations

  • Sina Farsiu
    • 1
  • Michael Elad
    • 2
  • Peyman Milanfar
    • 1
  1. 1.Electrical Engineering DepartmentUniversity of California Santa CruzSanta CruzUSA
  2. 2.Computer Science DepartmentTechnion – Israel Institute of TechnologyHaifaIsrael

Personalised recommendations