Error Concealment in Encoded Video Streams

  • Paul Salama
  • Ness B. Shroff
  • Edward J. Delp
Chapter

Abstract

In ATM networks cell loss causes data to be dropped, which results in the loss of entire macroblocks when MPEG video is being transmitted. In order to reconstruct the missing data, the location of these macroblocks must be known. We describe a technique for packing ATM cells with compressed data, with the aim of detecting the location of missing macroblocks in the encoded video stream. This technique also permits proper decoding of correctly received macroblocks, and thus prevents the loss of ATM cells from affecting the decoding process. We also describe spatial and temporal techniques for the recovery of lost macroblocks. The spatial techniques fall into two categories: deterministic and statistical. A deterministic spatial approach we provide aims at reconstructing each lost pixel by spatial interpolation from the nearest undamaged pixels. Another, recovers lost macroblocks by minimizing inter-sample variations within each block and across its boundaries. In the statistical approach, each frame is modeled as a Markov Random Field, and a maximum a posteriori (MAP) estimate of the missing macroblocks is obtained based on this model. The MAP estimate for each pixel within a lost macroblock is obtained by means of the iterative conditional modes (ICM) algorithm. The iterative method is guaranteed to converge to a global maximum, even though the global maximum is not unique. It is shown that, for each pixel, the median of its neighbors is a MAP estimate. In temporal reconstruction, a search is carried out over a reference frame for the macroblock sized region that will maximize the posterior distribution of the lost macroblock given its neighbors.

Keywords

Motion Vector Asynchronous Transfer Mode Discrete Cosine Transform Coefficient Error Concealment Markov Random Field Model 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    J. Watkinson, The art of digital video. Oxford, England: Focal Press, second ed., 1994.Google Scholar
  2. [2]
    C. P. Sandbank, Digital Television. John Wiley and Sons, 1992.Google Scholar
  3. [3]
    A. M. Tekalp, Digital Video Processing. Prentice Hall, 1995.Google Scholar
  4. [4]
    A. A. Rodriguez, C. E. Fogg, and E. J. Delp, “Video compression for multimedia applications,” in Image Technology: Advances in Image Processing, Multimedia and Machine Vision, pp. 613–679, Springer-Verlag, 1996.Google Scholar
  5. [5]
    MPEG-1 Standard, ISO/IEC 11172, Information Technology—coding of moving pictures and associated audio for digital storage media at up to about 1.5 Mbits/s. ISO, 1993.Google Scholar
  6. [6]
    MPEG-2 Standard, ISO/IEC 13818, Generic Coding of Moving Pictures and Associated Audio Information. ISO, 1995.Google Scholar
  7. [7]
    D. LeGall, “MPEG: A video compression standard for multimedia applications,” Communications of the ACM, vol. 34, no. 4, pp. 47–58, April 1991.MathSciNetGoogle Scholar
  8. [8]
    V. Bhaskaran and K. Konstantinides, Image and Video Compression Standards: Algorithms and Architectures. Kluwer Academic Publisher, 1995.Google Scholar
  9. [9]
    J. L. Black, W. B. Pennebaker, C. E. Fogg, and D. J. LeGall, MPEG Video Compression Standard. Digital Multimedia Standards Series, Chapman and Hall, 1996.Google Scholar
  10. [10]
    K. R. Rao and J. J. Hwang, Techniques and Standards for Image, Video and Audio Coding. Prentice Hall, 1996.Google Scholar
  11. [11]
    Requirements, Audio, DMIF, SNHC, Systems, Video, “Overview of the MPEG4 standard.” Stockholm meeting, document ISO/IEC JTC1/SC29/WG11 N1730, July 1997, July 1997.Google Scholar
  12. [12]
    MPEG Requirements Group, “MPEG-7: Context and objectives (version 4).” Stockholm meeting, document ISO/IEC JTC1/SC29/WG11 N1733, July 1997, July 1997.Google Scholar
  13. [13]
    A. K. Jain, Fundamentals of digital image processing. Prentice Hall Information and System Sciences Series, Prentice Hall, 1989.MATHGoogle Scholar
  14. [14]
    MPEG-1 Video Coding Standard, ISO/IEC 11172–2, Information Technology—coding of moving pictures and associated audio for digital storage media at up to about 1. 5 Mbits/s—Part2: Video. ISO,1993.Google Scholar
  15. [15]
    J. R. Jain and A. K. Jain, “Displacement measurement and its application in interframe image coding,” IEEE Transactions on Communications, vol. COM29, no. 12, pp. 1799–1808, December 1981.CrossRefGoogle Scholar
  16. [16]
    MPEG-1 Systems Standard, ISO/IEC 11172–1, Information Technology—coding of moving pictures and associated audio for digital storage media at up to about 1. 5 Mbits/s—Partl: Systems. ISO,1993.Google Scholar
  17. [17]
    MPEG-2 Systems Standard, ISO/IEC 13818–1, Generic Coding of Moving Pictures and Associated Audio Information—Part1: Systems. ISO,1995.Google Scholar
  18. [18]
    B. G. Haskell, A. Puri, and A. N. Netravali, Digital Video: An Introduction to MPEG-2. Digital Multimedia Standards Series, Chapman and Hall, 1997.Google Scholar
  19. [19]
    G. C. Kessler and P. Southwick, ISDN Concepts, Facilities and Services. McGraw-Hill Series on Computer Communications, McGraw-Hill, third ed., 1997.Google Scholar
  20. [20]
    U. Black, ATM: Foundation for broadband networks. Prentice Hall Series in Advanced Communications Technology, Prentice Hall, 1995.Google Scholar
  21. [21]
    V. Kumar, Broadband communications. McGraw-Hill Series on Computer Communications, McGraw-Hill, 1995.Google Scholar
  22. [22]
    L. G. Cuthbert and J.-C. Sapanel, ATM: The broadband telecommunications solution. IEE Telecommunications Series 29, IEE, 1993.Google Scholar
  23. [23]
    M. Wada, “Selective recovery of video packet loss using error concealment,” IEEE Journal on Selected Areas in Communication, vol. 7, no. 5, pp. 807–814, June 1989.CrossRefGoogle Scholar
  24. [24]
    F. Kishino, K. Matanabe, Y. Hayashi, and H. Yasuda, “Variable bit rate coding of video signals for ATM networks,” IEEE Journal on Selected Areas in Communications, vol. 7, no. 5, pp. 801–806, June 1989.CrossRefGoogle Scholar
  25. [25]
    M. Ghanbari and V. Seferidis, “Cell-loss concealment in ATM networks,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 3, no. 3, pp. 238–247, June 1993.Google Scholar
  26. [26]
    M. Ghanbari and C. Hughes, “Packing coded video signals into ATM cells,” IEEE/ACM Transactions on Networking, vol. 1, no. 5, pp. 505–508, October 1993.CrossRefGoogle Scholar
  27. [27]
    P. Pancha and M. El Zarki, “MPEG coding for variable bit rate video transmission,” IEEE Communications Magazine, vol. 32, no. 5, pp. 54–66, May 1994.CrossRefGoogle Scholar
  28. [28]
    W. Luo and M. El Zarki, “Analysis of error concealment schemes for MPEG-2 video transmission over ATM based networks,” Proceedings of the SPIE Conference on Visual Communications and Image Processing,vol. 2501/3, May 1995, Taipei, Taiwan, pp. 1358–1368.Google Scholar
  29. [29]
    L. H. Kieu and K. N. Ngan, “Cell loss concealment techniques for layered video codec in an ATM network,” Computers and Graphics: Image Communication, vol. 18, no. 1, pp. 11–19, January-February 1994.MathSciNetCrossRefGoogle Scholar
  30. [30]
    D. Raychaudhuri, H. Sun, and R. S. Girons, “ATM transport and cell-loss concealment techniques for MPEG video,” Proceedings of the International Conference on Acoustics, Speech and Signal Processing, November 1993, Minneapolis, Minnesota, pp. 117–120.Google Scholar
  31. [31]
    A. S. Tom, C. L. Yeh, and F. Chu, “Packet video for cell loss protection using deinterleaving and scrambling,” Proceedings of the International Conference on Acoustics, Speech and Signal Processing, May 1991, Toronto, Canada, pp. 2857–2860.Google Scholar
  32. [32]
    Q. Zhu, Y. Wang, and L. Shaw, “Coding and cell loss recovery in DCT based packet video,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 3, no. 3, pp. 248–258, June 1993.CrossRefGoogle Scholar
  33. [33]
    J.Y. Park, M.H. Lee, and K.J. Lee, “A simple concealment for ATM bursty cell loss,” IEEE Transactions on Consumer Electronics, vol. 39, no. 3, pp. 704–710, August 1993.CrossRefGoogle Scholar
  34. [34]
    Y. Wang, Q. Zhu, and L. Shaw, “Maximally smooth image recovery in transform coding,” IEEE Transactions on Communications, vol. 41, no. 10, pp. 1544–1551, October 1993.MATHCrossRefGoogle Scholar
  35. [35]
    Y. Wang and Q. Zhu, “Signal loss recovery in DCT-based image and video codecs,” Proceedings of the SPIE Conference on Visual Communications and Image Processing, vol. 1605, November 1991, Boston, Massachusetts, pp. 667–678.Google Scholar
  36. [36]
    L. T. Chia, D. J. Parish, and J. W. R. Griffiths, “On the treatment of video cell loss in in the transmission of motion-JPEG and JPEG images,” Computers and Graphics: Image Communication, vol. 18, no. 1, pp. 11–19, January-February 1994.CrossRefGoogle Scholar
  37. [37]
    H. Sun and J. Zdepski, “Adaptive error concealment algorithm for MPEG compressed video,” Proceedings of the SPIE Conference on Visual Communications and Image Processing, vol. 1818, November 1992, Boston, Massachusetts, pp. 814–824.Google Scholar
  38. [38]
    W. Kwok and H. Sun, “Multidirectional interpolation for spatial error concealment,” IEEE Transactions on Consumer Electronics, vol. 3, no. 39, pp. 455–460, August 1993.CrossRefGoogle Scholar
  39. [39]
    H. Sun and W. Kwok, “Concealment of damaged block transform coded images using projections onto convex sets,” IEEE Transactions on Image Processing, vol. 4, no. 4, pp. 470–477, April 1995.CrossRefGoogle Scholar
  40. [40]
    P. Salama, N. Shroff, E. J. Coyle, and E. J. Delp, “Error concealment techniques for encoded video streams,” Proceedings of the International Conference on Image Processing,October 23–26 1995, Washington, DC, pp. 9–12.Google Scholar
  41. [41]
    P. Salama, N. Shroff, and E. J. Delp, “A bayesian approach to error concealment in encoded video streams,” Proceedings of the International Conference on Image Processing,September 16–19 1996, Lausanne, Switzerland, pp. 49–52.Google Scholar
  42. [42]
    S. Aign and K. Fazel, “Error detection and concealment measures in MPEG-2 video decoder,” Proceedings of the International Workshop on HDTV, October 1994, Torino, Italy.Google Scholar
  43. [43]
    S. Aign and K. Fazel, “Temporal and spatial error concealment techniques for hierarchical MPEG-2 video codec,” Proceedings of the International Workshop on Communications,June 18–20 1995, Seattle, Washington, pp. 1778–1783.Google Scholar
  44. [44]
    S. Aign, “Error concealment enhancement by using the reliability outputs of a SOVA in MPEG-2 video decoder,” Proceedings of the URSI International Symposium on Signal, Systems, and Electronics,October 25–27 1995, San Francisco, California, pp. 59–62.Google Scholar
  45. [45]
    J. F. Shen and H. M. Hang, “Compressed image concealmnet and postprocessing for digital video recording,” Proceedings of the IEEE Asia-Pacific Conference on Circuits and Systems,December 5–8 1994, Taipei, Taiwan, pp. 636–641.Google Scholar
  46. [46]
    D. C. Youla and H. Webb, “Image restoration by the method of convex projections: Part 1 - theory,” IEEE Transactions on Medical Imaging, vol. MI-1, no. 2, pp. 81–94, October 1982.Google Scholar
  47. [47]
    W. Grimson, “An implementation of a computational theory of visual surface interpolation,” Computer Vision, Graphics, Image Processing, vol. 22, no. 1, pp. 39–69, April 1983.MathSciNetCrossRefGoogle Scholar
  48. [48]
    S. Geman and D. Geman, “Stochastic relaxation, gibbs distributions, and the bayesian restoration of images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-6, no. 6, pp. 721–741, November 1984.MATHCrossRefGoogle Scholar
  49. [49]
    R. L. Stevenson, B. E. Schmitz, and E. J. Delp, “Discontinuity preserving regularization of inverse visual problems,” IEEE Transactions on Systems Man and Cybernetics, vol. 24, no. 3, pp. 455–469, March 1994.CrossRefGoogle Scholar
  50. [50]
    D. Geman and G. Reynolds, “Constrained restoration and the recovery of discontinuities,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 3, pp. 367–382, March 1992.CrossRefGoogle Scholar
  51. [51]
    C. Bouman and K. Sauer, “A generalized gaussian image model for edge-preserving MAP estimation,” IEEE Transactions on Image Processing, vol. 2, no. 3, pp. 296–310, July 1993.CrossRefGoogle Scholar
  52. [52]
    J. Marroquin, S. Mitter, and T. Poggio, “Probablistic solution of ill-posed problems in computational vision,” Journal of the American Statistical Association, vol. 82, no. 397, pp. 76–89, March 1987.MATHCrossRefGoogle Scholar
  53. [53]
    R. Kinderman and J. L. Snell, Markov Random Fields and their Applications, vol. 1 of Contemporary Mathematics. American Mathematical Society, 1980.Google Scholar
  54. [54]
    J. Besag, “Spatial interaction and the statistical analysis of lattice systems,” Journal of the Royal Statistical Society, series B, vol. 36, pp. 192–326, 1974.MathSciNetMATHGoogle Scholar
  55. [55]
    R. Schultz and R. L. Stevenson, “A bayesian approach to image expansion for improved definition,” IEEE Transactions on Image Processing, vol. 3, no. 3, pp. 233–241, May 1994.CrossRefGoogle Scholar
  56. [56]
    P. J. Huber, Robust Statistics. John Wiley & Sons, 1981.Google Scholar
  57. [57]
    J. Besag, “On the statistical analysis of dirty pictures,” Journal of the Royal Statistical Society, series B, vol. 48, no. 3, pp. 259–302, 1986.MathSciNetMATHGoogle Scholar
  58. [58]
    D. G. Luenberger, Linear and Nonlinear Programming. Addison Wesley, second ed., 1989.Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 1998

Authors and Affiliations

  • Paul Salama
    • 1
  • Ness B. Shroff
    • 1
  • Edward J. Delp
    • 1
  1. 1.Video and Image Processing Laboratory (VIPER) School of Electrical and Computer EngineeringPurdue UniversityWest LafayetteUSA

Personalised recommendations