# A Robust Subpixel Motion Estimation Algorithm Using HOS in the Parametric Domain

- 1.5k Downloads
- 1 Citations

**Part of the following topical collections:**

## Abstract

Motion estimation techniques are widely used in todays video processing systems. The most frequently used techniques are the optical flow method and phase correlation method. The vast majority of these algorithms consider noise-free data. Thus, in the case of the image sequences are severely corrupted by additive Gaussian (perhaps non-Gaussian) noises of unknown covariance, the classical techniques will fail to work because they will also estimate the noise spatial correlation. In this paper, we have studied this topic from a viewpoint different from the above to explore the fundamental limits in image motion estimation. Our scheme is based on subpixel motion estimation algorithm using bispectrum in the parametric domain. The motion vector of a moving object is estimated by solving linear equations involving third-order hologram and the matrix containing Dirac delta function. Simulation results are presented and compared to the optical flow and phase correlation algorithms; this approach provides more reliable displacement estimates particularly for complex noisy image sequences. In our simulation, we used the database freely available on the web.

## Keywords

Optical Flow Motion Vector Motion Estimation Parametric Domain Phase Correlation## 1. Introduction

The importance of image sequence processing is constantly growing with the ever increasing use of television and video systems in consumer, commercial, medical, and scientific applications. Image sequences can be acquired by film-based motion picture cameras or electronic video cameras. In either case, there are several factors related to imaging sensor limitations that contribute to the graininess (noise) of resulting images. Electronic sensor noise and film grain are among these factors [1]. In many cases, graininess may result in visually disturbing degradation of the image quality, or it may mask important image information. Even if the noise may not be perceived at full-speed video due to the temporal masking effect of the eye, it often leads to unacceptable single-frame hardcopies and to poor-quality freeze-frames that adversely affect the performance of subsequent image analysis [2].

The motion estimation process must be able to track objects within a noisy source. In a noisy source, objects appear to change from frame to frame because of the noise, not necessarily as the result of object motion [3]. Tracking objects within a noisy environment is difficult, especially if the image frames are severely corrupted by additive Gaussian noises of unknown covariance; second-order statistics methods do not work well.

Higher-order statistics (HOS) in general and the bispectrum (order 3) in particular have recently been widely used as an important tool for signal processing. The classical methods based on the power spectrum are now being effectively superseded by the bispectral ones due to some definite disadvantages of the former. These include the inability to identify systems fed by non-Gaussian noise (NGN) inputs and nonminimum phase (NMP) systems and identification of system nonlinearity [4]. In these cases, the autocorrelation-based methods offer no answer. Out of all these, the identifiability of NMP systems has received the maximum attention from researchers.

HOS-based methods have been proposed to estimate motion between image frames [5, 6, 7, 8, 9]. In, the motion estimation is based on the bispectrum method for sub-pixel resolution of noisy image sequences. In [7], the displacement vector is obtained by maximizing a third-order statistics criterion. In [8], the global motion parameters are obtained by a new region recursive algorithm. In [6], several algorithms are developed based on a parametric cumulant method, a cumulant-matching method, and a mean kurtosis error criterion. The latter is an extension of the quadratic pixel-recursive method by Netravali and Robbins [10]. In [11], it is shown that such statistical parameters are insensitive to additive Gaussian noises. In particular, bispectrum parameters are insensitive to any symmetrically distributed noise and also exhibit the capability of better characterizing NGN and identifying NMP linear systems as well as nonlinear systems. Therefore, transformation to a higher-order domain reduces the effect of noise significantly. In this correspondence, a novel algorithm for the detection of motion vectors in video sequences is proposed. The algorithm uses bispectrum model-based subpixel motion estimation in the parametric domain for noisy image sequences to obtain a measure of content similarity for temporally adjacent frames and responds very well to scene motion vectors. The algorithm is insensitive to the presence of symmetrically distributed noise.

The outline of this paper is as follows. First, the problem formulation is introduced in Section 2. In Section 3, we first present briefly the definitions and properties of the bispecrum and cross-bispectrum. Next, we describe the motion estimation in the parametric domain. High-accuracy subpixel motion estimation is discussed in Section 4. Section 5 presents an evaluation of the computational complexity of our algorithm. The results of the experimental evaluation of the proposed method are shown in Section 6 and compared to existing methods while Section 7 concludes the paper.

## 2. Problem Formulation

where Open image in new window denotes spatial image position of a point; Open image in new window and Open image in new window are observed image intensities at instants Open image in new window and Open image in new window respectively; Open image in new window and Open image in new window are noise-free frames; Open image in new window and Open image in new window are assumed to be spatially and temporally stationary, zero-mean image Gaussian (or non-Gaussian) noise sequences with unknown covariance; and Open image in new window is the displacement vector of the object during the time interval Open image in new window .

The goal is to estimate Open image in new window from Open image in new window and Open image in new window .

## 3. Bispectrum-Based Image Motion Estimation

### 3.1. Definitions and Properties

In this subsection, some HOS functions are defined and their properties are described in order to provide the necessary tools to understand the motion estimation methodology.

where Open image in new window denotes the expectation operation; Open image in new window and Open image in new window are two shifted versions of the Open image in new window .

Also, Open image in new window can be non-Gaussian if it is independent and identically distributed (i.i.d.) and nonskewed (e.g., symmetrically distributed).

where Open image in new window denotes the 4D Fourier transform operation; Open image in new window and Open image in new window are the frequency coordinates for the 2D Fourier transform.

where Open image in new window indicates the complex conjugate.

These symmetry properties reduce the computational burden while calculating the bispectrum.

### 3.2. Parametric Model-Based Motion Estimation

where Open image in new window denotes the 4D inverse Fourier transform operation.

The least-squares solution Open image in new window is obtained and its maximum Open image in new window is determined. The image motion estimate is then Open image in new window .

## 4. High-Accuracy Subpixel Motion Estimation

Subpixel performance is a critical element of the proposed algorithm. With reference to our previously published work [16, 17], we are introducing a number of important new features, which improve the accuracy of the motion estimates.

where Open image in new window denotes the real part of complex array Open image in new window .

that is, the maximum peak of the phase correlation surface and its two neighboring values on either side, vertically and horizontally.

where Open image in new window .

The fractional part Open image in new window of the vertical component can be obtained in a similar way using (24) instead of (23).

Finally the horizontal and vertical components of the subpixel accurate motion estimate are obtained by computing the location of the maxima of each of the above fitted quadratics.

In [18], it is shown that half-pixel accuracy motion vectors lead to a very significant improvement when compared to one pixel accuracy, whereas a higher precision results in negligible changes. Therefore, a half-pixel accuracy was chosen in our simulations.

## 5. Computational Cost Comparison

The majority of the computational cost of the proposed bispectrum is due to the fast Fourier transform (FFT). Therefore, the fundamental computation required for bispectral estimates is given by (7), the triple product of the three individual Fourier transformations, while this computation is straightforward, limitations on computer time and statistical variance impose severe limitations on implementation of the definition of the bispectrum [19]. On the other hand, we take advantage of the symmetrical properties of the bispectrum to reduce the computational complexity and memory requirements of calculating third-order statistics. It can now be calculated in any one sector and mapped onto the others [20].

The phase correlation is estimated by multiplying each coefficient Open image in new window by its complex conjugate, but each component of the bispectrum is estimated by a triple product of Fourier coefficients as demonstrated in (7). Thus, the number of operations required to compute the bispectrum is significantly increased relative to the phase correlation. There are Open image in new window independent components of the bispectrum while there are only Open image in new window independent components of the phase correlation for an Open image in new window image [21].

## 6. Simulation Results

Our experiments have aimed at evaluating the performance of the proposed approach and comparing it with that of the optical flow and phase correlation techniques. For the optical flow method we used the implementation obtained from Bruhn method [22]. In our simulation we used the database freely available on the web at http://vision.middlebury.edu/flow/. We contribute three types of data to test different aspects of all techniques: real sequences of independent motion; realistic synthetic sequences; and high frame-rate video. These sequences have been chosen for their difficult motion and their different characteristics. Although the original sequences are in color, only the luminance component is used to estimate the motion vectors.

*Grove*sequence using the three aforementioned motion estimation methods. Note that for a fair comparison we used optical flow technique and phase correlation algorithm with half-pixel accuracy. The motion vectors estimated between frames 6 and 7 are shown for the

*Grove*sequence. For this particular sequence, our scheme provides the most consistent and reliable motion vector field. Both optical flow and phase correlation algorithms fail to detect the true motion vector. Similar results are shown in Figures 3 and 4 for the motion vectors estimated between frames 2 and 3, and between frames 5 and 6 in the

*Walking*and

*Mequon*sequences, respectively. Both optical flow and phase correlation algorithms produce abrupt motion vector fields. Although these abrupt motion vectors may lead to lower numerical mean squared errors (MSEs), they are incorrect motion vectors. Because of the noise resistant property of the parametric bispectrum method, it produces more reliable estimates. Therefore, our approach motion estimation results globally in motion fields more representative of the true motion in the scene.

*Beanbags*sequence as an example. The motion compensated pictures using three methods are shown in Figure 5. Portions of these three pictures are enlarged in Figure 6 to show the differences. We observe better compensated images by the proposed method. We also observe that the motion compensated images for our scheme are much closer to the original images. Thus, the scheme is able to measure the motion vector more accurately and is more robust in general. Overall, parametric bispectrum scheme typically offers better visual quality images than the other methods.

*Hydrangea*sequence, respectively. The bispectrum retains both amplitude and phase information from the Fourier transform of a signal, unlike the other methods. The phase of the Fourier transform contains important shape information. Therefore, the bispectrum minimizes the influence of the noise and simplifies the identification of the dominant peak on the correlation surface.

*Tempete*and

*Stefan*. These sequences were run for 60 frames with a frame rate of 30 frame/sec. Both sequences are degraded with additive zero-mean Gaussian noise to a signal-to-noise ratio (SNR) of 10 dB. Here we define

The comparison between three methods for the computation time.

Sequences | Methods | MECT (sec) |
---|---|---|

| Our method | 0.49 |

Optical flow | 0.25 | |

Phase correlation | 0.38 | |

| Our method | 0.50 |

Optical flow | 0.26 | |

Phase correlation | 0.37 | |

| Our method | 0.37 |

Optical flow | 0.18 | |

Phase correlation | 0.30 | |

| Our method | 0.37 |

Optical flow | 0.19 | |

Phase correlation | 0.31 | |

| Our method | 0.50 |

Optical flow | 0.25 | |

Phase correlation | 0.38 |

*Tempete*sequence. We perform the motion compensation procedure for each current frame Open image in new window with respect to reference frames Open image in new window , where Open image in new window and Open image in new window . The average PSNR of the motion compensated images is given in Table 2, with

*Tempete*sequence degraded with additive zero-mean Gaussian noise to an SNR of 10 dB.

Average PSNR of motion compensated images for the three motion estimation techniques (unit: dB) for *Tempete* sequence.

Our method | Optical flow | Phase correlation | |
---|---|---|---|

1 | 10.80 | 10.50 | 10,21 |

2 | 10,23 | 9.83 | 9,59 |

3 | 9,93 | 9.59 | 9,25 |

4 | 9,68 | 9.24 | 8,81 |

where Open image in new window is the measured PSNR for frame Open image in new window and Open image in new window is the total number of frames. In Table 2, we observe that the Open image in new window decreases with larger apparent disparity between the global motion of the background and the local motion of the foreground. For each value of Open image in new window , we see that the Open image in new window is higher for the proposed scheme than the other methods.

## 7. Conclusion

In this paper, subpixel motion estimation algorithm using bispectrum in the parametric domain was presented. We have presented a collection of datasets for the evaluation of our method, available on the web at http://vision.middlebury.edu/flow/. In the case of the data is severely corrupted by additive Gaussian noises of unknown covariance, our method suppresses the effects of noise and simplifies the identification of the dominant peak on the correlation surface, unlike other techniques. At high noise levels SNR around 10 dB the optical flow and phase correlation techniques fail, yet even under these extreme conditions, the parametric bispectrum provides improvement in performance over the other algorithms. Overall, our scheme produces smoother displacement vector field with a more accurate measure of object motion in different SNR scenarios.

## References

- 1.Benmoussat N, Faouzi Belbachir M, Benamar B:
**Motion estimation and compensation from noisy image sequences: a new filtering scheme.***Image and Vision Computing*2007,**25**(5):686-694. 10.1016/j.imavis.2006.05.010CrossRefGoogle Scholar - 2.Brailean JC, Kleihorst RP, Efstratiadis S, Katsaggelos AK, Lagendijk RL:
**Noise reduction filters for dynamic image sequences: a review.***Proceedings of the IEEE*1995,**83**(9):1272-1292. 10.1109/5.406412CrossRefGoogle Scholar - 3.Armitano RM, Schafer RW, Kitson FL, Bhaskaran V:
**Robust block-matching motion-estimation technique for noisy sources.***Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '97), April 1997, Munich, Germany***4:**2685-2688.Google Scholar - 4.Bhattacharya S, Ray NC, Sinha S:
**2-D signal modelling and reconstruction using third-order cumulants.***Signal Processing*1997,**62**(1):61-72. 10.1016/S0165-1684(97)00115-1CrossRefMATHGoogle Scholar - 5.Ismaili Aalaoui EM, Ibn-Elhaj E:
**Estimation of subpixel motion using bispectrum.***Research Letters in Signal Processing*2008,**2008:**-5.Google Scholar - 6.Anderson JMM, Giannakis GB:
**Image motion estimation algorithms using cumulants.***IEEE Transactions on Image Processing*1995,**4**(3):346-357. 10.1109/83.366482CrossRefGoogle Scholar - 7.Kleihorst RP, Lagendijk RL, Biemond J:
**Noise reduction of severely corrupted image sequences.***Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '93), April 1993, Minneapolis, Minn, USA***5:**293-296.Google Scholar - 8.Ibn-Elhaj E, Aboutajdine D, Pateux S, Morin L:
**HOS-based method of global motion estimation for noisy image sequences.***Electronics Letters*1999,**35**(16):1320-1322. 10.1049/el:19990913CrossRefGoogle Scholar - 9.Sayrol E, Gasull A, Fonollosa JR:
**Motion estimation using higher order statistics.***IEEE Transactions Image Processing*1996,**5**(6):1077-1084. 10.1109/83.503924CrossRefGoogle Scholar - 10.Netravali AN, Robbins JD:
**Motion-compensated television coding—part I.***Bell System Technical Journal*1979,**58**(3):629-668.CrossRefGoogle Scholar - 11.Murino V, Ottonello C, Pagnan S:
**Noisy texture classification: a higher-order statistics approach.***Pattern Recognition*1998,**31**(4):383-393. 10.1016/S0031-3203(97)00055-1CrossRefGoogle Scholar - 12.Sadler BM, Giannakis GB:
**Shift- and rotation-invariant object reconstruction using the bispectrum.***Journal of the Optical Society of America A*1992,**9**(1):57-69. 10.1364/JOSAA.9.000057CrossRefGoogle Scholar - 13.Raghuveer MR, Nikias CL:
**Bispectrum estimation: a parametric approach.***IEEE Transactions on Acoustics, Speech and Signal Processing*1985,**33**(5):1213-1230. 10.1109/TASSP.1985.1164679CrossRefGoogle Scholar - 14.Giannakis GB:
**On the identifiability of non-Gaussian ARMA models using cumulants.***IEEE Transactions on Automatic Control*1990,**35**(1):18-26. 10.1109/9.45139CrossRefMathSciNetMATHGoogle Scholar - 15.Mendel JM:
**Tutorial on higher-order statistics (spectra) in signal processing and system theory: theoretical results and some applications.***Proceedings of the IEEE*1991,**79**(3):278-305. 10.1109/5.75086CrossRefGoogle Scholar - 16.Ismaili Aalaoui EM, Ibn-Elhaj E:
**Estimation of motion fields from noisy image sequences: using generalized cross-correlation methods.***Proceedings of the IEEE International Conference on Signal Processing and Communications (ICSPC '07), November 2007, Dubai, UAE*Google Scholar - 17.Ismaili Aalaoui EM, Ibn Elhaj E:
**Estimation of displacement vector field from noisy data using maximum likelihood estimator.***Proceedings of the 14th IEEE International Conference on Electronics, Circuits, and Systems (ICECS '07), December 2007, Marrakech, Morocco*1380-1383.Google Scholar - 18.Madec G:
**Half pixel accuracy in block matching.***Proceedings on the Picture Coding Symposium (PCS '90), March 1990, Cambridge, Mass, USA*Google Scholar - 19.Lii KS, Helland KN:
**Cross-bispectrum computation and variance estimation.***ACM Transactions on Mathematical Software*1981,**7**(3):284-294. 10.1145/355958.355961CrossRefMathSciNetMATHGoogle Scholar - 20.Le Caillec J-M, Garello R:
**Comparison of statistical indices using third order statistics for nonlinearity detection.***Signal Processing*2004,**84**(3):499-525. 10.1016/j.sigpro.2003.11.013CrossRefMATHGoogle Scholar - 21.Means RW, Wallach B, Busby D:
**Bispectrum signal processing on HNC's SIMD numerical array processor (SNAP).***Proceedings of the ACM/IEEE Conference on Supercomputing (SC '93), November 1993, Portland, Ore, USA*535-537.CrossRefGoogle Scholar - 22.Bruhn A, Weickert J, Schnörr C:
**Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods.***International Journal of Computer Vision*2005,**61**(3):211-231.CrossRefGoogle Scholar

## Copyright information

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.