Robust hand gesture recognition system based on a new set of quaternion Tchebichef moment invariants

Abstract

Hand gesture recognition is a challenging task due to the complexity of hand movements and to the variety among the same gesture performed by distinct subjects. Recent technologies, such as Kinect sensor, provide new opportunities, allowing to capture both RGB and depth (RGB-D) images, which offer high discriminant information for efficient hand gesture recognition. In the aspect of feature extraction, the traditional methods process the RGB and depth information independently. In this paper, we propose a robust hand gesture recognition system based on a new feature extraction method, fusing RGB images and depth information simultaneously, by using the quaternion algebra that provide a more robust and holistical representation. In fact, we introduce, for the first time, a novel type of feature extraction method, named quaternion Tchebichef moment invariants. The novelty of the proposed method in this paper lies in the direct derivation of invariants from their orthogonal moments, based on the algebraic properties of the discrete Tchebichef polynomials. The proposed approach based on quaternion algebra is suggested to process the four components holistically, for a robust and efficient hand gesture recognition system. The obtained experimental and theoretical results demonstrate that the present approach is very effective for addressing the problem of hand gesture recognition and have proved its robustness against geometrical distortion, noisy conditions and complex background compared to the state of the art, indicating that it could be highly useful for many computer vision applications.

Appendices

Proof of proposition 1

With the help of Eqs. (15) and (15), the translated version of Tchebichef polynomials can be expressed as:

$$\begin{aligned} \begin{aligned} {t}_n (x-x_0;N)=\sum _{i=0}^{n}A(x-x_0)^i=\sum _{i=0}^{n}\sum _{s=0}^{i} \left( {\begin{array}{c}i\\ s\end{array}}\right) A_{n,i}(-1)^{i-s}\\ x^s(x_0)^{i-s}. \end{aligned} \end{aligned}$$

(33)

By substituting Eq. (15) into Eq. (33), we obtain the relationship between the translated version and traditional Tchebichef polynomials, as follows:

$$\begin{aligned} \begin{aligned} {t}_n(x-x_0;N)=\sum _{i=0}^{n}\sum _{s=0}^{i}\sum _{u=0}^{s}\left( {\begin{array}{c}i\\ s\end{array}}\right) A_{n,i}B_{s,u} (-1)^{i-s}\\ (x_0)^{i-s}{t}_u(x;N). \end{aligned} \end{aligned}$$

(34)

In a similar way, we also have:

$$\begin{aligned} {t}_m(y-y_0;N)=\sum _{j=0}^{m}\sum _{t=0}^{j}\sum _{v=0}^{t}\left( {\begin{array}{c}j\\ t\end{array}}\right) A_{m,j} B_{t,v}(-1)^{j-t}(y_0)^{j-t}{t}_v(y;N). \end{aligned}$$

(35)

Consequently, the $\hbox {QTM}_{n,m}^t$ of a translated image $f^t(x,y)$ can be written in terms of $\hbox {QTM}_n,m$ of the original image f(x, y) as:

$$\begin{aligned} \begin{aligned} \hbox {QTM}_{n,m}^t=&\sum _{i=0}^{n}\sum _{j=0}^{m}\sum _{s=0}^{i}\sum _{t=0}^{j}\sum _{u=0}^{s} \sum _{v=0}^{t}\left( {\begin{array}{c}i\\ s\end{array}}\right) \left( {\begin{array}{c}j\\ t\end{array}}\right) \times A_{n,i}A_{m,j}\\ {}&\times B_{s,u}B_{t,v}(-1)^{i-s+j-t} x_0^{i-s}y_0^{j-t}\hbox {QTM}_{u,v}. \end{aligned} \end{aligned}$$

(36)

As can be concluded, the QTM of any translated image by a translation vector $(x_0,y_0)$ can be expressed in terms of the QTM of the original image.

Therefore, the proof is completed.

Proof of proposition 2

The distorted version of Tchebichef polynomials can be expressed as follows:

$$\begin{aligned} \begin{aligned} {t}_n (a_{1,1}x+a_{1,2}y;N)=\sum _{i=0}^{n}A_{n,i}(a_{1,1}x+a_{1,2}y)^i=\\ \sum _{i=0}^{n}A_{n,i}\sum _{s=0}^{i}\left( {\begin{array}{c}i\\ s\end{array}}\right) (a_{1,1}x)^{i-s}(a_{1,2}y)^s\\ =\sum _{i=0}^{n}\sum _{s=0}^{i}\left( {\begin{array}{c}i\\ s\end{array}}\right) A_{n,i}(a_{1,1})^{i-s}(a_{1,2})^sx^{i-s}y^s. \end{aligned} \end{aligned}$$

(37)

Similarly

$$\begin{aligned} {t}_m (a_{2,1}x+a_{2,2}y;N)=\sum _{j=0}^{m}\sum _{t=0}^{j}\left( {\begin{array}{c}j\\ t\end{array}}\right) A_{m,j} (a_{2,1})^{j-t}(a_{2,2})^tx^{j-t}y^t. \end{aligned}$$

(38)

Consequently, using Eq. (37) and Eq. (38), the $\hbox {QTM}_{n,m}^d$ of a deformed image $f^d (x,y)$ can be written in terms of $\hbox {QTM}_{n,m}$ of the original image f(x, y) as:

$$\begin{aligned} \begin{aligned} \mathrm{QTM}_{n,m}^d=\sum _{i=0}^{n}\sum _{j=0}^{m}\sum _{s=0}^{i}\sum _{t=0}^{j} \sum _{u=0}^{i+j-s-t}\sum _{v=0}^{s+t}\left( {\begin{array}{c}i\\ s\end{array}}\right) \left( {\begin{array}{c}j\\ t\end{array}}\right) \times A_{n,i}A_{m,j}\\ \times B_{i+j-s-t,u}B_{s+t,v}(a_{1,1})^{i-s}(a_{1,2})^{s}(a_{2,1})^{j-t}(a_{2,2})^{t} \mathrm{QTM}_{u,v}. \end{aligned} \end{aligned}$$

(39)

Therefore, the proof is completed.