Keywords

1 Introduction

With the development of computer technology, the way of human computer interaction, from the initial punch card to keyboard mouse and to touch handwritten input has been in evolution and improvement. At present, use of touch screen input gives people an experience of new fashion. However, many mobile devices such as mobile phone are limited by size, resulting in that the handwriting input cannot be performed on the big screen. So, looking for more convenient, more efficient and more intelligent human-computer interaction method is an important research topic in the field of embedded human computer interaction. Touch-less handwritten input using Finger Gesture Movement instead of touch screen with camera is more people-oriented. The fundamental is recognizing the Finger Gesture of hand in the air firstly, and then tracking the finger’s image sequence acquired frame by frame using digital image processing technology so as to construct an image of the character written, recognize the character finally. Along with widely use of the electronic devices equipped with camera, the method of character input proposed in this paper will be a good choice, especially in all kinds of multimedia teaching and intelligent information processing. Using the Finger Gesture Movement spot’s characteristics of color, pixel area, and shape to overcome the environmental Finger Gesture Movement unstable factors impacting on the results of Finger Gesture Movement spot detection, so as to detected the Finger Gesture Movement spot effectively from the image sequences acquired by the camera, and it is very important for the subsequently recovering handwritten character information.

2 Related Work

A symbol of physical behavior or emotional expression is named Gesture. The mix of body gesture and hand gesture makes it. Static gesture [1,2,3,4] and Dynamic gesture [5,6,7,8] are main categories. For the previous, a proof is denoted by the posture of the body or the gesture of the hand. For the latter, some messages area unit sent by the movement of the body or the hand. As a tool of communication between computer and human, gesture is employed [9,10,11]. It is greatly completely different from the standard hardware based mostly strategies. Through gesture recognition, it can accomplish human-computer interaction. The gesture or movement of the body or body parts is recognized by the gesture recognition determines the user intent. Several researchers have strived to enhance the hand gesture recognition technology in within the past decades. To recognize America sign language, the authors discover the hand region; then track and analyze the moving path [12, 13]. The import role is contend by hand gesture recognition in several applications like sign language recognition [12,13,14,15], augmented reality (virtual reality) [16–19], signing interpreters for the disabled [20], and robot control [21, 22]. From completely different sources like vision, the recognition systems are ready to gather information regarding user’s performed gestures [23], accelerometers [24], touch-sensitive surfaces [25] gyroscopes, or perhaps magnetic field sensors. The last 3 parts are usually integrated as MEMS within mobile devices. Also, for authentication functions, the gestures are perceived as helpful data source within the field of security. Following the most plan of using gestures rather than commonplace text password completely different analysis and implementations are developed involving accelerometer-based recognition [24], remote palm-based gestures [26, 27] or touch-based drawing gestures [25]. Owing to their usability in sensible applications, we tend to specialize in the supposed natural kind of gestures. Although it may be classified as behavioral biometrics like handwritten signatures [28, 29]; over physiological biometric, this sort of gestures inherits additionally some blessings.

3 Proposed Approach

The proposed method comprises of 2 phases:

  1. 1.

    Finger Gesture Recognition (Fig. 1)

    Fig. 1.
    figure 1

    (Source: [32])

    Examples of natural gestures

    Full size image
  2. 2.

    Visible Spot Detection and Character Recognition

3.1 Figure and Gesture Recognition

The summary of the fingers gesture recognition is represented in Fig. 2. First, the hand is detected using the background subtraction methodology and therefore the result of hand detection is remodeled to a binary image. Then, the fingers and palm are segmented therefore as to facilitate the finger recognition. Moreover, the fingers are detected and recognized.

Fig. 2.
figure 2

The overview of the proposed method for fingers gesture recognition

Full size image

3.1.1 Detection of Hand

When camera mood on and the hand is in the camera reason, then with a standard camera i.e. webcam these images are captured. In Fig. 3, for hand gesture recognition, the original images employed in the work are incontestable. Under identical conditions, these hand images are taken. It is necessary that background of those images is identical. Therefore, using the background subtraction methodology, it is straightforward and effective to find the hand region from the original image. However, in some cases, there are alternative moving objects enclosed within the results of background subtraction. The coloring is used to discriminate the hand region from the opposite moving objects. The color of the skin is measured with the HSV (hue, saturation, and value) model.

Fig. 3.
figure 3

The procedure of hand detection (Source: [30]). (Color figure online)

Full size image

3.1.2 Fingers and Palm Segmentation

The output of the hand detection is a binary image during which the white pixels are the members of the hand region, whereas the black pixels belong to the background. Then, the subsequent procedure is enforced on the binary hand image to segment the fingers and palm

  • Point of Palm. The point of palm is defined as the center point of the palm. By using distance transform, it is found. Distance transform also called distance map is a representation of an image. In the distance transform image, each pixel records the distance of it and the nearest boundary pixel. To measure the distances between the pixels and the nearest boundary pixels, the block city distance is used.

  • Maximal Radius’ Inner Circle. It can draw a circle with the palm point because the center point within the palm once the palm point is found. Because of it’s included within the palm, the circle is termed the inner circle. Until it reaches the edge of the palm, the radius of the circle gradually increases. Once the black pixels are enclosed within the circle that is the radius of the circle stops to extend.

  • Wrist Points and Palm Mask. When the radius of the maximal inner circle is acquired, a larger circle the radius of which is 1.2 times of that of the maximal inner circle is produced (Fig. 4).

    Fig. 4.
    figure 4

    (Source: [30])

    Maximal radius’ wrist points, palm point, the wrist line and the inner circle

    Full size image

3.1.3 Recognition of Fingers’ Gesture

The labeling algorithmic rule is applied to mark the regions of the fingers within the segmentation image of fingers. The detected regions within which the amount of pixels is just too small is thought to be buzzing regions and discarded within the results of the labeling technique. Solely the regions of enough sizes are thought to be fingers and stay. For every remained region, that is, a finger, the minimal bounding box is found to surround the finger. Then, the center of the minimal bounding box is employed to represent the center point of the finger.

  • Thumb’s Detection and Recognition. The centers of the fingers are lined to the palm point. Then, the degrees between the lines and the gliding joint line are computed. It implies that the thumb seems within the hand image, if there is a degree smaller than 50°. The corresponding center is that the center purpose of the thumb, the thumb does not exist within the image, if all the degrees are larger than 50°.

  • Other Fingers’ Detection and Recognition. In order to observe and recognize the other fingers, the palm line parallels to the wrist joint line. The palm line is searched within the way: begin from the row of the wrist joint line. A line paralleling to the wrist joint line crosses the hand for every row. The road shifts upward, if there is only one connected set of white pixels within the intersection of the road and therefore the hand. The road is considered a candidate of the palm line, once there are quite one connected sets of white pixels within the intersection of the road and therefore the hand. The road crossing the hand with quite one connected sets of white pixels in their inter-section is chosen because the palm line, within the case of the thumb not detected. The road continues to maneuver upward with the edge points of the palm rather than the thumb because the starting point of the road, within the case of the thumb existing. There is only one connected set of pixels within the intersection of the road and therefore the hand, since the thumb is removed. It falls into sure components, according to the horizontal coordinate of the center point of a finger. It is the forefinger, if the finger falls into the first half. It is the center finger, if the finger belongs to the second part. The third part corresponds to the ring finger. The fourth part is that the little finger.

3.2 Finger Visible Spot Detection and Character Recognition

Using the finger spot’s characteristics of color, pixel area, and shape to beat the environmental light unstable factors impacting on the results of finger spot detection, thus on detected the finger spot effectively from the image sequences uninheritable by the camera, and it is important for the after convalescent handwritten character information.

3.2.1 The Improvement of Visible Finger Spot Detection

Finger spot detection is that the basis and premise for the following synthesis and recognition of handwritten character operations. The hardiness of finger spot detection technique adopted in paper is low and it is not appropriate for the nice changes setting. Additionally, it solely uses the objects visual property to segment object, within which H plane is employed to segment object roughly, then the thing is segmented intimately through the S or L plane that has the same H element [31]. So as to create the finger spot detection additional sturdy, over one characteristic of finger spot like color, pixel area, and form is adopted to notice finger spot effectively from the image sequence during this paper.

The specific segmentation method is as follows:

  • At first, the images of visual finger spot movement in air is segmented roughly utilizing the light spot color feature frame by frame

  • Secondly, using binary morphology operation the segmented images are processed so as to remove tiny noise and fill-in object voids.

  • Lastly, the processed images are filtered utilizing characteristic parameters - shape and pixel area of the finger spot to remove the big pixel area noise except the finger spot.

The detection of finger spot flow chart is shown in Fig. 5.

Fig. 5.
figure 5

The detection of finger spot flow chart

Full size image

The result of image segmentation is not an equivalent in numerous color space. Therefore, HSL color space has been chosen so as to segment finger spot from the image sequence effectively. The elaborated reason is that R, G, B color plane are robust correlation with one another in RGB color space, however, in HSL color space, solely H and S contain the image color information, and L is tangential with color information [33]. The result of image segmentation is not an equivalent in numerous color space. Therefore, HSL color space has been chosen so as to segment finger spot from the image sequence effectively. The elaborated reason is that R, G, B color plane are robust correlation with one another in RGB color space, however, in HSL color space, solely H and S contain the image color information, and L is tangential with color information [34]. The second technique above is chosen in this paper, the rationale is as follows. Once somebody is writing some characters with a finger in air, the camera captures not solely the finger spot image, however conjointly the one who are writing in air. The person as a part of the background has been regularly moving, as a result the background of the finger spot is correspondingly ever-changing and background threshold has also modified, this makes it laborious to spot applicable threshold to segment the finger spot. So the gray threshold segmentation technique is not appropriate for this case, the second image segmentation technique using color features is adopted during this paper. The precise segmentation steps are: get the color histogram of the finger spot in HSL color space within the first place, so acquire the binary image segmentation of finger spot roughly through the color histogram. As a result of color histogram is three-dimensional, so as to look at the distribution of each color plane intuitively, 3 separated one-dimension histogram is adopted to show H, S, and L color histogram of the finger spot. Finger spot color is misdistribution (the marginal color of the finger spot is darker), this end in that the H element histogram seems two peaks at intervals the scope of 0–2 and 210–215. So as to filter the background and different interference color effectively, H threshold hand-picked should embrace all pixels’ color. Therefore we decide 0–2 and 210–215 because the H threshold range. The S element seems single-peak in histogram as color saturation, all pixels’ color saturation distribute in 0–10 interval in L is brightness element, and it’s irrelevant with the color information, that has a single-peak within the scope of 250–255.

The segmentation of the finger spot in above uses solely color information of the finger spot, however the instability of environmental light leads to stronger background noise. It will be seen from the light outside of the window is brighter than the light within the space, and also the camera is facing to associate unstable surroundings light, which ends therein the segmental finger spot binary image not solely has terribly little particles noise, however additionally has giant area noise. So as to attain ideal segmentation impact, we should always adopt binary morphology operation and particle removal methodology to get rid of completely different styles of noise severally. Binary morphology open operation has operated of filter little noise and also the shut operation can swish object contour and fill-in object voids. Once the higher than operations with 3 × 3 square structural components. It will be said that the tiny noise is filtered effectively by binary morphology operation, however the noise whose component space is much outweigh the visible particle noise still exists. Therefore, it is necessary to filter to get rid of giant noise. As a result of the finger spot has bound pixel area and its image approximate to circular shape, the particle removal operation has been accustomed.

According to the above analysis, we can conclude that the measurement of shape and pixel area characteristic parameters of the finger spot is that the key. If characteristic screening parameters utilized in filter deviate from the practice price apparently, the finger spot are filtered or the noise will not be removed. Therefore it is necessary to measure and statistic the shape and pixel area parameters range of finger spot. The histograms of pixel space and shape parameter of the finger spot are calculated, in which 200–400 distribution of pixels area and 1.01–1.19 distribution of shape parameter. If detected finger spot is simply too massive, the recovered handwritten character stroke will be too thick in order that the character cannot be recognized. The detected finger spot is replaced with circle, within which center position is finger spot’s center coordinate and nine pixels is adopted as radius r (combined with the national standard characters related regulations) to revive the finger spot so as to ensure the character recognition rate. Ultimately mirroring image sequence of the finger spot to show within the right direction in image window after finger spot is restored.

3.2.2 Handwritten Input Character’s Synthesis and Recognition

In order to make an entire character image, logical ‘or’ operation has been per-formed on the all above processed images. a new frame image can be uninheritable once a logical ‘or’ operation is completed between the first frame image and also the second frame image, then a logical ‘or’ operation is completed once more between the new frame image and also the third frame image and then on, so that the finger spot movement will be continuously and displayed in image window at the same time, achieving writing by hand whereas simultaneously showing the handwritten character. By comparing the synthesis characters with the feature database, the character that has the highest similarity is that the final recognition result. In this paper, we use OCR training Interface module to complete character threshold segmentation, specify the interested region and regulate the kerning after completing single character division, then recognize every characters and output the result. The experiment result indicates that the OCR training Interface module can effectively identify synthetic handwritten character and provides a convenient way for character recognition.

3.3 Experimental Results

In the experiments, 2 data sets of hand gestures are accustomed appraise the performance of the projected technique. The data set 1 is an image assortment of thirteen gestures. For every gesture, a hundred pictures area unit captured. So, there are total 1300 images for hand gesture recognition. All the gesture images belong to three females and four males. The dimensions of one gesture image is 640 × 480 (Fig. 6).

Fig. 6.
figure 6

(Source: [30])

Experimental phase of the recognition of finger gesture

Full size image

The accuracy rate of this Hand Written Technique Using Touch-Less Finger Gesture Movement for Human Computer Interaction is about 80.21% (Fig. 7).

Fig. 7.
figure 7

The front panel of the proposed system

Full size image

4 Conclusion

A kind of hand written character input method based on computer vision is presented here. After detecting the finger gesture movement spot, we synthesize a character which conforms to the state prescribed standards of the input character and recognize it on line. By using the handwritten character input method adopted here, it is very helpful for future users to get rid of the traditional man-machine interactive way of keyboard, mouse and so on.