Real Time Hand Based Robot Control Using 2D/3D Images

Abstract

In the interaction between man and machine, an efficient, natural and intuitive commanding system plays a key role. Vision based techniques are usually used to provide such a system. This paper presents a new solution using 2D/3D images for real time hand detection, tracking and classification which is used as an interface for sending the commands to an industrial robot. 2D/3D images, including low resolution range data and high resolution color information, are provided by a novel monocular hybrid vision system, called MultiCam, at video frame rates. After region extraction and applying some preprocessing techniques, the range data are segmented using an unsupervised clustering approach. The segmented range image is then mapped to the corresponding 2D color image. Due to the monocular setup of the vision system, mapping 3D range data to the 2D color information is trivial and does not need any complicated calibration and registration techniques. Consequently, the segmentation of 2D color image becomes simple and fast. Haar-like features are then extracted from the segmented color image and used as the input features for an AdaBoost classifier to find the region of the hand in the image and track it in each frame. The hand region found by AdaBoost is improved through postprocessing techniques and finally the hand posture (palm and fist) is classified based on a very fast heuristic method. The proposed approach has shown promising results in real time application, even under challenging variant lighting conditions which was demonstrated at the Hannover fair in 2008.