A convolutional neural network with feature fusion for real-time hand posture recognition

Abstract

Gesture based human-computer interaction is both intuitive and versatile, with diverse applications such as in smart houses, operating theaters and vehicle infotainment systems. This work focuses on recognition of static hand gestures, also known as hand postures. A good hand posture recognition system has to be both robust to image variations and capable of real-time performance. Considering the recent success of convolutional neural networks (CNNs) and robustness of more traditional methods, this dissertation presents a novel architecture, combining a CNN and several traditional feature extractors, capable of accurate and real-time hand posture recognition. Several hyperparameters present in the proposed architecture are automatically selected by a model optimization algorithm. The traditional features are extracted from Zernike moments, Hu moments, Gabor filters and properties of the hand contour. This features are used to complement the information available to the classification layer of a CNN. Besides the proposed architecture, recent convolutional neural networks are evaluated on three distinct benchmarking datasets. This datasets are further divided in depth, binary and grayscale subsets in order to investigate the influence of image representation on recognition accuracy. Furthermore, architectures are compared in terms of speed and accuracy using rescaling with and without preserving aspect ratio and two common validation techniques: holdout and leave-one-subject-out. The proposed architecture is shown to obtain state-of-the art recognition rate in realtime, while being robust to different image representations and scalings. A recognition improvement of 5.93% on current best model is achieved on an RGBD dataset containing 81,000 images of 27 hand postures. A demo video is provided as supplementary material, containing real-time recognition by the proposed network of up to 27 gestures at 30 fps from a 3D camera.