Biblio

In this paper, we provide insights towards achieving more robust automatic facial expression recognition in smart environments based on our benchmark with three labeled facial expression databases. These databases are selected to test for desktop, 3D and smart environment application scenarios. This work is meant to provide a neutral comparison and guidelines for developers and researchers interested to integrate facial emotion recognition technologies in their applications, understand its limitations and adaptation as well as enhancement strategies. We also introduce and compare three different metrics for finding the primary expression in a time window of a displayed emotion. In addition, we outline facial emotion recognition limitations and enhancements for smart environments and non-frontal setups. By providing our comparison and enhancements we hope to build a bridge from affective computing research and solution providers to application developers that like to enhance new applications by including emotion based user modeling.

Facial expression recognition (FER) for emotion inference has become one of the most important research fields in human-computer interaction. Existing study on FER mainly focuses on visible images, whereas varying lighting conditions may influence their performances. Recent studies have demonstrated the advantages of infrared thermal images reflecting the temperature distributions, which are robust to lighting changes. In this paper, a novel infrared image sequence based FER method is proposed using spatiotemporal feature analysis and deep Boltzmann machines (DBM). Firstly, a dense motion field among infrared image sequences is generated using optical flow algorithm. Then, PCA is applied for dimension reduction and a three-layer DBM structure is designed for final expression classification. Finally, the effectiveness of the proposed method is well demonstrated based on several experiments conducted on NVIE database.

Facial expression recognition is a challenging problem in the field of computer vision. In this paper, we propose a deep learning approach that can learn the joint low-level and high-level features of human face to resolve this problem. Our deep neural networks utilize convolution and downsampling to extract the abstract and local features of human face, and reconstruct the raw input images to learn global features as supplementary information at the same time. We also add an adjustable weight in the networks when combining the two kinds of features for the final classification. The experimental results show that the proposed method can achieve good results, which has an average recognition accuracy of 93.65% on the test datasets.

A machine translation system that can convert South African Sign Language video to English audio or text and vice versa in real-time would be immensely beneficial to the Deaf and hard of hearing. Sign language gestures are characterised and expressed by five distinct parameters: hand location; hand orientation; hand shape; hand movement and facial expressions. The aim of this research is to recognise facial expressions and to compare the following feature descriptors: local binary patterns; compound local binary patterns and histogram of oriented gradients in two testing environments, a subset of the BU3D-FE dataset and the CK+ dataset. The overall accuracy, accuracy across facial expression classes, robustness to test subjects, and the ability to generalise of each feature descriptor within the context of automatic facial expression recognition are analysed as part of the comparison procedure. Overall, HOG proved to be a more robust feature descriptor to the LBP and CLBP. Furthermore, the CLBP can generally be considered to be superior to the LBP, but the LBP has greater potential in terms of its ability to generalise.

This paper proposes a novel deep two-view approach to learn features from both visible and thermal images and leverage the commonality among visible and thermal images for facial expression recognition from visible images. The thermal images are used as privileged information, which is required only during training to help visible images learn better features and classifier. Specifically, we first learn a deep model for visible images and thermal images respectively, and use the learned feature representations to train SVM classifiers for expression classification. We then jointly refine the deep models as well as the SVM classifiers for both thermal images and visible images by imposing the constraint that the outputs of the SVM classifiers from two views are similar. Therefore, the resulting representations and classifiers capture the inherent connections among visible facial image, infrared facial image and target expression labels, and hence improve the recognition performance for facial expression recognition from visible images during testing. Experimental results on the benchmark expression database demonstrate the effectiveness of our proposed method.