Biblio

Facial is the most dynamic part of the human body that conveys information about emotions. The level of diversity in facial geometry and facial look makes it possible to detect various human expressions. To be able to differentiate among numerous facial expressions of emotion, it is crucial to identify the classes of facial expressions. The methodology used in this article is based on convolutional neural networks (CNN). In this paper Deep Learning CNN is used to examine Alex net architectures. Improvements were achieved by applying the transfer learning approach and modifying the fully connected layer with the Support Vector Machine(SVM) classifier. The system succeeded by achieving satisfactory results on icv-the MEFED dataset. Improved models achieved around 64.29 %of recognition rates for the classification of the selected expressions. The results obtained are acceptable and comparable to the relevant systems in the literature provide ideas a background for further improvements.

Facial expression intensity, which quantifies the degree of facial expression, has been proposed. It is calculated based on how much facial feature values change compared to an expressionless face. The estimation has two aspects. One is to classify facial expressions, and the other is to estimate their intensity. However, it is difficult to do them at the same time. There- fore, in this work, the estimation of intensity and the classification of expression are separated. We suggest an explicit method and an implicit method. In the explicit one, a classifier determines which types of expression the inputs are, and each regressor determines its intensity. On the other hand, in the implicit one, we give zero values or non-zero values to regressors for each type of facial expression as ground truth, depending on whether or not an input image is the correct facial expression. We evaluated the two methods and, as a result, found that they are effective for facial expression recognition.

Artificial intelligence (AI) cooperates with multiple industries to improve the overall industry framework. Especially, human emotion recognition plays an indispensable role in supporting medical care, psychological counseling, crime prevention and detection, and crime investigation. The research on emotion recognition includes emotion-specific intonation patterns, literal expressions of emotions, and facial expressions. Recently, the deep learning model of facial emotion recognition aims to capture tiny changes in facial muscles to provide greater recognition accuracy. Hybrid models in facial expression recognition have been constantly proposed to improve the performance of deep learning models in these years. In this study, we proposed an ensemble learning algorithm for the accuracy of the facial emotion recognition model with three deep learning models: VGG16, InceptionResNetV2, and EfficientNetB0. To enhance the performance of these benchmark models, we applied transfer learning, fine-tuning, and data augmentation to implement the training and validation of the Facial Expression Recognition 2013 (FER-2013) Dataset. The developed algorithm finds the best-predicted value by prioritizing the InceptionResNetV2. The experimental results show that the proposed ensemble learning algorithm of priorities edges up 2.81% accuracy of the model identification. The future extension of this study ventures into the Internet of Things (IoT), medical care, and crime detection and prevention.

The major mode of communication between hearing-impaired or mute people and others is sign language. Prior, most of the recognition systems for sign language had been set simply to recognize hand signs and convey them as text. However, the proposed model tries to provide speech to the mute. Firstly, hand gestures for sign language recognition and facial emotions are trained using CNN (Convolutional Neural Network) and then by training the emotion to speech model. Finally combining hand gestures and facial emotions to realize the emotion and speech.

The human face conveys a significant amount of information. Through facial expressions, the face is able to communicate numerous sentiments without the need for verbalisation. Visual emotion recognition has been extensively studied. Recently several end-to-end trained deep neural networks have been proposed for this task. However, such models often lack generalisation ability across datasets. In this paper, we propose the Deep Facial Expression Vector ExtractoR (DeepFEVER), a new deep learning-based approach that learns a visual feature extractor general enough to be applied to any other facial emotion recognition task or dataset. DeepFEVER outperforms state-of-the-art results on the AffectNet and Google Facial Expression Comparison datasets. DeepFEVER’s extracted features also generalise extremely well to other datasets – even those unseen during training – namely, the Real-World Affective Faces (RAF) dataset.

In view of the problem that the convolution neural network research of facial expression recognition ignores the internal relevance of the key links, which leads to the low accuracy and speed of facial expression recognition, and can't meet the recognition requirements, a series cascade algorithm model for expression recognition of educational robot is constructed and enables the educational robot to recognize multiple students' facial expressions simultaneously, quickly and accurately in the process of movement, in the balance of the accuracy, rapidity and stability of the algorithm, based on the cascade convolution neural network model. Through the CK+ and Oulu-CASIA expression recognition database, the expression recognition experiments of this algorithm are compared with the commonly used STM-ExpLet and FN2EN cascade network algorithms. The results show that the accuracy of the expression recognition method is more than 90%. Compared with the other two commonly used cascade convolution neural network methods, the accuracy of expression recognition is significantly improved.

With the development of artificial intelligence, facial expression recognition (FER) has greatly improved performance in deep learning, but there is still a lot of room for improvement in the study of combining attention to focus the network on key parts of the face. For facial expression recognition, this paper designs a network model, which use spatial transformer network to transform the input image firstly, and then adding channel attention and spatial attention to the convolutional network. In addition, in this paper, the GELU activation function is used in the convolutional network, which improves the recognition rate of facial expressions to a certain extent.

Automated analysis of facial expressions is one of the most interesting and challenging problems in many areas such as human-computer interaction. Facial images are affected by many factors, such as intensity, pose and facial expressions. These factors make facial expression recognition problem a challenge. The aim of this paper is to propose a new method based on the pyramid local binary pattern (PLBP) and the pyramid local phase quantization (PLPQ), which are the extension of the local binary pattern (LBP) and the local phase quantization (LPQ) as two methods for extracting texture features. LBP operator is used to extract LBP feature in the spatial domain and LPQ operator is used to extract LPQ feature in the frequency domain. The combination of features in spatial and frequency domains can provide important information in both domains. In this paper, PLBP and PLPQ operators are separately used to extract features. Then, these features are combined to create a new feature vector. The advantage of pyramid transform domain is that it can recognize facial expressions efficiently and with high accuracy even for very low-resolution facial images. The proposed method is verified on the CK+ facial expression database. The proposed method achieves the recognition rate of 99.85% on CK+ database.

Aiming at the problems of the traditional convolutional neural network (CNN), such as too many parameters, single scale feature and inefficiency by some useless features, a lightweight multi-scale network with attention is proposed for facial expression recognition. The network uses the lightweight convolutional neural network model Xception and combines with the convolutional block attention module (CBAM) to learn key facial features; In addition, depthwise separable convolution module with convolution kernel of 3 × 3, 5 × 5 and 7 × 7 are used to extract features of facial expression image, and the features are fused to expand the receptive field and obtain more rich facial feature information. Experiments on facial expression datasets Fer2013 and KDEF show that the expression recognition accuracy is improved by 2.14% and 2.18% than the original Xception model, and the results further verify the effectiveness of our methods.

Teaching evaluation is an indispensable key link in the modern education model. Its purpose is to promote learners' cognitive and non-cognitive development, especially emotional development. However, today's education has increasingly neglected the emotional process of learners' learning. Therefore, a method of using machines to analyze the emotional changes of learners during learning has been proposed. At present, most of the existing emotion recognition algorithms use the extraction of two-dimensional facial features from images to perform emotion prediction. Through research, it is found that the recognition rate of 2D facial feature extraction is not optimal, so this paper proposes an effective the algorithm obtains a single two-dimensional image from the input end and constructs a three-dimensional face model from the output end, thereby using 3D facial information to estimate the continuous emotion of the dimensional space and applying this method to an online learning system. Experimental results show that the algorithm has strong robustness and recognition ability.

Over the last few years, there has been an increasing number of studies about facial emotion recognition because of the importance and the impact that it has in the interaction of humans with computers. With the growing number of challenging datasets, the application of deep learning techniques have all become necessary. In this paper, we study the challenges of Emotion Recognition Datasets and we also try different parameters and architectures of the Conventional Neural Networks (CNNs) in order to detect the seven emotions in human faces, such as: anger, fear, disgust, contempt, happiness, sadness and surprise. We have chosen iCV MEFED (Multi-Emotion Facial Expression Dataset) as the main dataset for our study, which is relatively new, interesting and very challenging.

Emotions are a powerful tool in communication and one way that humans show their emotions is through their facial expressions. One of the challenging and powerful tasks in social communications is facial expression recognition, as in non-verbal communication, facial expressions are key. In the field of Artificial Intelligence, Facial Expression Recognition (FER) is an active research area, with several recent studies using Convolutional Neural Networks (CNNs). In this paper, we demonstrate the classification of FER based on static images, using CNNs, without requiring any pre-processing or feature extraction tasks. The paper also illustrates techniques to improve future accuracy in this area by using pre-processing, which includes face detection and illumination correction. Feature extraction is used to extract the most prominent parts of the face, including the jaw, mouth, eyes, nose, and eyebrows. Furthermore, we also discuss the literature review and present our CNN architecture, and the challenges of using max-pooling and dropout, which eventually aided in better performance. We obtained a test accuracy of 61.7% on FER2013 in a seven-classes classification task compared to 75.2% in state-of-the-art classification.

Motions of facial components convey significant information of facial expressions. Although remarkable advancement has been made, the dynamic of facial topology has not been fully exploited. In this paper, a novel facial expression recognition (FER) algorithm called Spatial Temporal Semantic Graph Network (STSGN) is proposed to automatically learn spatial and temporal patterns through end-to-end feature learning from facial topology structure. The proposed algorithm not only has greater discriminative power to capture the dynamic patterns of facial expression and stronger generalization capability to handle different variations but also higher interpretability. Experimental evaluation on two popular datasets, CK+ and Oulu-CASIA, shows that our algorithm achieves more competitive results than other state-of-the-art methods.

As a part of body language, facial expression is a psychological state that reflects the current emotional state of the person. Recognition of facial expressions can help to understand others and enhance communication with others. We propose a facial expression recognition method based on convolutional neural network ensemble learning in this paper. Our model is composed of three sub-networks, and uses the SVM classifier to Integrate the output of the three networks to get the final result. The recognition accuracy of the model's expression on the FER2013 dataset reached 71.27%. The results show that the method has high test accuracy and short prediction time, and can realize real-time, high-performance facial recognition.

Facial expressions are one of the most powerful, natural and immediate means for human being to present their emotions and intensions. In this paper, we present a novel method for fully automatic facial expression recognition. The facial landmarks are detected for characterizing facial expressions. A graph convolutional neural network is proposed for feature extraction and facial expression recognition classification. The experiments were performed on the three facial expression databases. The result shows that the proposed FER method can achieve good recognition accuracy up to 95.85% using the proposed method.

This paper explores the benefits of 3D face modeling for in-the-wild facial expression recognition (FER). Since there is limited in-the-wild 3D FER dataset, we first construct 3D facial data from available 2D dataset using recent advances in 3D face reconstruction. The 3D facial geometry representation is then extracted by deep learning technique. In addition, we also take advantage of manipulating the 3D face, such as using 2D projected images of 3D face as additional input for FER. These features are then fused with that of 2D FER typical network. By doing so, despite using common approaches, we achieve a competent recognition accuracy on Real-World Affective Faces (RAF) database and Static Facial Expressions in the Wild (SFEW 2.0) compared with the state-of-the-art reports. To the best of our knowledge, this is the first time such a deep learning combination of 3D and 2D facial modalities is presented in the context of in-the-wild FER.

Every so often Humans utilize non-verbal gestures (e.g. facial expressions) to express certain information or emotions. Moreover, countless face gestures are expressed throughout the day because of the capabilities possessed by humans. However, the channels of these expression/emotions can be through activities, postures, behaviors & facial expressions. Extensive research unveiled that there exists a strong relationship between the channels and emotions which has to be further investigated. An Automatic Facial Expression Recognition (AFER) framework has been proposed in this work that can predict or anticipate seven universal expressions. In order to evaluate the proposed approach, Frontal face Image Database also named as Japanese Female Facial Expression (JAFFE) is opted as input. This database is further processed with a frequency domain technique known as Discrete Cosine transform (DCT) and then classified using Artificial Neural Networks (ANN). So as to check the robustness of this novel strategy, the random trial of K-fold cross validation, leave one out and person independent methods is repeated many times to provide an overview of recognition rates. The experimental results demonstrate a promising performance of this application.

In this paper, a merged convolution neural network (MCNN) is proposed to improve the accuracy and robustness of real-time facial expression recognition (FER). Although there are many ways to improve the performance of facial expression recognition, a revamp of the training framework and image preprocessing renders better results in applications. When the camera is capturing images at high speed, however, changes in image characteristics may occur at certain moments due to the influence of light and other factors. Such changes can result in incorrect recognition of human facial expression. To solve this problem, we propose a statistical method for recognition results obtained from previous images, instead of using the current recognition output. Experimental results show that the proposed method can satisfactorily recognize seven basic facial expressions in real time.

Deep learning based facial expression recognition (FER) has received a lot of attention in the past few years. Most of the existing deep learning based FER methods do not consider domain knowledge well, which thereby fail to extract representative features. In this work, we propose a novel FER framework, named Facial Motion Prior Networks (FMPN). Particularly, we introduce an addition branch to generate a facial mask so as to focus on facial muscle moving regions. To guide the facial mask learning, we propose to incorporate prior domain knowledge by using the average differences between neutral faces and the corresponding expressive faces as the training guidance. Extensive experiments on three facial expression benchmark datasets demonstrate the effectiveness of the proposed method, compared with the state-of-the-art approaches.

In the past few years, there has been increasing interest in the perception of human expressions and mental states by machines, and Facial Expression Recognition (FER) has attracted increasing attention. Facial Action Unit (AU) is an early proposed method to describe facial muscle movements, which can effectively reflect the changes in people's facial expressions. In this paper, we propose a high-performance facial expression recognition method based on facial action unit, which can run on low-configuration computer and realize video and real-time camera FER. Our method is mainly divided into two parts. In the first part, 68 facial landmarks and image Histograms of Oriented Gradients (HOG) are obtained, and the feature values of action units are calculated accordingly. The second part uses three classification methods to realize the mapping from AUs to FER. We have conducted many experiments on the popular human FER benchmark datasets (CK+ and Oulu CASIA) to demonstrate the effectiveness of our method.

In this paper, we propose a method for improving the robustness of real-time facial expression recognition. Although there are many ways to improve the accuracy of facial expression recognition, a revamp of the training framework and image preprocessing allow better results in applications. One existing problem is that when the camera is capturing images in high speed, changes in image characteristics may occur at certain moments due to the influence of light and other factors. Such changes can result in incorrect recognition of the human facial expression. To solve this problem for smooth system operation and maintenance of recognition speed, we take changes in image characteristics at high speed capturing into account. The proposed method does not use the immediate output for reference, but refers to the previous image for averaging to facilitate recognition. In this way, we are able to reduce interference by the characteristics of the images. The experimental results show that after adopting this method, overall robustness and accuracy of facial expression recognition have been greatly improved compared to those obtained by only the convolution neural network (CNN).

Currently, fusing visible and thermal images for facial expression recognition requires two modalities during both training and testing. Visible cameras are commonly used in real-life applications, and thermal cameras are typically only available in lab situations due to their high price. Thermal imaging for facial expression recognition is not frequently used in real-world situations. To address this, we propose a novel thermally enhanced facial expression recognition method which uses thermal images as privileged information to construct better visible feature representation and improved classifiers by incorporating adversarial learning and similarity constraints during training. Specifically, we train two deep neural networks from visible images and thermal images. We impose adversarial loss to enforce statistical similarity between the learned representations of two modalities, and a similarity constraint to regulate the mapping functions from visible and thermal representation to expressions. Thus, thermal images are leveraged to simultaneously improve visible feature representation and classification during training. To mimic real-world scenarios, only visible images are available during testing. We further extend the proposed expression recognition method for partially unpaired data to explore thermal images' supplementary role in visible facial expression recognition when visible images and thermal images are not synchronously recorded. Experimental results on the MAHNOB Laughter database demonstrate that our proposed method can effectively regularize visible representation and expression classifiers with the help of thermal images, achieving state-of-the-art recognition performance.

This work proposes an approach to find and learn informative representations from 2 dimensional gray-level images for facial expression recognition application. The learned features are obtained from a designed convolutional neural network (CNN). The developed CNN enables us to learn features from the images in a highly efficient manner by cascading different layers together. The developed model is computationally efficient since it does not consist of a huge number of layers and at the same time it takes into consideration the overfitting problem. The outcomes from the developed CNN are compared to handcrafted features that span texture and shape features. The experiments conducted on the Bosphours database show that the developed CNN model outperforms the handcrafted features when coupled with a Support Vector Machines (SVM) classifier.

Facial emotion recognition is an essential aspect in human-machine interaction. In the real-world conditions, it faces many challenges, i.e., illumination changes, large pose variations and partial or full occlusions, which cause different facial areas with different sharpness and completeness. Inspired by this fact, we focus on facial expression recognition based on partial faces in this paper. We compare contribution of seven facial areas of low-resolution images, including nose areas, mouse areas, eyes areas, nose to mouse areas, nose to eyes areas, mouth to eyes areas and the whole face areas. Through analysis on the confusion matrix and the class activation map, we find that mouth regions contain much emotional information compared with nose areas and eyes areas. In the meantime, considering larger facial areas is helpful to judge the expression more precisely. To sum up, contributions of this paper are two-fold: (1) We reveal concerned areas of human in emotion recognition. (2) We quantify the contribution of different facial parts.

Facial expression is the most powerful and natural non-verbal emotional communication method. Facial Expression Recognition(FER) has significance in machine learning tasks. Deep Learning models perform well in FER tasks, but it doesn't provide any justification for its decisions. Based on the hypothesis that facial expression is a combination of facial muscle movements, we find that Facial Action Coding Units(AUs) and Emotion label have a relationship in CK+ Dataset. In this paper, we propose a model which utilises AUs to explain Convolutional Neural Network(CNN) model's classification results. The CNN model is trained with CK+ Dataset and classifies emotion based on extracted features. Explanation model classifies the multiple AUs with the extracted features and emotion classes from the CNN model. Our experiment shows that with only features and emotion classes obtained from the CNN model, Explanation model generates AUs very well.