AI summary · arXiv: 2002.03557v1 · cs.CV
FAU, Facial Expressions, Valence and Arousal: A Multi-task Solution
The essentials
- The study focuses on training a unified model for three key tasks: predicting Facial Action Units (FAU, identifying seven basic facial expressions, and determining valence and arousal levels.
- Scarcity of fully-annotated datasets is a primary challenge in this endeavor.
- Authors propose an innovative algorithm for their multitask model to effectively learn from partial labels.
- Algorithm involves training a teacher model to execute all three tasks individually and utilizing its outputs as soft labels for training a student model.
- Student model outperforms the teacher model across all tasks due to exposure to complete set of labels during training.
- Ensemble modeling technique is implemented to further enhance performance on all three tasks.
- Research showcases novel methodology for addressing challenges in facial expression analysis through leveraging partial labels and ensemble modeling strategies.
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Multi-Task Learning for Emotion Descriptors Estimation at the fourth ABAW ChallengeYanan Chang, Yi Wu, Xiangyu Miao et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 6 times · Not summarized yet
Abstract
Facial valence/arousal, expression and action unit are related tasks in facial affective analysis. However, the tasks only have limited performance in the wild due to the various collected conditions. The 4th competition on affective behavior analysis in the wild (ABAW) provided images with valence/arousal, expression and action unit labels. In this paper, we introduce multi-task learning framework to enhance the performance of three related tasks in the wild. Feature sharing and label fusion are used to utilize their relations. We conduct experiments on the provided training and validating data.
-
A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior AnalysisLing-Feng Wang, Shi-Sen Wang, Jin Qi et al. · 2021 · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · cited 36 times · Not summarized yet
Abstract
Affective Behavior Analysis is an important part in human-computer interaction. Existing multi-task affective behavior recognition methods suffer from the problem of in-complete labeled datasets. To tackle this problem, this paper presents a semi-supervised model with a mean teacher framework to leverage additional unlabeled data. To be specific, a multi-task model is proposed to learn three different kinds of facial affective representations simultaneously. After that, the proposed model is assigned to be student and teacher networks. When training with unlabeled data, the teacher network is employed to predict pseudo labels for student network training, which allows it to learn from unlabe…
-
Multitask Multi-database Emotion RecognitionManh-Tu Vu, M. Beurton-Aimar, Serge Marchand · 2021 · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · cited 35 times · Not summarized yet
Abstract
This work has been initiated for the 2nd Affective Behavior Analysis in-the-wild (ABAW 2021) competition. We train a unified deep learning model on multi-databases to perform two tasks: seven basic facial expressions prediction and valence-arousal estimation. Since these databases do not contain labels for all the two tasks, we have applied the distillation knowledge technique to train two networks: one teacher and one student model. Both of these models are based on CNN-RNN hybrid architecture. The student model will be trained using both ground truth labels and soft labels derived from the pretrained teacher model. During the training, we have added one more task, which is the combination …
-
A Shared Latent for Partially-Labeled Multi-Task Facial Affect RecognitionH. Nguyen, Sy Phan Van, Soo-Hyung Kim et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet
Abstract
Facial affect in the wild is naturally multi-task: valence-arousal, discrete expressions, and facial action units describe the same face. Yet real corpora annotate these tasks only partially and unevenly, so most systems mask the missing labels or impute pseudo-labels and forgo the cross-task signal. We instead cast partially-labeled multi-task learning as marginalization over a shared affect latent: one variational bottleneck mediates all three task decoders, so a frame annotated for one task shapes the representation the others use, and the masked objective reappears as the reconstruction term of an evidence lower bound. On s-Aff-Wild2, where only 37% of frames carry all three labels, the …
-
Emotion Recognition with Incomplete Labels Using Modified Multi-task Learning TechniqueThinh Tran Dac Phan, Hung Manh Hoang, Hyung-Jeong Yang et al. · 2021 · arXiv.org · preprint, not peer reviewed · cited 19 times · Not summarized yet
Abstract
The task of predicting affective information in the wild such as seven basic emotions or action units from human faces has gradually become more interesting due to the accessibility and availability of massive annotated datasets. In this study, we propose a method that utilizes the association between seven basic emotions and twelve action units from the AffWild2 dataset. The method based on the architecture of ResNet50 involves the multi-task learning technique for the incomplete labels of the two tasks. By combining the knowledge for two correlated tasks, both performances are improved by a large margin compared to those with the model employing only one kind of label.
-
An Ensemble Approach for Multiple Emotion Descriptors Estimation Using Multi-task LearningIrfan Haider, Minh-Trieu Tran, Soo-Hyung Kim et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 6 times · Not summarized yet
Abstract
This paper illustrates our submission method to the fourth Affective Behavior Analysis in-the-Wild (ABAW) Competition. The method is used for the Multi-Task Learning Challenge. Instead of using only face information, we employ full information from a provided dataset containing face and the context around the face. We utilized the InceptionNet V3 model to extract deep features then we applied the attention mechanism to refine the features. After that, we put those features into the transformer block and multi-layer perceptron networks to get the final multiple kinds of emotion. Our model predicts arousal and valence, classifies the emotional expression and estimates the action units simultan…
-
Facial Affect Analysis: Learning from Synthetic Data & Multi-Task Learning ChallengesSiyang Li, Yifan Xu, Huan-Yu Wu et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 16 times · Not summarized yet
Abstract
Facial affect analysis remains a challenging task with its setting transitioned from lab-controlled to in-the-wild situations. In this paper, we present novel frameworks to handle the two challenges in the 4th Affective Behavior Analysis In-The-Wild (ABAW) competition: i) Multi-Task-Learning (MTL) Challenge and ii) Learning from Synthetic Data (LSD) Challenge. For MTL challenge, we adopt the SMM-EmotionNet with a better ensemble strategy of feature vectors. For LSD challenge, we propose respective methods to combat the problems of single labels, imbalanced distribution, fine-tuning limitations, and choice of model architectures. Experimental results on the official validation sets from the c…
-
Facial Action Units as a Joint Dataset Training Bridge for Facial Expression RecognitionShuyi Mao, Xin-Peng Li, Fan Zhang et al. · 2025 · IEEE transactions on multimedia · cited 16 times · Not summarized yet
Abstract
Label biases in facial expression recognition (FER) datasets, caused by annotators' subjectivity, pose challenges in improving the performance of target datasets when auxiliary labeled data are used. Moreover, training with multiple datasets can lead to visible degradations in the target dataset. To address these issues, we propose a novel framework called the AU-aware Vision Transformer (AU-ViT), which leverages unified action unit (AU) information and discards expression annotations of auxiliary data. AU-ViT integrates an elaborately designed AU branch in the middle part of a master ViT to enhance representation learning during training. Through qualitative and quantitative analyses, we de…
Prior work it builds on
-
Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFaceD. Kollias, Stefanos P. Zafeiriou · 2019 · British Machine Vision Conference · cited 423 times · Not summarized yet
Abstract
Affective computing has been largely limited in terms of available data resources. The need to collect and annotate diverse in-the-wild datasets has become apparent with the rise of deep learning models, as the default approach to address any computer vision task. Some in-the-wild databases have been recently proposed. However: i) their size is small, ii) they are not audiovisual, iii) only a small part is manually annotated, iv) they contain a small number of subjects, or v) they are not annotated for all main behavior tasks (valence-arousal estimation, action unit detection and basic expression classification). To address these, we substantially extend the largest available in-the-wild dat…
-
Face Behavior à la carte: Expressions, Affect and Action Units in a Single NetworkD. Kollias, V. Sharmanska, Stefanos P. Zafeiriou · 2019 · arXiv.org · preprint, not peer reviewed · cited 259 times · Not summarized yet
Abstract
Automatic facial behavior analysis has a long history of studies in the intersection of computer vision, physiology and psychology. However it is only recently, with the collection of large-scale datasets and powerful machine learning methods such as deep neural networks, that automatic facial behavior analysis started to thrive. Three of its iconic tasks are automatic recognition of basic expressions (e.g. happy, sad, surprised), estimation of continuous emotions (e.g., valence and arousal), and detection of facial action units (activations of e.g. upper/inner eyebrows, nose wrinkles). Up until now these tasks have been mostly studied independently collecting a dataset for the task. We pres…
-
Analysing Affective Behavior in the First ABAW 2020 CompetitionD. Kollias, Attila Schulc, Elnar Hajiyev et al. · 2020 · IEEE International Conference on Automatic Face & Gesture Recognition · cited 335 times · Not summarized yet
Abstract
The Affective Behavior Analysis in-the-wild (ABAW) 2020 Competition is the first Competition aiming at automatic analysis of the three main behavior tasks of valencearousal estimation, basic expression recognition and action unit detection. It is split into three Challenges, each one addressing a respective behavior task. For the Challenges, we provide a common benchmark database, Aff-Wild2, which is a large scale in-the-wild database and the first one annotated for all these three tasks. In this paper, we describe this Competition, to be held in conjunction with the IEEE Conference on Face and Gesture Recognition, May 2020, in Buenos Aires, Argentina. We present the three Challenges, with t…
-
From Facial Expression Recognition to Interpersonal Relation PredictionZhanpeng Zhang, Ping Luo, Chen Change Loy et al. · 2016 · International Journal of Computer Vision · cited 318 times · Not summarized yet
Abstract
Interpersonal relation defines the association, e.g., warm, friendliness, and dominance, between two or more people. We investigate if such fine-grained and high-level relation traits can be characterized and quantified from face images in the wild. We address this challenging problem by first studying a deep network architecture for robust recognition of facial expressions. Unlike existing models that typically learn from facial expression labels alone, we devise an effective multitask network that is capable of learning from rich auxiliary attributes such as gender, age, and head pose, beyond just facial expression data. While conventional supervised training requires datasets with complet…
-
Aff-Wild2: Extending the Aff-Wild Database for Affect RecognitionDimitrios Kollias, Stefanos Zafeiriou · 2018 · Computer Vision and Pattern Recognition · preprint, not peer reviewed · cited 221 times · AI summary available
Abstract
Automatic understanding of human affect using visual signals is a problem that has attracted significant interest over the past 20 years. However, human emotional states are quite complex. To appraise such states displayed in real-world settings, we need expressive emotional descriptors that are capable of capturing and describing this complexity. The circumplex model of affect, which is described in terms of valence (i.e., how positive or negative is an emotion) and arousal (i.e., power of the activation of the emotion), can be used for this purpose. Recent progress in the emotion recognition domain has been achieved through the development of deep neural architectures and the availability …
Papers that cite it
-
An Ensemble Approach for Facial Expression Analysis in VideoH. Nguyen, Văn-Thống Huỳnh, Soo-Hyung Kim · 2022 · arXiv.org · preprint, not peer reviewed · cited 24 times · Not summarized yet
Abstract
Human emotions recognization contributes to the development of human-computer interaction. The machines understanding human emotions in the real world will significantly contribute to life in the future. This paper will introduce the Affective Behavior Analysis in-the-wild (ABAW3) 2022 challenge. The paper focuses on solving the problem of the valence-arousal estimation and action unit detection. For valence-arousal estimation, we conducted two stages: creating new features from multimodel and temporal learning to predict valence-arousal. First, we make new features; the Gated Recurrent Unit (GRU) and Transformer are combined using a Regular Networks (RegNet) feature, which is extracted from…
-
MTMSN: Multi-Task and Multi-Modal Sequence Network for Facial Action Unit and Expression RecognitionYue Jin, Tianqing Zheng, Chao Gao et al. · 2021 · 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · cited 10 times · Not summarized yet
Abstract
Facial action unit (AU) and basic expression recognition are two basic tasks in the area of human affective behavior analysis. Most of the existing methods are developed in restricted scenarios which are not practical for in-the-wild settings. The Affective Behavior Analysis in-the-wild (ABAW) 2021 Contest provides a benchmark for this in-the-wild problem.In this paper, we propose a multi-task and multi-modal sequence network (MTMSN) to mine the relationships between the above two different tasks and effectively utilize both visual and audio information of the video. We use both AU and expression annotations to train the model and apply a sequence model to further extract associations betwee…
-
AU-Aware Vision Transformers for Biased Facial Expression RecognitionShuyi Mao, Xin-Peng Li, Q. Wu et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 5 times · Not summarized yet
Abstract
Studies have proven that domain bias and label bias exist in different Facial Expression Recognition (FER) datasets, making it hard to improve the performance of a specific dataset by adding other datasets. For the FER bias issue, recent researches mainly focus on the cross-domain issue with advanced domain adaption algorithms. This paper addresses another problem: how to boost FER performance by leveraging cross-domain datasets. Unlike the coarse and biased expression label, the facial Action Unit (AU) is fine-grained and objective suggested by psychological studies. Motivated by this, we resort to the AU information of different FER datasets for performance boosting and make contributions …
-
A Multi-modal and Multi-task Learning Method for Action Unit and Expression RecognitionYue Jin, Tianqing Zheng, Chao Gao et al. · 2021 · arXiv.org · preprint, not peer reviewed · cited 44 times · Not summarized yet
Abstract
Analyzing human affect is vital for human-computer interaction systems. Most methods are developed in restricted scenarios which are not practical for in-the-wild settings. The Affective Behavior Analysis in-the-wild (ABAW) 2021 Contest provides a benchmark for this in-the-wild problem. In this paper, we introduce a multi-modal and multi-task learning method by using both visual and audio information. We use both AU and expression annotations to train the model and apply a sequence model to further extract associations between video frames. We achieve an AU score of 0.712 and an expression score of 0.477 on the validation set. These results demonstrate the effectiveness of our approach in im…
-
ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning ChallengesD. Kollias · 2022 · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · cited 267 times · Not summarized yet
Abstract
This paper describes the third Affective Behavior Analysis in-the-wild (ABAW) Competition, held in conjunction with IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2022. The 3rd ABAW Competition is a continuation of the Competitions held at ICCV 2021, IEEE FG 2020 and IEEE CVPR 2017 Conferences, and aims at automatically analyzing affect. This year the Competition encompasses four Challenges: i) uni-task Valence-Arousal Estimation, ii) uni-task Expression Classification, iii) uni-task Action Unit Detection, and iv) MultiTask-Learning. All the Challenges are based on a common benchmark database, Aff-Wild2, which is a large scale in-the-wild database and the fi…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 2 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.