Our survey 'Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective' has released link: https://arxiv.org/abs/2409.07388

Multimodal Affective Computing Papers

We summarize the related work for the following tasks:

• Multimodal Sentiment Analysis
• Multimodal Emotion Recognition in Conversation
• Multimodal Aspect-based Sentiment Analysis
• Multimodal Multi-label Emotion Recognition
• Multimodal Emotion-cause Pair Extraction in Conversation

We will release a review of multimodal affective computing soon

---

Multimodal Sentiment Analysis (MSA)

Research

• [1] Huisheng Mao, Baozheng Zhang, Hua Xu, Ziqi Yuan, Yihe Liu. [Robust-MSA: Understanding the Impact of Modality Noise on Multimodal Sentiment Analysis]

• [2] S Afzal, HA Khan, IU Khan, MJ Piran…. [A Comprehensive Survey on Affective Computing Challenges, Trends, Applications, and Future Directions_]

• [3] J Peng, T Wu, W Zhang, F Cheng, S Tan, F Yi…. [A fine-grained modal label-based multi-stage network for multimodal sentiment analysis]

• [4] Y Zhi, J Li, H Wang, J Chen…. [A Fine-Grained Tri-Modal Interaction Model for Multimodal Sentiment Analysis]

• [5] J LIU, H SONG, DP CHEN, B WANG, ZW ZHANG. [A Multimodal Sentiment Analysis Model Enhanced with Non-verbal Information and Contrastive Learning]

• [6] X Sun, X Ren, X Xie. [A NOVEL MULTIMODAL SENTIMENT ANALYSIS MODEL BASED ON GATED FUSION AND MULTI-TASK LEARNING]

• [7] MM Amin, R Mao, E Cambria, BW Schuller. [A Wide Evaluation of ChatGPT on Affective Computing Tasks]

• [8] AAB Zadeh, PP Liang, S Poria, E Cambria…. [CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph]

• [9] W Yu, H Xu, F Meng, Y Zhu, Y Ma, J Wu…. [A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotations of Modality]

• [10] K Kim, S Park. [All-modalities-in-One BERT for multimodal sentiment analysis]

• [11] Hui Ma, Jian Wang, Hongfei Lin, Bo Zhang, Yijia Zhang, Bo Xu. [A_Transformer-Based_Model_With_Self-Distillation_for_Multimodal_Emotion_Recognition_in_Conversations]

• [12] J Tang, D Liu, X Jin, Y Peng, Q Zhao…. [Bi-Direction Attention Based Fusion Network for Multimodal Sentiment Analysis]

• [13] Z Li, B Xu, C Zhu, T Zhao. [CLMLF-A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detection]

• [14] J Yang, Y Yu, D Niu, W Guo, Y Xu. [Contrastive Feature Decomposition for Multimodal Sentiment Analysis]

• [15] Y Yu, M Zhao, S Qi, F Sun, B Wang, W Guo…. [Contrastive Knowledge Injection for Multimodal Sentiment Analysis]

• [16] DS Chauhan, MS Akhtar, A Ekbal…. [Context-aware Interactive Attention for Multi-modal Sentiment and Emotion Analysis]

• [17] M Huang, C Qing, J Tan, X Xu. [Context-Based Adaptive Multimodal Fusion Network for Continuous Frame-Level Sentiment Prediction]

• [18] Y Zeng, W Yan, S Mai, H Hu. [Disentanglement Translation Network for multimodal sentiment analysis]

• [19] S Mai, H Hu, S Xing. [Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing]

• [20] Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih. [Multimodal Phased Transformer for Sentiment Analysis]

• [21] A Martins, R Astudillo. [A Sparse Model of Attention and Multi-Label Classification]

• [22] H Liang, W Xie, X He, S Song…. [GUIDED CIRCULAR DECOMPOSITION AND CROSS-MODAL RECOMBINATION FOR MULTIMODAL SENTIMENT ANALYSIS]

• [23] S Mai, Y Zeng, S Zheng, H Hu. [Hybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis]

• [24] X Zhao, Y Chen, W Li, L Gao…. [An Extended Multimodal Adaptation Gate for Multimodal Sentiment Analysis]

• [25] D Hazarika, S Poria, R Mihalcea…. [Interactive Conversational Memory Network for Multimodal Emotion Detection]

• [26] C Busso, M Bulut, CC Lee, A Kazemzadeh…. [Interactive emotional dyadic motion capture database]

• [27] W Han, H Chen, S Poria. [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis]

• [28] W Rahman, MK Hasan, S Lee, A Zadeh…. [Integrating Multimodal Information in Large Pretrained Transformers]

• [29] C Chen, H Hong, J Guo, B Song. [Inter-Intra Modal Representation Augmentation With Trimodal Collaborative Disentanglement Network for Multimodal Sentiment Analysis]

• [30] VW Liang, Y Zhang, Y Kwon…. [Understanding the Modality Gap in Multi-modal Contrastive Representation Learning]

• [31] D Hazarika, R Zimmermann, S Poria. [Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis]

• [32] J Zeng, J Zhou, T Liu. [Mitigating Inconsistencies in Multimodal Sentiment Analysis under Uncertain Missing Modalities]

• [33] K Yang, H Xu, K Gao. [Cross-Modal BERT for Text-Audio Sentiment Analysis]

• [34] S Mai, H Hu, S Xing. [Modality to Modality Translation An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion]

• [35] Z Liu, B Zhou, D Chu, Y Sun, L Meng. [Modality translation-based multimodal sentiment analysis under uncertain missing modalities]

• [36] Z Lin, B Liang, Y Long, Y Dang…. [Hierarchical Graph Contrastive Learning for Multimodal Sentiment Analysis]

• [37] A Zadeh, R Zellers, E Pincus, LP Morency. [Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos]

• [38] J Yang, Y Wang, R Yi, Y Zhu, A Rehman…. [Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences]

• [39] L Xiao, X Wu, W Wu, J Yang…. [MULTI-CHANNEL ATTENTIVE GRAPH CONVOLUTIONAL NETWORK WITH SENTIMENT FUSION FOR MULTIMODAL SENTIMENT ANALYSIS]

• [40] J Zheng, S Zhang, Z Wang, X Wang…. [Multi-Channel Weight-Sharing Autoencoder Based on Cascade Multi-Head Attention for Multimodal Emotion Recognition]

• [41] L Fang, G Liu, R Zhang. [MULTI-GRAINED MULTIMODAL INTERACTION NETWORK FOR SENTIMENT ANALYSIS]

• [42] S Poria, E Cambria, D Hazarika…. [Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis]

• [43] MS Akhtar, DS Chauhan, D Ghosal, S Poria…. [Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis]

• [44] R Lin, H Hu. [Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis]

• [45] PP Liang, Z Liu, A Zadeh, LP Morency. [Multimodal Language Analysis with Recurrent Multistage Fusion]

• [46] Z Wu, Z Gong, J Koo, J Hirschberg. [Multimodal Multi-loss Fusion Network for Sentiment Analysis]

• [47] J Cheng, I Fostiropoulos, B Boehm…. [Multimodal Phased Transformer for Sentiment Analysis]

• [48] J Zheng, S Zhang, X Wang, Z Zeng. [Multimodal Representations Learning Based on Mutual Information Maximization and Minimization and Identity Embedding for Multimodal Sentiment Analysis]

• [49] Z Li, Q Guo, C Feng, L Deng, Q Zhang…. [Multimodal Sentiment Analysis Based on Interactive Transformer and Soft Mapping]

• [50] M Chen, S Wang, PP Liang, T Baltrušaitis…. [Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning]

• [51] R Kaur, S Kautish. [Multimodal Sentiment Analysis: A Survey and Comparison]

• [52] A Gandhi, K Adhvaryu, S Poria, E Cambria, A Hussain. [A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions]

• [53] X Yang, S Feng, Y Zhang, D Wang. [Multimodal Sentiment Detection Based on Multi-channel Graph Neural Networks]

• [54] Huan Deng, Zhenguo Yang, Tianyong Hao, Qing Li, Wenyin Liu. [_Multimodal_Affective_Computing_With_Dense_Fusion_Transformer_for_Inter-and_Intra-Modality_Interactions]

• [55] MS Akhtar, DS Chauhan, D Ghosal, S Poria…. [NAACL19_Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis(1)]

• [56] F Wang, S Tian, L Yu, J Liu, J Wang, K Li…. [TEDT: Transformer-Based Encoding–Decoding Translation Network for Multimodal Sentiment Analysis]

• [57] H Luo, L Ji, Y Huang, B Wang, S Ji, T Li. [Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors]

• [58] H Yang, X Gao, J Wu, T Gan, N Ding…. [Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition]

• [59] F Qian, J Han, Y He, T Zheng…. [Sentiment Knowledge Enhanced Self-supervised Learning for Multimodal Sentiment Analysis]

• [60] Y Wu, Y Zhao, H Yang, S Chen, B Qin, X Cao…. [Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors]

• [61] S Lai, J Li, G Guo, X Hu, Y Li, Y Tan, Z Song…. [Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learning]

• [62] C Zhu, M Chen, S Zhang, C Sun, H Liang, Y Liu…. [Sentiment Knowledge Enhanced Attention Fusion Network for multimodal sentiment analysis]

• [63] M Chen, X Li. [Sentimental Words Aware Fusion Network for Multimodal Sentiment Analysis]

• [64] Y Wei, S Yuan, R Yang, L Shen, Z Li…. [Tackling Modality Heterogeneity with Multi-View Calibration Network for Multimodal Sentiment Detection]

• [65] J Zeng, T Liu, J Zhou. [Tag-assisted Multimodal Sentiment Analysis under Uncertain]

• [66] J Yu, J Wang, R Xia, J Li. [Targeted Multimodal Sentiment Classifcation Based on Coarse-to-Fine Grained Image-Target Matching]

• [67] M Zhou, W Quan, Z Zhou, K Wang, T Wang…. [Text-oriented Cross Attention Network for Multimodal Sentiment Analysis]

• [68] C Huang, J Zhang, X Wu, Y Wang, M Li…. [Text-centered fusion network with crossmodal attention for multimodal sentiment analysis]

• [69] A Zadeh, M Chen, S Poria, E Cambria…. [Tensor Fusion Network for Multimodal Sentiment Analysis]

• [70] D Wang, X Guo, Y Tian, J Liu, LH He, X Luo. [A text enhanced transformer fusion network for multimodal sentiment analysis]

• [71] L Stappen, A Baird, L Schumann…. [The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements]

• [72] X Zhao, Y Chen, S Liu, X Zang, Y Xiang…. [A New Token Mixup Multimodal Data Augmentation for Multimodal Sentiment Analysis]

• [73] Y Li, W Weng, C Liu. [two-stage contrastive learning and feature hierarchical fusion network for multimodal sentiment analysis]

• [74] Y Zeng, S Mai, H Hu. [Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis]

• [75] W Yu, H Xu, Z Yuan, J Wu. [_AAAI21_Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis]

• [76] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, Ruslan Salakhutdinov. [_Multimodal Transformer for Unaligned Multimodal Language Sequences]

• [77] J Tang, K Li, X Jin, A Cichocki, Q Zhao…. [Hierarchical Learning for Multimodal Sentiment Analysis Using Coupled-Translation Fusion Network]

• [78] A Zadeh, YS Cao, S Hessner, PP Liang…. [A Multimodal Language Dataset for Spanish, Portuguese, German and French]

• [79] W Han, H Chen, S Poria. [_EMNLP21_Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis]

• [80] Z Yuan, W Li, H Xu, W Yu. [_MM21_Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment Analysis]

Multimodal Emotion Recognition in Conversation (MERC)

Research

• [1] S Harata, T Sakuma, S Kato. [Audio-Visual Shared Emotion Representation for Robust Emotion Recognition on Modality Missing Using Hemi-hyperspherical Embedding and Latent Space Unification]

• [2] FA Rahman, G Lu. [A Contextualized Real-Time Multimodal Emotion Recognition for Conversational Agents using Graph Convolutional Networks in Reinforcement Learning]

• [3] W Zheng, J Yu, R Xia, S Wang. [A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations]

• [4] Y Zhang, J Wang, Y Liu, L Rong, Q Zheng, D Song…. [A Multitask learning model for multimodal sarcasm, sentiment and emotion recognition in conversations]

• [5] M Sharafi, M Yazdchi, R Rasti, F Nasimi. [A novel spatio-temporal convolutional neural framework for multimodal emotion recognition]

• [6] B Pan, K Hirota, Z Jia, Y Dai. [A review of multimodal emotion recognition from datasets, preprocessing, features, and fusion methods]

• [7] DS Chauhan, GV Singh, A Arora, A Ekbal…. [A Sentiment and Emotion aware Multimodal Multiparty Humor Recognition in Multilingual Conversational Setting]

• [8] X Li, J Song, Z Zhao, C Wang…. [A SUPERVISED INFORMATION ENHANCED MULTI-GRANULARITY CONTRASTIVE LEARNING FRAMEWORK FOR EEG BASED EMOTION RECOGNITION]

• [9] H Ma, J Wang, H Lin, B Zhang…. [A Transformer-based Model with Self-distillation for Multimodal Emotion Recognition in Conversations]

• [10] K Ali, CE Hughes. [A Unified Transformer-based Network for Multimodal Emotion Recognition]

• [11] J Vazquez-Rodriguez, G Lefebvre…. [ACCOMMODATING MISSING MODALITIES IN TIME-CONTINUOUS MULTIMODAL EMOTION RECOGNITION]

• [12] S Poria, D Hazarika, N Majumder, G Naik…. [A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations]

• [13] JA Miranda-Correa, MK Abadi…. [A Dataset for Affect, Personality and Mood Research on Individuals and Groups]

• [14] NK Devulapally, S Anand, SD Bhattacharjee…. [Adaptive Multimodal Analysis for Speaker Emotion Recognition in Group Conversations]

• [15] Fei Ma, Shao-Lun Huang, Lin Zhang. [An_Efficient_Approach_for_Audio-Visual_Emotion_Recognition_With_Missing_Labels_And_Missing_Modalities]

• [16] Hui Ma, Jian Wang, Hongfei Lin, Bo Zhang, Yijia Zhang, Bo Xu. [A_Transformer-Based_Model_With_Self-Distillation_for_Multimodal_Emotion_Recognition_in_Conversations]

• [17] A Jia, Y He, Y Zhang, S Uprety, D Song…. [A Multi-Modal Dataset for Human Desire Understanding]

• [18] X Zhang, Y Li. [A Cross-Modality Context Fusion and Semantic Refinement Network for Emotion Recognition in Conversation]

• [19] A Joshi, A Bhat, A Jain, A Singh…. [COntextualized GNN based Multimodal Emotion recognitioN]

• [20] Yahui Fu, Shogo Okada, Longbiao Wang, Lili Guo, Yaodong Song. [_Context-and_Knowledge-Aware_Graph_Convolutional_Network_for_Multimodal_Emotion_Recognition]

• [21] WY Choi, KY Song, CW Lee. [Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data]

• [22] Y Wang, Y Li, P Bell, C Lai - researchgate.net. [Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition]

• [23] SAM Zaidi, S Latif, J Qadi. [Cross-Language Speech Emotion Recognition Using Multimodal Dual Attention Transformers]

• [24] Y Zhang, H Liu, D Wang, D Zhang, T Lou…. [Cross-modal credibility modelling for EEG-based multimodal emotion recognition]

• [25] L Xiao, X Wu, S Yang, J Xu, J Zhou, L He. [Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis]

• [26] Y Li, Y Wang, Z Cui. [Decoupled Multimodal Distilling for Emotion Recognition]

• [27] F Zhang, XC Li, CP Lim, Q Hua, CR Dong, JH Zhai. [Deep Emotional Arousal Network for Multimodal Sentiment Analysis and Emotion Recognition]

• [28] AI Middya, B Nag, S Roy. [Deep learning based multimodal emotion recognition using model-level fusion of audio–visual modalities]

• [29] G Arora, M Sabharwal, P Kapila…. [Deep Residual Adaptive Neural Network Based Feature Extraction for Cognitive Computing with Multimodal Sentiment Sensing and Emotion Recognition Process]

• [30] Y Mao, Q Sun, G Liu, X Wang, W Gao, X Li…. [Exploring Multi-Modal Emotion Dynamics in Conversations]

• [31] Y Guo, C Tang, H Wu, B Chen. [EEG EMOTION RECOGNITION BASED ON DYNAMICAL GRAPH ATTENTION NETWORK]

• [32] GV Singh, M Firdaus, A Ekbal…. [A Multimodal Transformer for Identifying Emotions and Intents in Social Conversations]

• [33] K Ezzameli, H Mahersia. [Emotion recognition from unimodal to multimodal analysis: A review]

• [34] J Vazquez-Rodriguez, G Lefebvre…. [Emotion Recognition with Pre-Trained Transformers Using Multimodal Signals]

• [35] D Andreoletti, F Cardoso, A Arzillo, L Luceri… - 2021 - ceur-ws.org. [Emotion_Recognition_and_Detection_Methods_A_Compre]

• [36] Lili Guo, Longbiao Wang, Jianwu Dang, Yahui Fu, Jiaxing Liu. [Emotion_Recognition_With_Multimodal_Transformer_Fusion_Framework_Based_on_Acoustic_and_Lexical_Information]

• [37] MM Islam, S Nooruddin, F Karray…. [A deep learning based model-level fusion approach]

• [38] H Zuo, R Liu, J Zhao, G Gao, H Li. [EXPLOITING MODALITY-INVARIANT FEATURE FOR ROBUST MULTIMODAL EMOTION RECOGNITION WITH MISSING MODALITIES]

• [39] Haolin Zuo, Rui Liu, Jinming Zhao, Guanglai Gao, Haizhou Li. [Exploiting_Modality-Invariant_Feature_for_Robust_Multimodal_Emotion_Recognition_with_Missing_Modalities]

• [40] Z Fang, A He, Q Yu, B Gao, W Ding, T Zhang…. [A novel multimodal emotion recognition approach integrating face, body and text]

• [41] X Yang, S Feng, D Wang, S Qi, W Wu, Y Zhang…. [Few-shot Joint Multimodal Aspect-Sentiment Analysis Based on Generative Multimodal Prompt]

• [42] YP Ruan, S Han, T Li, Y Wu. [FUSING MODALITY-SPECIFIC REPRESENTATIONS AND DECISIONS FOR MULTIMODAL EMOTION RECOGNITION]

• [43] M Jin, J Li. [Learning Deep Representations for Multimodal Emotion Recognition]

• [44] J Li, X Wang, G Lv, Z Zeng. [A graph network based multimodal fusion technique for emotion recognition in conversation]

• [45] M Sun, X Zhang, J Ma, S Xie, Y Liu…. [A Knowledge-guided Dual-inconsistency Network for Multi-modal Rumor Detection]

• [46] Haiyang Xu, Hui Zhang, Kun Han, Yun Wang, Yiping Peng, Xiangang Li. [[Interspeech19_Learning Alignment for Multimodal Emotion Recognition from Speech]](Interspeech19_Learning Alignment for Multimodal Emotion Recognition from Speech)

• [47] C Ziems, W Held, O Shaikh, J Chen, Z Zhang…. [Is GPT a Computational Model of Emotion_]

• [48] D Li, Y Wang, K Funakoshi, M Okumura. [Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognition]

• [49] J Sun, S Han, YP Ruan, X Zhang…. [Layer-wise Fusion with Modality Independence Modeling for Multi-modal Emotion Recognition]

• [50] Z Tang, Q Xiao, X Zhou, Y Li, C Chen, K Li. [Learning discriminative multi-relation representations for multimodal sentiment analysis]

• [51] P Wang, S Zeng, J Chen, L Fan, M Chen, Y Wu…. [Leveraging Label Information for Multimodal Emotion Recognition]

• [52] V Chudasama, P Kar, A Gudmalwar…. [Multi-modal Fusion Network for Emotion Recognition in Conversation]

• [53] N Wang, H Cao, J Zhao, R Chen…. [Missing-Modality Robust Emotion Recognition Framework With Iterative Data Augmentation]

• [54] Y Zhang, A Jia, B Wang, P Zhang, D Zhao, P Li…. [A Multi-modal, Multi-task Interactive Graph Attention Network for Conversational Sentiment Analysis and Emotion Recognition]

• [55] M Ren, X Huang, J Liu, M Liu, X Li…. [Multimodal Adversarial Learning Network for Conversational Emotion Recognition]

• [56] M Firdaus, H Chauhan, A Ekbal…. [A Multimodal Multi-Label Emotion, Intensity and Sentiment Dialogue Dataset for Emotion Recognition and Sentiment Analysis in Conversations]

• [57] Z Lian, H Sun, L Sun, Z Wen, S Zhang, S Chen…. [Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition]

• [58] J Zhao, R Li, Q Jin. [Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities]

• [59] X Chen. [RULE-BASED NETWORK FOR MULTIMODAL EMOTION RECOGNITION]

• [60] J Zheng, S Zhang, Z Wang, X Wang…. [Multi-Channel Weight-Sharing Autoencoder Based on Cascade Multi-Head Attention for Multimodal Emotion Recognition]

• [61] T Shi, SL Huang. [An Attention-Based Correlation-Aware Multimodal Fusion Framework for Emotion Recognition in Conversations]

• [62] Q Li, Y Gao, C Wang, Y Deng, J Xue…. [Multilevel_Transformer_for_Multimodal_Emotion_Recognition]

• [63] G Aguilar, V Rozgić, W Wang, C Wang. [Multimodal and Multi-view Models for Emotion Recognition]

• [64] J Cheng, I Fostiropoulos, B Boehm…. [Multimodal Phased Transformer for Sentiment Analysis]

• [65] S Zou, X Huang, X Shen. [Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition in Conversation]

• [66] X Sun, H He, H Tang, K Zeng…. [Multimodal rough set transformer for sentiment analysis and emotion recognition]

• [67] L Sun, B Liu, J Tao, Z Lian. [Multimodal Speech Emotion Recognition using Cross Attnention with Aligned Audio and Text]

• [68] F Chen, J Shao, S Zhu…. [Rethinking Graph Neural Networks for Emotion Recognition in Conversation]

• [69] A Joshi, A Bhat, A Jain, A Singh…. [COntextualized GNN based Multimodal Emotion recognitioN]

• [70] Z Li, Y Zhou, Y Liu, F Zhu, C Yang…. [A Quantum-Inspired Adaptive-Priority-Learning Model for Multimodal Emotion Recognition]

• [71] C Zhang, Y Zhang, B Cheng. [A REINFORCEMENT LEARNING FRAMEWORK FOR MULTIMODAL EMOTION RECOGNITION]

• [72] H Yang, X Gao, J Wu, T Gan, N Ding…. [Self-adaptive Context and Modal-interaction Modeling For Multimodal Emotion Recognition]

• [73] H Chen, C Guo, Y Li, P Zhang, D Jiang. [Semi-Supervised Multimodal Emotion Recognition with Class-Balanced Pseudo-Labeling]

• [74] U Chinta, J Kalita, A Atyabi. [Facial Images and EEG]

• [75] B Yao, W Shi. [SPEAKER-CENTRIC MULTIMODAL FUSION NETWORKS FOR EMOTION RECOGNITION IN CONVERSATIONS]

• [76] D Zhang, F Chen, J Chang, X Chen…. [Structure Aware Multi-Graph Network for Multi-Modal Emotion Recognition in Conversations]

• [77] X Song, L Huang, H Xue, S Hu. [Supervised Prototypical Contrastive Learning for Emotion Recognition in Conversation]

• [78] X Li. [Token-channel compounded Cross Attention for Multimodal Emotion Recognition]

• [79] Z Zhao, Y Wang, Y Xu, J Zhang. [Transformer-Based Deep-Scale Fusion Network for Multimodal Emotion Recognition]

• [80] L Stappen, A Baird, L Christ, L Schumann…. [Sentiment, Emotion, Physiological-Emotion, and Stress]

• [81] S Qiu, N Sekhar, P Singhal. [Topic and Style-aware Transformer for Multimodal Emotion Recognition]

• [82] G Hu, TE Lin, Y Zhao, G Lu, Y Wu, Y Li. [Towards Unified Multimodal Sentiment Analysis and Emotion Recognition]

• [83] D Sun, Y He, J Han. [USING AUXILIARY TASKS IN MULTIMODAL FUSION OF WAV2VEC 2.0 AND BERT FOR MULTIMODAL EMOTION RECOGNITION]

• [84] J Hu, Y Liu, J Zhao, Q Jin. [Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation]

• [85] L Stappen, A Baird, L Christ, L Schumann…. [Sentiment, Emotion, Physiological-Emotion, and Stress]

Multimodal Aspect-based Sentiment Analysis (MABSA)

Research

• [1] H Zhao, M Yang, X Bai, H Liu. [A Survey on Multimodal Aspect-Based Sentiment Analysis]

• [2] C Wang, Y Luo, C Meng, F Yuan. [An Adaptive Dual Graph Convolution Fusion Network for Aspect-Based Sentiment Analysis]

• [3] R Zhou, W Guo, X Liu, S Yu, Y Zhang…. [AoM Detecting Aspect-oriented Information for Multimodal]

• [4] L Xiao, X Wu, S Yang, J Xu, J Zhou, L He. [Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis]

• [5] L Yang, JC Na, J Yu. [Cross-Modal Multitask Transformer for End-to-End Multimodal Aspect-Based Sentiment Analysis]

• [6] Z Yu, J Wang, LC Yu, X Zhang. [Dual-Encoder Transformers with Cross-modal Alignment for Multimodal Aspect-based Sentiment Analysis]

• [7] Jianfei Yu, Jing Jiang, Rui Xia. [Entity-Sensitive_Attention_and_Fusion_Network_for_Entity-Level_Multimodal_Sentiment_Classification]

• [8] X Yang, S Feng, D Wang, S Qi, W Wu, Y Zhang…. [Few-shot Joint Multimodal Aspect-Sentiment Analysis Based on Generative Multimodal Prompt]

• [9] X Yang, S Feng, D Wang, S Qi, W Wu, Y Zhang…. [Few-shot joint multimodal aspect-sentiment analysis based on generative]

• [10] Jun Zhao, Fuping Yang. [Fusion_with_GCN_and_SE-ResNeXt_Network_for_Aspect_Based_Multimodal_Sentiment_Analysis]

• [11] Junjie Xu, Shuwen Yang, Luwei Xiao, Zhichao Fu, Xingjiao Wu, Tianlong Ma,... . [Graph_Convolution_over_the_Semantic-syntactic_Hybrid_Graph_Enhanced_by_Affective_Knowledge_for_Aspect-level_Sentiment_Classification]

• [12] Jianfei Yu, Kai Chen, Rui Xia. [Hierarchical_Interactive_Multimodal_Transformer_for_Aspect-Based_Multimodal_Sentiment_Analysis]

• [13] K Zhang, K Zhang, M Zhang, H Zhao, Q Liu…. [Incorporating Dynamic Semantics into Pre-Trained Language Model for Aspect-based Sentiment Analysis]

• [14] X Ju, D Zhang, R Xiao, J Li, S Li…. [Joint Multi-modal Aspect-Sentiment Analysis with Auxiliary Cross-modal Relation Detection]

• [15] X Ju, D Zhang, R Xiao, J Li, S Li…. [Joint Multi-modal Aspect-Sentiment Analysis with Auxiliary Cross-modal]

• [16] F Zhao, C Li, Z Wu, Y Ouyang, J Zhang…. [M2DF Multi-grained Multi-curriculum Denoising Framework for]

• [17] F Zhao, C Li, Z Wu, Y Ouyang, J Zhang…. [M2DF- Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis]

• [18] J Zhou, J Zhao, JX Huang, QV Hu, L He. [MASAL A large-scale dataset for multimodal aspect-based sentiment]

• [19] PictureCarol Xu, PictureXuan Luo, PictureDan Wang. [MMCPR: A Chinese Product Review Dataset for Multimodal Aspect-Based Sentiment Analysis]

• [20] Jie Mu, Feiping Nie, Wei Wang, Jian Xu, Jing Zhang, Han Liu. [MOCOLNet_A_Momentum_Contrastive_Learning_Network_for_Multimodal_Aspect-Level_Sentiment_Analysis]

• [21] Z Zhang, Z Wang, X Li, N Liu, B Guo, Z Yu. [ModalNet_an Aspect-level sentiment classification model by exploring multimodal data with fusion discriminant attentional networks]

• [22] H Jin, J Tan, L Liu, L Qiu, S Yao, X Chen…. [MSRA- A Multi-Aspect Semantic Relevance Approach for E-Commerce via Multimodal Pre-Training]

• [23] J Yang, Y Xiao, X Du. [Multi-grained fusion network with self-distillation for aspect-based multimodal sentiment analysis]

• [24] N Xu, W Mao, G Chen. [Multi-Interactive Memory Network for Aspect Based Multimodal Sentiment Analysis]

• [25] N Xu, W Mao, G Chen. [Multi_Interactive Memory Network for Aspect Based Multimodal Sentiment Analysis]

• [26] M Anschütz, T Eder, G Groh. [Retrieving Users’ Opinions on Social Media with Multimodal Aspect-Based Sentiment Analysis]

• [27] J Yu, J Wang, R Xia, J Li. [Targeted Multimodal Sentiment Classification Based on]

• [28] Donghong Gu, Jiaqian Wang, Shaohua Cai, Chi Yang, Zhengxin Song. [Targeted_Aspect-Based_Multimodal_Sentiment_Analysis_An_Attention_Capsule_Extraction_and_Multi-Head_Fusion_Network]

• [29] Y Ling, J Yu, R Xia. [Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis 2]

• [30] Y Ling, J Yu, R Xia. [Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis]

• [31] QT Truong, HW Lauw. [VistaNet_Visual Aspect Attention Network for Multimodal Sentiment Analysis]

• [32] D Lu, L Neves, V Carvalho, N Zhang…. [Visual Attention Model for Name Tagging in Multimodal Social Media]

Multimodal Multi-label Emotion Recognition (MMER)

Research

• [1] P Yang, F Luo, S Ma, J Lin, X Sun. [A Deep Reinforced Sequence-to-Set Model for Multi-Label Classification]

• [2] D Zhou, Y Xiang, L Zhang, C Ye…. [A Divide-And-Conquer Approach for Multi-label Multi-hop Relation Detection in Knowledge Base Question Answering]

• [3] C Wu, L Cao, Y Ge, Y Liu, M Zhang, J Su. [A Label Dependence-Aware Sequence Generation Model for Multi-Level Implicit Discourse Relation Recognition]

• [4] R You, Z Guo, L Cui, X Long, Y Bao…. [Cross-Modality Attention with Semantic Graph Embedding for Multi Label Classification]

• [5] S Qian, D Xue, H Zhang, Q Fang, C Xu. [Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval]

• [6] Y Wang, Y Xie, Y Liu, K Zhou, X Li. [Fast Graph Convolution Network Based Multi-label Image Recognition via Cross-modal Fusion]

• [7] L Qiu, Y Yang, CC Cao, Y Zheng, H Ngai…. [Generating Perturbation-based Explanations with Robustness to Out-of-Distribution Data]

• [8] H Lian, C Lu, S Li, Y Zhao, C Tang, Y Zong…. [Label Distribution Adaptation for Multimodal Emotion Recognition with Multi-label Learning]

• [9] L Xiao, X Huang, B Chen, L Jing. [Label-Specific Document Representation for Multi-Label Text Classification]

• [10] Q Ma, C Yuan, W Zhou, S Hu. [Label-Specific Dual Graph Neural Network for Multi-Label Text Classification]

• [11] S Ge, Z Jiang, Z Cheng, C Wang, Y Yin…. [Learning Robust Multi-Modal Representation for Multi-Label Emotion Recognition via Adversarial Masking and Perturbation]

• [12] T Chen, M Xu, X Hui, H Wu…. [Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition]

• [13] J Zhao, T Zhang, J Hu, Y Liu, Q Jin, X Wang…. [M3ED- Multi-modal Multi-scene Multi-label Emotional Dialogue Database]

• [14] J Zhao, Y Zhao, J Li. [M3TR- Multi-modal Multi-label Recognition with Transformer]

• [15] ZM Chen, XS Wei, P Wang…. [Multi-Label Image Recognition with Graph Convolutional Networks∗]

• [16] HD Le, GS Lee, SH Kim, S Kim, HJ Yang. [Multi-Label Multimodal Emotion Recognition With Transformer-Based Fusion and Emotion-Level Representation Learning]

• [17] D Zhang, X Ju, J Li, S Li, Q Zhu…. [Multi-modal Multi-label Emotion Detection with Modality and Label Dependence]

• [18] D Zhang, X Ju, W Zhang, J Li, S Li, Q Zhu…. [Multi-modal Multi-label Emotion Recognition with Heterogeneous Hierarchical Message Passing]

• [19] A Illendula, A Sheth. [Multimodal Emotion Classification]

• [20] M Wołczyk, M Proszewska, Ł Maziarka…. [PluGeN Multi-Label Conditional Generation from Pre-trained Models]

• [21] K Zhu, J Wu. [Residual Attention- A Simple but Effective Method for Multi-Label Recognition]

• [22] P Yang, X Sun, W Li, S Ma, W Wu, H Wang. [SGM- Sequence Generation Model for Multi-Label Classification]

• [23] A Ando, R Masumura, H Kamiyama…. [Speech Emotion Recognition based on Multi-Label Emotion Existence Model]

• [24] Y Zhang, M Chen, J Shen, C Wang. [Tailor Versatile Multi-Modal Learning for Multi-Label Emotion Recognition]

• [25] J Shen, W Qiu, Y Meng, J Shang, X Ren…. [TaxoClass- Hierarchical Multi-Label Text Classification Using Only Class Names]

• [26] X Ju, D Zhang, J Li, G Zhou. [Transformer-based Label Set Generation for Multi-modal Multi-label Emotion Detection]

Multimodal Emotion-cause Pair Extraction (MECPE)###

Research

• [1] W Li, Y Li, V Pandelea, M Ge, L Zhu…. [Cause Pair Extraction in Conversations]

• [2] SV Mathur, AR Jindal, H Mittal…. [Exploring Multimodal Emotion Cause Pair Extraction as Sequence Labelling Task]

• [3] F Wang, Z Ding, R Xia, Z Li, J Yu. [Cause Pair Extraction in Conversations]

Multimodal Affective Computing with Missing_modality

• [1] T Zhou, S Canu, P Vera, S Ruan. [modality fusion based deep neural network for brain tumor segmentation with missing MR modalities]

• [2] J Zhao, R Li, Q Jin. [Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities]

• [3] W Luo, M Xu, H Lai. [Multimodal Reconstruct and Align Net for Missing Modality Problem in Sentiment Analysis]

• [4] Q Zhang, L Shi, P Liu, Z Zhu, L Xu - 2023 - Springer. [integrating consistency and difference networks by transformer for multimodal sentiment analysis]

• [5] J Zeng, T Liu, J Zhou. [assisted Multimodal Sentiment Analysis under Uncertain Missing Modalities]

• [6] S Parthasarathy, S Sundaram. [Visual Expression Recognition]

• [7] C Shang, A Palmer, J Sun, KS Chen…. [Missing View Imputation with Generative Adversarial Networks]

Probing

• [1] T Le Scao, A Fan, C Akiki, E Pavlick, S Ilić, D Hesslow… - 2023 - inria.hal.science. [A 176B-Parameter Open-Access Multilingual Language Model]

• [2] Z Elyoseph, D Hadar-Shoval, K Asraf…. [ChatGPT outperforms humans in emotional awareness evaluations]

• [3] K Schaaff, C Reinig, T Schlippe. [Exploring ChatGPT’s Empathic Abilities]

• [4] W Zhao, Y Zhao, X Lu, S Wang, Y Tong…. [Is ChatGPT Equipped with Emotional Dialogue Capabilities_]

• [5] H Touvron, L Martin, K Stone, P Albert…. [Open Foundation and Fine-Tuned Chat Models]

Survey###

• [1] Liu, H., Lou, T., Zhang, Y., Wu, Y., Xiao, Y., Jense…. [EEG-Based Multimodal Emotion Recognition: A Machine Learning Perspective_]

Multimodal Affective Computing With ChatGPT

• [1] [Exploring aspect‐based sentiment quadruple extraction with implicit aspects, opinions, and ChatGPT: a comprehensive survey_]
• [2] [Leveraging ChatGPT As Text Annotation Tool For Sentiment Analysis_]
• [3] [Sentiment Analysis of Using ChatGPT in Education_]