One for All: A Unified Generative Framework for Image Emotion Classification

Ge Shi, Sinuo Deng, Bo Wang, Chong Feng, Yan Zhuang, Xiaomei Wang

科研成果: 期刊稿件文章同行评审

1 引用 (Scopus)

摘要

Image Emotion Classification (IEC) is an essential research area, offering valuable insights into user emotional states for a wide range of applications, including opinion mining, recommendation systems, and mental health treatment. The challenges associated with IEC are mainly attributed to the complexity and ambiguity of human emotions, the lack of a universally accepted emotion model, and excessive dependence on prior knowledge. To address these challenges, we propose a novel Unified Generative framework for Image Emotion Classification (UGRIE), which is capable of simultaneously modeling various emotion models and capturing intricate semantic relationships between emotion labels. Our approach employs a flexible natural language template, converting the IEC task into a template-filling process that can be easily adapted to accommodate a diverse range of IEC tasks. To further enhance the performance, we devise a mapping mechanism to seamlessly integrate the multimodal pre-training model CLIP with the text generation pre-training model BART, thus leveraging the strengths of both models. A comprehensive set of experiments conducted on multiple public datasets demonstrates that our proposed method consistently outperforms existing approaches to a large margin in supervised settings, exhibits remarkable performance in low-resource scenarios, and unifies distinct emotion models within a single, versatile framework.

源语言英语
页(从-至)1
页数1
期刊IEEE Transactions on Circuits and Systems for Video Technology
DOI
出版状态已接受/待刊 - 2023

指纹

探究 'One for All: A Unified Generative Framework for Image Emotion Classification' 的科研主题。它们共同构成独一无二的指纹。

引用此

Shi, G., Deng, S., Wang, B., Feng, C., Zhuang, Y., & Wang, X. (已接受/印刷中). One for All: A Unified Generative Framework for Image Emotion Classification. IEEE Transactions on Circuits and Systems for Video Technology, 1. https://doi.org/10.1109/TCSVT.2023.3341840