Learning user-emotion and user-feature couplings for image emotion classification

Over the past few years, image emotion classification (IEC) has received increasing research interest. Existing works usually define IEC as a multi-class classification problem from features to emotions, while the subjectivity of user perception is often ignored. However, our experimental study shows that there are coupling relationships between users and emotions, as well as users and features. To address such issues, in this paper, we propose a new IEC model, called CoupledIEC. In CoupledIEC, to capture the user-emotion coupling, a clustering-based embedding model is proposed to encode users of similar emotion preferences with close representations. To model the user-feature coupling, a convolutional neural network-based coupling learning model is developed, where the Hadamard product and the matrix product are employed respectively to capture the explicit and the implicit user-feature coupling information. The two models are then integrated in a unified neural network. The experimental results on real-world image collection demonstrate that the IEC performance can be improved significantly by taking into account user-emotion and user-feature couplings.