Cardiff University | Prifysgol Caerdydd ORCA
Online Research @ Cardiff 
WelshClear Cookie - decide language by browser settings

APSE: Attention-aware polarity-sensitive embedding for emotion-based image retrieval

Yao, Xingxu, Zhao, Sicheng, Lai, Yu-Kun, She, Dongyu, Liang, Jie and Yang, Jufeng 2020. APSE: Attention-aware polarity-sensitive embedding for emotion-based image retrieval. IEEE Transactions on Multimedia 10.1109/TMM.2020.3042664

[img]
Preview
PDF - Accepted Post-Print Version
Download (1MB) | Preview

Abstract

With the popularity of social media, an increasing number of people are accustomed to expressing their feelings and emotions online using images and videos. An emotion-based image retrieval (EBIR) system is useful for obtaining visual contents with desired emotions from a massive repository. Existing EBIR methods mainly focus on modeling the global characteristics of visual content without considering the crucial role of informative regions of interest in conveying emotions. Further, they ignore the hierarchical relationships between coarse polarities and fine categories of emotions. In this paper, we design an attention-aware polarity-sensitive embedding (APSE) network to address these issues. First, we develop a hierarchical attention mechanism to automatically discover and model the informative regions of interest. Specifically, both polarity-and emotion-specific attended representations are aggregated for discriminative feature embedding. Second, we propose a generated emotion-pair (GEP) loss to simultaneously consider the inter-and intra-polarity relationships of the emotion labels. Moreover, we adaptively generate negative examples of different hard levels in the feature space guided by the attention module to further improve the performance of feature embedding. Extensive experiments on four popular benchmark datasets demonstrate that the proposed APSE method outperforms the state-of-the-art EBIR approaches by a large margin.

Item Type: Article
Date Type: Published Online
Status: In Press
Schools: Computer Science & Informatics
Publisher: Institute of Electrical and Electronics Engineers (IEEE)
ISSN: 1520-9210
Date of First Compliant Deposit: 19 December 2020
Date of Acceptance: 22 November 2020
Last Modified: 20 Jan 2021 15:22
URI: http://orca.cf.ac.uk/id/eprint/137133

Actions (repository staff only)

Edit Item Edit Item

Downloads

Downloads per month over past year

View more statistics