Stroke-based semantic segmentation for scene-level free-hand sketches

Zhang, Zhengming, Deng, Xiaoming, Li, Jinyao, Lai, Yukun ORCID: https://orcid.org/0000-0002-2094-5680, Ma, Cuixia, Liu, Yongjin and Wang, Hongan 2023. Stroke-based semantic segmentation for scene-level free-hand sketches. Visual Computer 39 , pp. 6309-6321. 10.1007/s00371-022-02731-8

Preview

PDF - Accepted Post-Print Version Download (18MB) | Preview

Abstract

Sketching is a simple and efficient way for humans to express their perceptions of the world. Sketch semantic segmentation plays a key role in sketch understanding and is widely used in sketch recognition, sketch-based image retrieval, or editing. Due to modality difference between images and sketches, existing image segmentation methods may not perform best, which overlook the sparse nature and stroke-based representation in sketches. The existing sketch semantic segmentation methods are mainly designed for single-instance sketches. In this paper, we present a new stroke-based sequential-spatial neural network (S3NN) for scene-level free-hand sketch semantic segmentation, which leverages a bidirectional LSTM and graph convolutional network to capture the sequential and spatial features of sketches. In order to address the data lacking issue, we propose the first scene-level free-hand sketch dataset (SFSD). SFSD is composed of 12K sketch-photo pairs over 40 object categories, where the sketches were completely hand-drawn and each contains 7 objects on average. We conduct comparative and ablative experiments on SFSD to evaluate the effectiveness of our method. The experimental results demonstrate that our method outperforms state-of-the-art methods. The code, models, and dataset will be made public after acceptance.

Item Type:	Article
Date Type:	Publication
Status:	Published
Schools:	Computer Science & Informatics
Publisher:	Springer
ISSN:	0178-2789
Date of First Compliant Deposit:	29 December 2022
Date of Acceptance:	14 November 2022
Last Modified:	07 Dec 2023 16:37
URI:	https://orca.cardiff.ac.uk/id/eprint/155304

Actions (repository staff only)

Edit Item