Mining Fashion Outfit Composition Using An End-to-End Deep Learning Approach on Set Data

Abstract: Fashion composition involves deep understanding of fashion standards while incorporating creativity for choosing multiple fashion items (e.g., Jewelry, Bag, Pants, Dress). In fashion websites, popular or high-quality fashion compositions are usually designed by fashion experts and followed by large audiences. In this paper, we aim to employ a machine learning strategy to compose fashion compositions by learning directly from the fashion websites. We propose an end-to-end system to learn a fashion item embedding that helps disentangle the factors contributing to fashion popularity, such as instance aesthetics and set compatibility. Our learning system consists of 1) deep convolutional network embedding of fashion images, 2) title embedding, and 3) category embedding. To leverage the multimodal information, we develop a multiple-layer perceptron module with different pooling strategies to predict the set popularity. For our experiments, we have collected a large-scale fashion set from the fashion website Polyvore. Although fashion composition is a rather challenging task, the performance of our system is quite encouraging: we have achieved an AUC of 85\% for the fashion set popularity prediction task on the Polyvore fashion set.