Using Computer Vision to Model Fashion Outfit Compatibility

Abstract

Fashion retailers generate large amounts of clothing related data such as clothing item images, clothing item metadata, and outfits. Computer vision can aid in the task of building outfits by creating a model that learns both similarity between clothing items that are interchangeable and compatibilitybetween clothing items of different type that go well together in an outfit. To achieve this, a model needs to compare images across various similarity conditions such as color, shape, and category. A recent state-of-the-art method named Similarity Condition Embedding Network (SCE-Net) learns multiple similarity conditions without explicit supervision from a unified embedding space that produce image embeddings thatcan be used to score outfits. In this paper, we examine the performance of this network on outfit compatibility and fill-in-the-blank tasks for an online clothing retail dataset from H&M to better understand how the network learns concepts of similarity and compatibility in the fashion domain. To further explore its performance we also create a messaging app that acts as a virtual stylist by using the trained model.