The AI Stylist

Can machine learning create your next look?

Neural Nets on the (Fashion) Runway

If artificial intelligence is to be as useful as it could be, it must learn the varied nuances of real intelligence—and few things are more nuanced, not to mention varied, than a sense of style.

To improve AI-generated recommendations, a team of computer scientists at UC San Diego and Pinterest worked together to develop an AI-powered tool that accurately matches shoes and accessories to an outfit, based on just one initial photo from the user.

Similar recommendation systems currently rely on pictures of individual articles of clothing, such as products Amazon. But these lack context, such as the season, the user’s body type, and, perhaps most importantly, the clothes they typically like to wear. This is all information that can be found in real-life images (just about any selfie, for instance). And where current recommendation systems tend to focus on similar items, they do not recommend items that are complementary.

Wang-Cheng Kang, PhD ’20, who was an intern with Pinterest at the time, built “Complete the Look,” an AI-powered tool to solve this problem. He used data from Pinterest’s “Shop the Look” service, which finds clothing items for sale similar to those displayed in any given picture of people. Training the AI to teach it a sense of style was a bit like helping it solve a puzzle—researchers divided pictures into pieces: clothes, shoes, accessories, etc., and then taught the AI which pieces are compatible to make a complete puzzle; in this case, a good outfit.

When the AI was trained enough to put together looks on its own, the tool was able to identify compatible items 86.5 to 75.3 percent of the time—better than all other similar computer tools. Researchers even gave the AI an eye for interior design—it is able to identify compatible items 80 percent of the time for home décor as well.

Researchers also wanted to know how well the tool did compared to a real person with real fashion sense. They asked four fashion experts to complete the same compatibility task—and they did not do better than the tool.

“Complete the Look” still needs to be refined further before it’s ready to give the world style tips. But in one aspect at least, it met the goal researchers set out to achieve: “The model appears to have learned a complex notion of style,” the researchers write. But since AI can’t dress itself—at least for now—it’ll have to settle for helping us dress ourselves instead.