Researchers reveal AI algorithm for analysing great works of art

MIT CSAIL PhD student Mark Hamilton and his team of researchers have unveiled an AI algorithm called MosAIc that looks for hidden links in great works of art.


To develop it, Hamilton and his team have partnered with Microsoft.


Once activated, the algorithm can scan for similarities across a myriad of paintings to uncover parallels in:


  • Motifs
  • Themes
  • Visual styles

MosAIc is currently being run on works of art in the Rijksmuseum and the Metropolitan Museum of Art.

A single image is all it takes to uncover similarities across different cultures.

For example, the algorithm was able to make a connection between the Dutch Double Face Banyan (an 18th-century piece of clothing) and a Chinese ceramic figurine.

The flow of porcelain and iconography is similar between the Chinese and Dutch markets despite being centuries apart.

While developing the algorithm, the researchers relied on the ‘k-nearest neighbors’ (KNN) algorithm, which is used to discover similar objects.

However, the original KNN algorithm has certain limitations.

For instance, using the bare-bones KNN algorithm to compare paintings, it is only possible to query for similar artwork from a specific artist.

While ‘unconditional’ queries are an option, the results it returns would still have to be filtered by hand, which is a time-consuming and costly process.

The team of researchers created a conditional image retrieval process (CIR) – this takes the KNN algorithm as a foundation but upgrades it so that it doesn’t stop searching until pre-determined conditions are fulfilled to find the closest match, such as:

  • Pose
  • Texture
  • Content
  • Colour

The new algorithm can also be used beyond the scope of art – for example, it can detect images that are deepfakes.

Hamilton hopes that fields such as medicine and social science will also embrace the technology.