About the AI Explorer
Overview
Starting in 2016 the Harvard Art Museums department of Digital Infrastructure and Emerging Technology (DIET) began using artificial intelligence to describe the museums collection. Since then DIET has built a research dataset of 46,620,957 machine-generated descriptions and tags covering 315,929 images of artworks. Ranging from feature recognition to face analysis that predicts gender, age, and emotion, the data reveals how computers interpret paintings, photographs, and sculptures. This website allows users to explore the extensive collection of data by searching for artworks by machine-generated keyword and looking at aggregated data for individual pieces.
Why?
The Harvard Art Museums is using computer vision and AI for two primary reasons. The first is to categorize, tag, describe, and annotate its collection of art pieces in ways that the staff of curators don't. Since the computer lacks any context or formal training in art history, the machine views and annotates our collection as if walking into an art museum for the first time. The perspective offered by AI leans closer to reflecting the public rather than experts. Currently, the Harvard Art Museums’ search interface relies on descriptions written by art historians. The addition of AI-generated annotations to our database makes the Harvard Art Museums’ art collection more accessible to non-specialists.
The second reason is to build a dataset for researching how AI services operate. All of the services we use are black boxes. This means the services do not disclose the algorithms and training sets used in their systems so we are left to guess how they operate. We use them and provide the data, in part, to call attention to the differences and biases inherent in AI services.
How?
The Harvard Art Museums has collected artificially-generated data on artworks from five different AI and computer-vision services: Amazon Rekognition, Clarifai, Imagga, Google Vision, and Microsoft Cognitive Services. For each art piece, these services provide interpretations otherwise known as “annotations” that include generated tags and captions and object, face, and text recognition. When a user searches for a keyword, this site takes the user-inputted keyword and finds artworks that contain a matching machine-generated tag. From there, the user can go to an individual piece to see and compare the annotations from the five AI services.
What?
We've learned a bit about our collections thanks to these services. Hear about what we've discovered in the presentation, ‘Elephants on Parade or: A Cavalcade of Discoveries from Five CV Systems‘, given by Jeff Steward at the AEOLIAN Network workshop ‘Reimagining Industry / Academic / Cultural Heritage Partnerships in AI’ on Monday, October 25, 2021.
