Why?

The Harvard Art Museums is using computer vision and AI for two primary reasons. The first is to categorize, tag, describe, and annotate its collection of art pieces in ways that the staff of curators don't. Since the computer lacks any context or formal training in art history, the machine views and annotates our collection as if walking into an art museum for the first time. The perspective offered by AI leans closer to reflecting the public rather than experts. Currently, the Harvard Art Museums’ search interface relies on descriptions written by art historians. The addition of AI-generated annotations to our database makes the Harvard Art Museums’ art collection more accessible to non-specialists.

The second reason is to build a dataset for researching how AI services operate. All of the services we use are black boxes. This means the services do not disclose the algorithms and training sets used in their systems so we are left to guess how they operate. We use them and provide the data, in part, to call attention to the differences and biases inherent in AI services.

How?

The Harvard Art Museums has collected artificially-generated data on artworks from five different AI and computer-vision services: Amazon Rekognition, Clarifai, Imagga, Google Vision, and Microsoft Cognitive Services. For each art piece, these services provide interpretations otherwise known as “annotations” that include generated tags and captions and object, face, and text recognition. When a user searches for a keyword, this site takes the user-inputted keyword and finds artworks that contain a matching machine-generated tag. From there, the user can go to an individual piece to see and compare the annotations from the five AI services.