Statistics
Total Volume of Data
The database contains 53,597,750 publicly accessible machine-generated annotations covering 380,395 images.
Annotations date from 2018-02-08 to 2024-12-16.
Each annotation indicates an area of interest in an image. An area can be a small portion of an image or it can be the entire image. Small regions typically contain a human face or words. Annotations on the full image are typically tags (e.g. cat, watermelon, rock) and descriptions (e.g. a cow standing in a field).
Access to annotations as data is available via the annotation endpoint on the Harvard Art Museums API.
Sources
Annotations are generated from 5 computer vision services and 3 large language models.
Each service supplies a variety of functions for detecting and categorizing features. These are the services and features we use.
- Anthropic Claude (via AWS Bedrock)
- Description generation via prompting
- AWS Rekognition
- Labels, face detection, text detection
- Clarifai
- Concepts
- Google Vision
- Labels, face detection, text detection, landmark detection
- Imagga
- Tags, categories
- Meta Llama (via AWS Bedrock)
- Description generation via prompting
- Microsoft Cognitive Sources
- Tags, categories, descriptions, face detection
- OpenAI GPT (via Azure)
- Description generation via prompting
Annotation counts by source.
The raw data.
| AWS Rekognition | Anthropic | Azure OpenAI Service | Clarifai | Google Vision | Imagga | Meta | Microsoft Cognitive Services | |
|---|---|---|---|---|---|---|---|---|
| category | 0 | 0 | 0 | 0 | 0 | 952,822 | 0 | 0 |
| description | 0 | 76,337 | 76,222 | 0 | 0 | 0 | 75,452 | 826,783 |
| face | 319,697 | 0 | 0 | 0 | 107,452 | 0 | 0 | 80,369 |
| tag | 4,047,626 | 0 | 0 | 7,443,594 | 4,156,754 | 26,868,579 | 0 | 2,333,932 |
| text | 2,858,551 | 0 | 0 | 0 | 3,373,198 | 0 | 0 | 0 |
| Total | 7,225,874 | 76,337 | 76,222 | 7,443,594 | 7,637,404 | 27,821,401 | 75,452 | 3,241,084 |
Annotation Types
Annotation types generally describe what the annotation contains or depicts. These are the possible types.
- category
- a broad categorization of the contents of the image
- description
- a full sentence caption of the contents of the image
- face
- a human face is found within the annotation
- tag
- a term or set of terms describing all or part of the image
- text
- some text is found within the annotation
Distribution of sources across annotation types.
The raw data.
| category | description | face | tag | text | |
|---|---|---|---|---|---|
| AWS Rekognition | 0 | 0 | 319,697 | 4,047,626 | 2,858,551 |
| Anthropic | 0 | 76,337 | 0 | 0 | 0 |
| Azure OpenAI Service | 0 | 76,222 | 0 | 0 | 0 |
| Clarifai | 0 | 0 | 0 | 7,443,594 | 0 |
| Google Vision | 0 | 0 | 107,452 | 4,156,754 | 3,373,198 |
| Imagga | 952,822 | 0 | 0 | 26,868,579 | 0 |
| Meta | 0 | 75,452 | 0 | 0 | 0 |
| Microsoft Cognitive Services | 0 | 826,783 | 80,369 | 2,333,932 | 0 |
| Total | 952,822 | 1,054,794 | 507,518 | 44,850,866 | 6,231,750 |
Vocabularies1
The tags and descriptions cluster and breakdown in to distinct terms, descriptive phrases, concepts, and named entities.
Terms
The vocabulary of tags contains 15,133 distinct terms.
The size of vocabulary by source.
The raw data.
| # of terms | |
|---|---|
| AWS Rekognition | 2,724 |
| Clarifai | 4,541 |
| Imagga | 9,582 |
| Google Vision | 8,149 |
| Microsoft Cognitive Services | 3,471 |
Sample set of terms:
- 25 29 years (19)
- cake stand (25)
- capers (2)
- climate change (10)
- convent (91)
- cuckoo (16)
- ginger family (1)
- grandma (4,211)
- hospital ship (3)
- icebreaker (29)
- imprinted (1)
- luminous (1)
- lying (1,475)
- moses (88)
- notepaper (415)
- ring (4,354)
- tarnish (1)
- tide pool (1)
- western yellow pine (5)
- workwear (93)
Descriptive Phrases
The vocabulary of descriptions contains 41,345 descriptive phrases.
Sample set of descriptive phrases:
- a bathroom with a sink and a mirror in a room (2)
- a bird sitting on top of a field (7)
- a boat is in the middle of a body of water (3)
- a brown and black fish (1)
- a brown teddy bear sitting next to a painting (1)
- a close up of a garden (11)
- a group of men riding on the back of an open field (2)
- a laptop computer sitting on top of a book (3)
- a large building with a clock on a table (1)
- a man preparing food while standing in front of a building (1)
- a vintage photo of a horse in front of a window (2)
- a vintage photo of a man holding a cat (2)
- a vintage photo of a person in a library (1)
- a vintage photo of a person riding a jet ski in the water (1)
- a vintage photo of brigham young et al posing for the camera (1)
- a vintage photo of dan brouthers and woman posing for a picture (1)
- a white bed in a room (1)
- a woman sitting on a couch (4)
- alberto marvelli et al posing for the camera (1)
- charles augustin sainte beuve wearing a suit and tie standing in front of a window (1)
Named Entities2
Descriptions can be parsed further in to clusters of named entities.
The vocabulary of named entities contains 1,754 people.
Sample set of people:
Alexander II (13)
The entity Alexander II is present in 13 descriptions.
- a black and white photo of Alexander II of Russia
- a black and white photo of Alexander II of Russia standing posing for the camera
- a vintage photo of Alexander II of Russia
- a vintage photo of Alexander II of Russia holding a book
- a vintage photo of Alexander II of Russia in a suit and tie
- a vintage photo of Alexander II of Russia wearing a suit and tie
- Alexander II of Russia holding a book
- Alexander II of Russia holding a book posing for the camera
- Alexander II of Russia standing next to a book
- an old black and white photo of Alexander II of Russia
- an old photo of Alexander II of Russia
- an old photo of Alexander II of Russia wearing a suit and tie
- old photo of Alexander II of Russia
Charles R. Woods (4)
The entity Charles R. Woods is present in 4 descriptions.
- a vintage photo of Charles R. Woods
- a vintage photo of Charles R. Woods holding a book
- a vintage photo of Charles R. Woods sitting in a chair
- an old photo of Charles R. Woods
David Beatty (3)
The entity David Beatty is present in 3 descriptions.
- a vintage photo of David Beatty, 1st Earl Beatty
- a vintage photo of David Beatty, 1st Earl Beatty in a suit and tie
- a vintage photo of David Beatty, 1st Earl Beatty wearing a suit and tie
Edward Bellamy (7)
The entity Edward Bellamy is present in 7 descriptions.
- a black and white photo of Edward Bellamy
- a vintage photo of Edward Bellamy
- a vintage photo of Edward Bellamy in a suit and tie
- a vintage photo of Edward Bellamy wearing a suit and tie
- an old photo of Edward Bellamy
- Edward Bellamy in a suit and tie
- Edward Bellamy posing for a photo
Eva Mae LeFevre (2)
The entity Eva Mae LeFevre is present in 2 descriptions.
- Eva Mae LeFevre et al. standing in a kitchen
- Eva Mae LeFevre et al. standing in the kitchen
Francisco Lachowski (3)
The entity Francisco Lachowski is present in 3 descriptions.
- Francisco Lachowski holding a book
- Francisco Lachowski sitting on a book
- Francisco Lachowski sitting on top of a book
Garrett Morgan (3)
The entity Garrett Morgan is present in 3 descriptions.
- Garrett Morgan et al. around each other
- Garrett Morgan et al. sitting and standing in front of a building
- Garrett Morgan et al. sitting around each other
George Curtis (7)
The entity George Curtis is present in 7 descriptions.
- a old photo of George Curtis
- a vintage photo of George Curtis
- an old photo of George Curtis
- an old photo of George Curtis in a suit and tie
- George Curtis wearing a suit and tie
- old photo of George Curtis
- old photo of George Curtis in a suit and tie
Herman Melville (5)
The entity Herman Melville is present in 5 descriptions.
- Herman Melville in a dark room
- Herman Melville looking at the camera
- Herman Melville standing in a dark room
- Herman Melville standing in front of a door
- Herman Melville that is standing in the dark
John M. Clayton (3)
The entity John M. Clayton is present in 3 descriptions.
- a vintage photo of John M. Clayton
- a vintage photo of John M. Clayton holding a book
- a vintage photo of John M. Clayton in a white shirt and black text
Karl Donitz (3)
The entity Karl Donitz is present in 3 descriptions.
- a black and white photo of Karl Donitz
- a vintage photo of Karl Donitz
- an old black and white photo of Karl Donitz
Laxmi Narayan Temple (3)
The entity Laxmi Narayan Temple is present in 3 descriptions.
- a group of people walking in front of Laxmi Narayan Temple
- a person riding a horse drawn carriage in front of Laxmi Narayan Temple
- a person riding a horse in front of Laxmi Narayan Temple
Mei Lanfang (3)
The entity Mei Lanfang is present in 3 descriptions.
- Mei Lanfang sitting in front of a television
- Mei Lanfang sitting in front of a window
- Mei Lanfang standing in front of a window
Mimi Aguglia (1)
The entity Mimi Aguglia is present in 1 descriptions.
- an old photo of Mimi Aguglia
Rosa Gumataotao Rios (3)
The entity Rosa Gumataotao Rios is present in 3 descriptions.
- Rosa Gumataotao Rios posing for a picture
- Rosa Gumataotao Rios posing for the camera
- Rosa Gumataotao Rios smiling for the camera
Schuyler Hamilton (3)
The entity Schuyler Hamilton is present in 3 descriptions.
- a black and white photo of Schuyler Hamilton
- a vintage photo of Schuyler Hamilton holding a book
- an old black and white photo of Schuyler Hamilton
Sidney Dillon (3)
The entity Sidney Dillon is present in 3 descriptions.
- a black and white photo of Sidney Dillon
- a vintage photo of Sidney Dillon in a suit and tie
- an old photo of Sidney Dillon in a suit and tie
Tom Hood (2)
The entity Tom Hood is present in 2 descriptions.
- an old photo of Tom Hood
- old photo of Tom Hood
Tom Howard (2)
The entity Tom Howard is present in 2 descriptions.
- a vintage photo of Tom Howard and woman posing for a picture
- a vintage photo of Tom Howard et al. posing for a picture
William Sprague (4)
The entity William Sprague is present in 4 descriptions.
- a vintage photo of William Sprague IV
- a vintage photo of William Sprague IV holding a book
- William Sprague IV in a suit standing in front of a television
- William Sprague IV standing in front of a mirror posing for the camera
The vocabulary of named entities contains 54 places.
Sample set of places:
289 Washington St. (1)
The entity 289 Washington St. is present in 1 descriptions.
- The image appears to be a vintage business card or calling card for a photographer named S. Masury, who was a "Photographic Artist" located at 289 Washington St. in Boston.
Brazil (8)
The entity Brazil is present in 8 descriptions.
- a close up of Pedro II of Brazil holding a book
- a vintage photo of Pedro II of Brazil
- a vintage photo of Pedro II of Brazil holding a book
- a vintage photo of Pedro II of Brazil sitting on a book
- an old photo of Pedro II of Brazil
- an old photo of Pedro II of Brazil holding a book
- old photo of Pedro II of Brazil
- Pedro II of Brazil holding a book
Bulgaria (2)
The entity Bulgaria is present in 2 descriptions.
- Ferdinand I of Bulgaria in a room
- Ferdinand I of Bulgaria with collar shirt
Egypt (2)
The entity Egypt is present in 2 descriptions.
- Abbas II of Egypt in a box
- Abbas II of Egypt sitting in a box
France (6)
The entity France is present in 6 descriptions.
- a vintage photo of Louis XIV of France
- a vintage photo of Louis XIV of France holding a book
- an old photo of Louis XIV of France
- an old photo of Louis XVI of France
- an old photo of Louis XVI of France sitting on a bed
- Louis XVI of France sitting on a bed
Georgia (3)
The entity Georgia is present in 3 descriptions.
- an old photo of Georgia O'Keeffe
- Georgia O'Keeffe looking at the camera
- the face of Georgia O'Keeffe
Greece (15)
The entity Greece is present in 15 descriptions.
- a close up of George I of Greece holding a sign
- a old photo of George I of Greece
- a vintage photo of George I of Greece holding a book
- a vintage photo of George I of Greece in a suit and tie
- a vintage photo of George I of Greece wearing a suit and tie
- an old photo of George I of Greece
- an old photo of Princess Cecilie of Greece and Denmark
- an old photo of Princess Cecilie of Greece and Denmark and woman posing for a picture
- George I of Greece holding a sign
- George I of Greece standing in front of a sign
- old photo of George I of Greece
- old photo of Princess Cecilie of Greece and Denmark et al. standing in front of a window
- Princess Cecilie of Greece and Denmark, Bobby Jordan posing for a photo
- Princess Cecilie of Greece and Denmark, Bobby Jordan posing for a picture
- Princess Cecilie of Greece and Denmark, Bobby Jordan posing for the camera
Israel (2)
The entity Israel is present in 2 descriptions.
- a vintage photo of Israel Silvestre
- a vintage photo of Israel Silvestre holding a book
Jerusalem (3)
The entity Jerusalem is present in 3 descriptions.
- a close up of a book cover with Temple in Jerusalem in the background
- a close up of a book with Temple in Jerusalem in the background
- close up of a book with Temple in Jerusalem in the background
Milan (3)
The entity Milan is present in 3 descriptions.
- an old photo of a large building with Milan Cathedral in the background
- an old photo of Milan Cathedral
- an old photo of Milan Cathedral street
Montenegro (5)
The entity Montenegro is present in 5 descriptions.
- a vintage photo of Elena of Montenegro
- a vintage photo of Elena of Montenegro et al. posing for a photo
- a vintage photo of Elena of Montenegro et al. posing for a picture
- a vintage photo of Elena of Montenegro et al. posing for the camera
- an old photo of Elena of Montenegro
Rio (3)
The entity Rio is present in 3 descriptions.
- Rio Reiser sitting in a chair
- Rio Reiser wearing a black hat
- Rio Reiser wearing a hat
Romania (19)
The entity Romania is present in 19 descriptions.
- a vintage photo of Carol I of Romania holding a book
- a vintage photo of Carol I of Romania holding a book posing for the camera
- a vintage photo of Carol I of Romania standing in front of a book
- a vintage photo of Carol I of Romania wearing a suit and tie
- a vintage photo of Ferdinand I of Romania
- a vintage photo of Marie of Romania et al. posing for the camera
- a vintage photo of Marie of Romania et al. sitting in a box
- a vintage photo of Marie of Romania, Ferdinand I of Romania posing for a picture
- a vintage photo of Marie of Romania, Ferdinand I of Romania posing for a picture
- a vintage photo of Marie of Romania sitting in a box
- an old photo of Elisabeth of Romania
- Elisabeth of Romania posing for a photo
- Elisabeth of Romania posing for a picture
- Marie of Romania, Ferdinand I of Romania are posing for a picture
- Marie of Romania, Ferdinand I of Romania are posing for a picture
- Marie of Romania in a white box
- Marie of Romania posing for a photo
- Marie of Romania sitting in a box
- Marie of Romania with collar shirt
Rome (12)
The entity Rome is present in 12 descriptions.
- a close up of an old building with Pantheon, Rome in the background
- a vintage photo of a busy city street with Pantheon, Rome in the background
- a vintage photo of a church with Pantheon, Rome in the background
- a vintage photo of a horse drawn carriage on Pantheon, Rome street
- a vintage photo of an old building in the background with Pantheon, Rome in the background
- a vintage photo of an old building with Pantheon, Rome in the background
- a vintage photo of an old church with Pantheon, Rome in the background
- a vintage photo of an old stone building with Pantheon, Rome in the background
- a vintage photo of Pantheon, Rome
- a vintage photo of Pantheon, Rome street
- an old photo of a large building with Pantheon, Rome in the background
- an old photo of Pantheon, Rome
Spain (12)
The entity Spain is present in 12 descriptions.
- a black and white photo of Alfonso XIII of Spain
- a close up of Philip III of Spain wearing a costume
- a vintage photo of Alfonso XII of Spain holding a book
- a vintage photo of Alfonso XIII of Spain
- a vintage photo of Isabella II of Spain
- a vintage photo of Philip III of Spain
- Alfonso XIII of Spain posing for a photo
- an old black and white photo of Alfonso XIII of Spain
- an old photo of Alfonso XIII of Spain
- old photo of Alfonso XIII of Spain
- Philip III of Spain wearing a costume
- Philip III of Spain wearing a dress
St (6)
The entity St is present in 6 descriptions.
- a close up of a church with St Mark's Basilica in the background
- a group of people riding on the back of a church with St Mark's Basilica in the background
- a large building with St Mark's Basilica in the background
- a large old building with many windows with St Mark's Basilica in the background
- an old church with St Mark's Basilica in the background
- an old photo of a church with St Mark's Basilica in the background
Texas (3)
The entity Texas is present in 3 descriptions.
- a black and white photo of Texas Jack Omohundro wearing a suit and tie
- a vintage photo of Texas Jack Omohundro wearing a suit and tie
- Texas Jack Omohundro wearing a suit and tie
Virginia (5)
The entity Virginia is present in 5 descriptions.
- Virginia Mayo in front of a mirror posing for the camera
- Virginia Mayo standing in front of a mirror posing for the camera
- Virginia Weidler et al. posing for a photo
- Virginia Weidler et al. posing for a picture
- Virginia Weidler et al. posing for the camera
Washington (14)
The entity Washington is present in 14 descriptions.
- a black and white photo of Washington Irving
- a large statue in front of Washington Square Park
- a large stone statue in front of Washington Square Park
- a statue in front of Washington Square Park
- a vintage photo of Washington Irving
- a vintage photo of Washington Irving holding a book
- a vintage photo of Washington Irving in a suit and tie
- a vintage photo of Washington Irving in a white shirt and black text
- a vintage photo of Washington Irving wearing a suit and tie
- an old black and white photo of Washington Irving
- an old photo of Washington Irving
- Washington Irving posing for a photo
- Washington Irving posing for a picture
- Washington Irving wearing a suit and tie
london (2)
The entity london is present in 2 descriptions.
- a large clock tower towering over the city of london
- a tall clock tower towering over the city of london
The vocabulary of named entities contains 11 organizations.
Sample set of organizations:
Andy Warhol Museum (1)
The entity Andy Warhol Museum is present in 1 descriptions.
- a close up of a book with The Andy Warhol Museum in the background
Bruton Parish Church (3)
The entity Bruton Parish Church is present in 3 descriptions.
- a house that has a sign on the side of Bruton Parish Church
- a sign in front of a house with Bruton Parish Church in the background
- a sign on the side of a house with Bruton Parish Church in the background
Lincoln Memorial (21)
The entity Lincoln Memorial is present in 21 descriptions.
- a group of people posing for a photo with Lincoln Memorial in the background
- a group of people posing for a picture with Lincoln Memorial in the background
- a group of people posing for the camera with Lincoln Memorial in the background
- a man and a woman standing in front of a window with Lincoln Memorial in the background
- a man sitting on a bench with Lincoln Memorial in the background
- a man standing in front of a mirror with Lincoln Memorial in the background
- a man standing in front of a window with Lincoln Memorial in the background
- a vintage photo of a group of people posing for a picture with Lincoln Memorial in the background
- a vintage photo of a group of people posing for the camera with Lincoln Memorial in the background
- a vintage photo of a man holding a book with Lincoln Memorial in the background
- a vintage photo of a man sitting on a bench with Lincoln Memorial in the background
- a vintage photo of a man standing in front of a book with Lincoln Memorial in the background
- a vintage photo of a man with Lincoln Memorial in the background
- a vintage photo of a person holding a book with Lincoln Memorial in the background
- a vintage photo of a person standing in front of Lincoln Memorial
- a vintage photo of a person with Lincoln Memorial in the background
- a vintage photo of an old man standing in front of a book with Lincoln Memorial in the background
- an old photo of a cake with Lincoln Memorial in the background
- an old photo of a man with Lincoln Memorial in the background
- an old photo of a person with Lincoln Memorial in the background
- old photo of a person with Lincoln Memorial in the background
Luther King Jr. Memorial (3)
The entity Luther King Jr. Memorial is present in 3 descriptions.
- a statue of a person with Martin Luther King Jr. Memorial in the background
- a stone statue of a person with Martin Luther King Jr. Memorial in the background
- an old photo of a person with Martin Luther King Jr. Memorial in the background
Notre (6)
The entity Notre is present in 6 descriptions.
- a close up of a church with Notre Dame de Paris in the background
- a close up of an old building with Notre Dame de Paris in the background
- a close up of an old church with Notre Dame de Paris in the background
- an old photo of a large building with Notre Dame de Paris in the background
- an old photo of Notre Dame de Paris
- old photo of Notre Dame de Paris
Pergamon Museum (3)
The entity Pergamon Museum is present in 3 descriptions.
- a close up of a sign with Pergamon Museum in the background
- a sign on a wall with Pergamon Museum in the background
- close up of a sign with Pergamon Museum in the background
Robert Ford (3)
The entity Robert Ford is present in 3 descriptions.
- a vintage photo of Robert Ford et al. posing for a picture
- a vintage photo of Robert Ford et al. posing for the camera
- a vintage photo of Robert Ford et al. sitting posing for the camera
Salisbury Cathedral (3)
The entity Salisbury Cathedral is present in 3 descriptions.
- a vintage photo of a church with Salisbury Cathedral in the background
- a vintage photo of an old building with Salisbury Cathedral in the background
- a vintage photo of an old church with Salisbury Cathedral in the background
Trinity Church (6)
The entity Trinity Church is present in 6 descriptions.
- a close up of a tall building in Trinity Church
- a tall building in Trinity Church
- a vintage photo of a castle with Trinity Church in the background
- a vintage photo of an old building with Trinity Church in the background
- an old photo of a castle with Trinity Church in the background
- an old photo of Trinity Church
Wallace Ford (3)
The entity Wallace Ford is present in 3 descriptions.
- a vintage photo of Wallace Ford et al. posing for a photo
- a vintage photo of Wallace Ford et al. posing for a picture
- a vintage photo of Wallace Ford et al. posing for the camera
red cross (1)
The entity red cross is present in 1 descriptions.
- The image shows a metal can or container with a red cross symbol on it, and a screwdriver with a wooden handle.
1 Vocabularies are slow to compile so they are built about once a month. The current vocabulary was built on 2024-11-14.
2 Named entities are extracted using Compromise, a javascript NLP library. It's not exact, but pretty close.
