Statistics
Total Volume of Data
The database contains 53,516,727 publicly accessible machine-generated annotations covering 380,022 images.
Annotations date from 2018-02-08 to 2024-12-09.
Each annotation indicates an area of interest in an image. An area can be a small portion of an image or it can be the entire image. Small regions typically contain a human face or words. Annotations on the full image are typically tags (e.g. cat, watermelon, rock) and descriptions (e.g. a cow standing in a field).
Access to annotations as data is available via the annotation endpoint on the Harvard Art Museums API.
Sources
Annotations are generated from 5 computer vision services and 3 large language models.
Each service supplies a variety of functions for detecting and categorizing features. These are the services and features we use.
- Anthropic Claude (via AWS Bedrock)
- Description generation via prompting
- AWS Rekognition
- Labels, face detection, text detection
- Clarifai
- Concepts
- Google Vision
- Labels, face detection, text detection, landmark detection
- Imagga
- Tags, categories
- Meta Llama (via AWS Bedrock)
- Description generation via prompting
- Microsoft Cognitive Sources
- Tags, categories, descriptions, face detection
- OpenAI GPT (via Azure)
- Description generation via prompting
Annotation counts by source.
The raw data.
| AWS Rekognition | Anthropic | Azure OpenAI Service | Clarifai | Google Vision | Imagga | Meta | Microsoft Cognitive Services | |
|---|---|---|---|---|---|---|---|---|
| category | 0 | 0 | 0 | 0 | 0 | 952,822 | 0 | 0 |
| description | 0 | 49,257 | 49,124 | 0 | 0 | 0 | 48,607 | 826,783 |
| face | 319,697 | 0 | 0 | 0 | 107,452 | 0 | 0 | 80,369 |
| tag | 4,047,626 | 0 | 0 | 7,443,594 | 4,156,754 | 26,868,579 | 0 | 2,333,932 |
| text | 2,858,551 | 0 | 0 | 0 | 3,373,198 | 0 | 0 | 0 |
| Total | 7,225,874 | 49,257 | 49,124 | 7,443,594 | 7,637,404 | 27,821,401 | 48,607 | 3,241,084 |
Annotation Types
Annotation types generally describe what the annotation contains or depicts. These are the possible types.
- category
- a broad categorization of the contents of the image
- description
- a full sentence caption of the contents of the image
- face
- a human face is found within the annotation
- tag
- a term or set of terms describing all or part of the image
- text
- some text is found within the annotation
Distribution of sources across annotation types.
The raw data.
| category | description | face | tag | text | |
|---|---|---|---|---|---|
| AWS Rekognition | 0 | 0 | 319,697 | 4,047,626 | 2,858,551 |
| Anthropic | 0 | 49,257 | 0 | 0 | 0 |
| Azure OpenAI Service | 0 | 49,124 | 0 | 0 | 0 |
| Clarifai | 0 | 0 | 0 | 7,443,594 | 0 |
| Google Vision | 0 | 0 | 107,452 | 4,156,754 | 3,373,198 |
| Imagga | 952,822 | 0 | 0 | 26,868,579 | 0 |
| Meta | 0 | 48,607 | 0 | 0 | 0 |
| Microsoft Cognitive Services | 0 | 826,783 | 80,369 | 2,333,932 | 0 |
| Total | 952,822 | 973,771 | 507,518 | 44,850,866 | 6,231,750 |
Vocabularies1
The tags and descriptions cluster and breakdown in to distinct terms, descriptive phrases, concepts, and named entities.
Terms
The vocabulary of tags contains 15,133 distinct terms.
The size of vocabulary by source.
The raw data.
| # of terms | |
|---|---|
| AWS Rekognition | 2,724 |
| Clarifai | 4,541 |
| Imagga | 9,582 |
| Google Vision | 8,149 |
| Microsoft Cognitive Services | 3,471 |
Sample set of terms:
Descriptive Phrases
The vocabulary of descriptions contains 41,345 descriptive phrases.
Sample set of descriptive phrases:
- a bird swimming in water next to the ocean (2)
- a black and white photo of a cup (6)
- a close up of a coffee cup in front of a bowl (1)
- a close up of a pink screen (1)
- a close up of a window in a dark room (1)
- a dining room table in the dark (1)
- a display of chain (13)
- a group of women cooking in a kitchen (2)
- a person is sitting on a rock (3)
- a person standing in front of a beach (71)
- a pile of rocks (14)
- a pizza sitting on top of a wooden table (3)
- a red and white flag (6)
- a vintage photo of a young man wearing a hat posing for the camera (1)
- a vintage photo of alma reville et al posing for a picture (2)
- a vintage photo of joshua chamberlain (1)
- a vintage photo of prince jerome napoleon holding a book posing for the camera (1)
- a whiteboard with black text on a white background (3)
- alexandra of denmark posing for a photo in front of a book (1)
- thomas lemuel james holding a book (2)
Named Entities2
Descriptions can be parsed further in to clusters of named entities.
The vocabulary of named entities contains 1,754 people.
Sample set of people:
Anne Holm (5)
The entity Anne Holm is present in 5 descriptions.
- a vintage photo of John Ruskin, Anne Holm posing for a picture
- a vintage photo of John Ruskin, Anne Holm posing for the camera
- Anne Holm et al. sitting on a bench
- Anne Holm et al. that are sitting on a bench
- John Ruskin, Anne Holm posing for a photo
Benjamin Disraeli (13)
The entity Benjamin Disraeli is present in 13 descriptions.
- a black and white photo of Benjamin Disraeli
- a old photo of Benjamin Disraeli
- a vintage photo of Benjamin Disraeli
- a vintage photo of Benjamin Disraeli holding a book
- a vintage photo of Benjamin Disraeli holding a book posing for the camera
- a vintage photo of Benjamin Disraeli in a suit and tie
- a vintage photo of Benjamin Disraeli standing in front of a book
- a vintage photo of Benjamin Disraeli standing in front of a building
- an old black and white photo of Benjamin Disraeli
- an old photo of Benjamin Disraeli
- an old photo of Benjamin Disraeli holding a book
- Benjamin Disraeli holding a book
- old photo of Benjamin Disraeli
Charles Sanders Peirce (7)
The entity Charles Sanders Peirce is present in 7 descriptions.
- a vintage photo of Charles Sanders Peirce
- a vintage photo of Charles Sanders Peirce in a suit and tie
- a vintage photo of Charles Sanders Peirce in a suit posing for the camera
- a vintage photo of Charles Sanders Peirce wearing a suit and tie
- an old photo of Charles Sanders Peirce
- an old photo of Charles Sanders Peirce wearing a suit and tie
- old photo of Charles Sanders Peirce
David Smith (3)
The entity David Smith is present in 3 descriptions.
- a vintage photo of David Smith
- a vintage photo of David Smith in a room
- a vintage photo of David Smith standing in front of a store
Enid Lyons (1)
The entity Enid Lyons is present in 1 descriptions.
- a vintage photo of Enid Lyons
Francis Gregory (3)
The entity Francis Gregory is present in 3 descriptions.
- a vintage photo of Francis Gregory in a suit and tie
- a vintage photo of Francis Gregory wearing a suit and tie
- an old photo of Francis Gregory in a suit and tie
Frederic Edwin Church (3)
The entity Frederic Edwin Church is present in 3 descriptions.
- a black and white photo of Frederic Edwin Church wearing a suit and tie
- a vintage photo of Frederic Edwin Church in a suit and tie
- a vintage photo of Frederic Edwin Church wearing a suit and tie
George H. W. (6)
The entity George H. W. is present in 6 descriptions.
- a black and white photo of George H. W. Bush
- a vintage photo of George H. W. Bush wearing a suit and tie
- an old black and white photo of George H. W. Bush
- an old photo of George H. W. Bush
- an old photo of George H. W. Bush in a suit and tie
- an old photo of George H. W. Bush wearing a suit and tie
George Q. Cannon (4)
The entity George Q. Cannon is present in 4 descriptions.
- a black and white photo of George Q. Cannon wearing a suit and tie
- a vintage photo of George Q. Cannon in a suit and tie
- a vintage photo of George Q. Cannon wearing a suit and tie
- an old photo of George Q. Cannon wearing a suit and tie
Jack Omohundro (3)
The entity Jack Omohundro is present in 3 descriptions.
- a black and white photo of Texas Jack Omohundro wearing a suit and tie
- a vintage photo of Texas Jack Omohundro wearing a suit and tie
- Texas Jack Omohundro wearing a suit and tie
James Garner (3)
The entity James Garner is present in 3 descriptions.
- a vintage photo of James Garner in a suit and tie
- a vintage photo of James Garner wearing a suit and tie
- an old photo of James Garner in a suit and tie
James Savage (3)
The entity James Savage is present in 3 descriptions.
- James Savage in a suit and tie sitting on a bench
- James Savage sitting on a bench
- James Savage wearing a suit and tie sitting on a bench
John Parke (3)
The entity John Parke is present in 3 descriptions.
- a black and white photo of John Parke
- an old photo of John Parke
- old photo of John Parke
John Van Buren (9)
The entity John Van Buren is present in 9 descriptions.
- a vintage photo of John Van Buren
- a vintage photo of John Van Buren holding a book
- a vintage photo of John Van Buren in a suit and tie
- a vintage photo of John Van Buren wearing a suit and tie
- a vintage photo of John Van Buren with an open door
- an old photo of John Van Buren
- an old photo of John Van Buren in a suit and tie
- an old photo of John Van Buren wearing a suit and tie
- John Van Buren in a suit and tie
Katharine Hepburn (3)
The entity Katharine Hepburn is present in 3 descriptions.
- Katharine Hepburn in a white shirt
- Katharine Hepburn in a white shirt and black hair
- Katharine Hepburn looking at the camera
Princess Elisabeth Helene (1)
The entity Princess Elisabeth Helene is present in 1 descriptions.
- a vintage photo of Princess Elisabeth Helene of Thurn and Taxis
Samuel Cooper (3)
The entity Samuel Cooper is present in 3 descriptions.
- a vintage photo of Samuel Cooper
- an old photo of Samuel Cooper
- old photo of Samuel Cooper
Seamus Heaney (3)
The entity Seamus Heaney is present in 3 descriptions.
- an old photo of Seamus Heaney
- old photo of Seamus Heaney
- Seamus Heaney sitting in a room
Seth Kinman (1)
The entity Seth Kinman is present in 1 descriptions.
- Seth Kinman holding a book
Varina Davis (2)
The entity Varina Davis is present in 2 descriptions.
- a vintage photo of Varina Davis
- an old photo of Varina Davis
The vocabulary of named entities contains 54 places.
Sample set of places:
289 Washington St. (1)
The entity 289 Washington St. is present in 1 descriptions.
- The image appears to be a vintage business card or calling card for a photographer named S. Masury, who was a "Photographic Artist" located at 289 Washington St. in Boston.
Albania (5)
The entity Albania is present in 5 descriptions.
- a vintage photo of Zog I of Albania et al. posing for a picture
- a vintage photo of Zog I of Albania et al. posing for the camera
- a vintage photo of Zog I of Albania standing in front of a building
- Zog I of Albania and woman posing for a photo
- Zog I of Albania et al. posing for a photo
Bavaria (25)
The entity Bavaria is present in 25 descriptions.
- a black and white photo of King Ludwig II of Bavaria
- a black and white photo of King Ludwig II of Bavaria holding a book
- a screen shot of a photo of King Ludwig II of Bavaria
- a screen shot of King Ludwig II of Bavaria
- a screen shot of King Ludwig II of Bavaria in a suit and tie
- a vintage photo of King Ludwig II of Bavaria
- a vintage photo of King Ludwig II of Bavaria and woman posing for a picture
- a vintage photo of King Ludwig II of Bavaria et al. posing for a picture
- a vintage photo of King Ludwig II of Bavaria holding a book
- a vintage photo of King Ludwig II of Bavaria in a suit and tie
- a vintage photo of King Ludwig II of Bavaria in a white shirt and black text
- a vintage photo of King Ludwig II of Bavaria making a face for the camera
- an old photo of King Ludwig II of Bavaria
- an old photo of King Ludwig II of Bavaria holding a book
- an old photo of King Ludwig II of Bavaria in a suit and tie
- an old photo of Otto of Bavaria
- King Ludwig II of Bavaria, Pope Pius IX are posing for a picture
- King Ludwig II of Bavaria, Pope Pius IX posing for a photo
- King Ludwig II of Bavaria posing for a photo
- King Ludwig II of Bavaria posing for a photo in front of a book
- King Ludwig II of Bavaria posing for the camera
- King Ludwig II of Bavaria wearing a suit and tie
- old photo of King Ludwig II of Bavaria
- Otto of Bavaria posing for a photo
- Otto of Bavaria posing for a picture
Boston. (1)
The entity Boston. is present in 1 descriptions.
- The image appears to be a vintage business card or calling card for a photographer named S. Masury, who was a "Photographic Artist" located at 289 Washington St. in Boston.
Brazil (8)
The entity Brazil is present in 8 descriptions.
- a close up of Pedro II of Brazil holding a book
- a vintage photo of Pedro II of Brazil
- a vintage photo of Pedro II of Brazil holding a book
- a vintage photo of Pedro II of Brazil sitting on a book
- an old photo of Pedro II of Brazil
- an old photo of Pedro II of Brazil holding a book
- old photo of Pedro II of Brazil
- Pedro II of Brazil holding a book
Denmark (27)
The entity Denmark is present in 27 descriptions.
- a statue of Alexandra of Denmark
- a statue of Alexandra of Denmark in a suit and tie
- a statue of Alexandra of Denmark wearing a suit and tie
- a vintage photo of Alexandra of Denmark
- a vintage photo of Alexandra of Denmark et al. posing for a photo
- a vintage photo of Alexandra of Denmark et al. posing for a picture
- a vintage photo of Alexandra of Denmark et al. posing for the camera
- a vintage photo of Alexandra of Denmark et al. sitting on a bench
- a vintage photo of Alexandra of Denmark holding a book
- a vintage photo of Alexandra of Denmark holding a sign posing for the camera
- a vintage photo of Alexandra of Denmark in a suit and tie
- a vintage photo of Princess Thyra of Denmark
- a vintage photo of Princess Thyra of Denmark holding a book
- Alexandra of Denmark holding a book
- Alexandra of Denmark holding a book posing for the camera
- Alexandra of Denmark posing for a photo in front of a book
- Alexandra of Denmark posing for the camera
- an old photo of Princess Cecilie of Greece and Denmark
- an old photo of Princess Cecilie of Greece and Denmark and woman posing for a picture
- an old photo of Princess Thyra of Denmark
- old photo of Princess Cecilie of Greece and Denmark et al. standing in front of a window
- Prince Harald of Denmark posing for the camera
- Prince Harald of Denmark standing posing for the camera
- Prince Harald of Denmark wearing a suit and tie
- Princess Cecilie of Greece and Denmark, Bobby Jordan posing for a photo
- Princess Cecilie of Greece and Denmark, Bobby Jordan posing for a picture
- Princess Cecilie of Greece and Denmark, Bobby Jordan posing for the camera
England (1)
The entity England is present in 1 descriptions.
- an old photo of Henry III of England
France (6)
The entity France is present in 6 descriptions.
- a vintage photo of Louis XIV of France
- a vintage photo of Louis XIV of France holding a book
- an old photo of Louis XIV of France
- an old photo of Louis XVI of France
- an old photo of Louis XVI of France sitting on a bed
- Louis XVI of France sitting on a bed
George Perkins Marsh (3)
The entity George Perkins Marsh is present in 3 descriptions.
- an old photo of George Perkins Marsh
- George Perkins Marsh posing for a photo
- old photo of George Perkins Marsh
Guadalupe (3)
The entity Guadalupe is present in 3 descriptions.
- an old photo of Mariano Guadalupe Vallejo
- an old photo of Mariano Guadalupe Vallejo holding a book
- Mariano Guadalupe Vallejo holding a book
Joe Hill (3)
The entity Joe Hill is present in 3 descriptions.
- a black and white photo of Joe Hill
- a vintage photo of Joe Hill in a suit and tie
- a vintage photo of Joe Hill wearing a suit and tie
Morocco (3)
The entity Morocco is present in 3 descriptions.
- an old photo of Princess Lalla Aicha of Morocco et al. in a room
- an old photo of Princess Lalla Aicha of Morocco et al. standing in a room
- Princess Lalla Aicha of Morocco et al. standing in a room
Netherlands (3)
The entity Netherlands is present in 3 descriptions.
- a black and white photo of Wilhelmina of the Netherlands
- an old black and white photo of Wilhelmina of the Netherlands
- an old photo of Wilhelmina of the Netherlands
New Mexico (9)
The entity New Mexico is present in 9 descriptions.
- a close up of an orange with Ohkay Owingeh, New Mexico in the background
- a piece of bread with Ohkay Owingeh, New Mexico in the background
- a piece of cake on a table with Ohkay Owingeh, New Mexico in the background
- a piece of food on a table with Ohkay Owingeh, New Mexico in the background
- a piece of food with Ohkay Owingeh, New Mexico in the background
- an orange cut in half and sitting on a table with Ohkay Owingeh, New Mexico in the background
- an orange cut in half on a plate with Ohkay Owingeh, New Mexico in the background
- an orange cut in half with Ohkay Owingeh, New Mexico in the background
- an orange sitting on a table with Ohkay Owingeh, New Mexico in the background
Oregon Trail (3)
The entity Oregon Trail is present in 3 descriptions.
- a vintage photo of a horse with Oregon Trail in the background
- a vintage photo of a person on a horse with Oregon Trail in the background
- a vintage photo of a person riding a horse with Oregon Trail in the background
Romania (19)
The entity Romania is present in 19 descriptions.
- a vintage photo of Carol I of Romania holding a book
- a vintage photo of Carol I of Romania holding a book posing for the camera
- a vintage photo of Carol I of Romania standing in front of a book
- a vintage photo of Carol I of Romania wearing a suit and tie
- a vintage photo of Ferdinand I of Romania
- a vintage photo of Marie of Romania et al. posing for the camera
- a vintage photo of Marie of Romania et al. sitting in a box
- a vintage photo of Marie of Romania, Ferdinand I of Romania posing for a picture
- a vintage photo of Marie of Romania, Ferdinand I of Romania posing for a picture
- a vintage photo of Marie of Romania sitting in a box
- an old photo of Elisabeth of Romania
- Elisabeth of Romania posing for a photo
- Elisabeth of Romania posing for a picture
- Marie of Romania, Ferdinand I of Romania are posing for a picture
- Marie of Romania, Ferdinand I of Romania are posing for a picture
- Marie of Romania in a white box
- Marie of Romania posing for a photo
- Marie of Romania sitting in a box
- Marie of Romania with collar shirt
Spain (12)
The entity Spain is present in 12 descriptions.
- a black and white photo of Alfonso XIII of Spain
- a close up of Philip III of Spain wearing a costume
- a vintage photo of Alfonso XII of Spain holding a book
- a vintage photo of Alfonso XIII of Spain
- a vintage photo of Isabella II of Spain
- a vintage photo of Philip III of Spain
- Alfonso XIII of Spain posing for a photo
- an old black and white photo of Alfonso XIII of Spain
- an old photo of Alfonso XIII of Spain
- old photo of Alfonso XIII of Spain
- Philip III of Spain wearing a costume
- Philip III of Spain wearing a dress
St (6)
The entity St is present in 6 descriptions.
- a close up of a church with St Mark's Basilica in the background
- a group of people riding on the back of a church with St Mark's Basilica in the background
- a large building with St Mark's Basilica in the background
- a large old building with many windows with St Mark's Basilica in the background
- an old church with St Mark's Basilica in the background
- an old photo of a church with St Mark's Basilica in the background
Sweden (5)
The entity Sweden is present in 5 descriptions.
- a vintage photo of Oscar II of Sweden
- a vintage photo of Oscar II of Sweden in a white shirt and black text
- a vintage photo of Oscar II of Sweden in white shirt and black text
- an old photo of Oscar II of Sweden
- old photo of Oscar II of Sweden
Tennessee (3)
The entity Tennessee is present in 3 descriptions.
- a vintage photo of Tennessee Williams in a suit and tie
- a vintage photo of Tennessee Williams wearing a suit and tie
- an old photo of Tennessee Williams wearing a suit and tie
The vocabulary of named entities contains 11 organizations.
Sample set of organizations:
Andy Warhol Museum (1)
The entity Andy Warhol Museum is present in 1 descriptions.
- a close up of a book with The Andy Warhol Museum in the background
Bruton Parish Church (3)
The entity Bruton Parish Church is present in 3 descriptions.
- a house that has a sign on the side of Bruton Parish Church
- a sign in front of a house with Bruton Parish Church in the background
- a sign on the side of a house with Bruton Parish Church in the background
Lincoln Memorial (21)
The entity Lincoln Memorial is present in 21 descriptions.
- a group of people posing for a photo with Lincoln Memorial in the background
- a group of people posing for a picture with Lincoln Memorial in the background
- a group of people posing for the camera with Lincoln Memorial in the background
- a man and a woman standing in front of a window with Lincoln Memorial in the background
- a man sitting on a bench with Lincoln Memorial in the background
- a man standing in front of a mirror with Lincoln Memorial in the background
- a man standing in front of a window with Lincoln Memorial in the background
- a vintage photo of a group of people posing for a picture with Lincoln Memorial in the background
- a vintage photo of a group of people posing for the camera with Lincoln Memorial in the background
- a vintage photo of a man holding a book with Lincoln Memorial in the background
- a vintage photo of a man sitting on a bench with Lincoln Memorial in the background
- a vintage photo of a man standing in front of a book with Lincoln Memorial in the background
- a vintage photo of a man with Lincoln Memorial in the background
- a vintage photo of a person holding a book with Lincoln Memorial in the background
- a vintage photo of a person standing in front of Lincoln Memorial
- a vintage photo of a person with Lincoln Memorial in the background
- a vintage photo of an old man standing in front of a book with Lincoln Memorial in the background
- an old photo of a cake with Lincoln Memorial in the background
- an old photo of a man with Lincoln Memorial in the background
- an old photo of a person with Lincoln Memorial in the background
- old photo of a person with Lincoln Memorial in the background
Luther King Jr. Memorial (3)
The entity Luther King Jr. Memorial is present in 3 descriptions.
- a statue of a person with Martin Luther King Jr. Memorial in the background
- a stone statue of a person with Martin Luther King Jr. Memorial in the background
- an old photo of a person with Martin Luther King Jr. Memorial in the background
Notre (6)
The entity Notre is present in 6 descriptions.
- a close up of a church with Notre Dame de Paris in the background
- a close up of an old building with Notre Dame de Paris in the background
- a close up of an old church with Notre Dame de Paris in the background
- an old photo of a large building with Notre Dame de Paris in the background
- an old photo of Notre Dame de Paris
- old photo of Notre Dame de Paris
Pergamon Museum (3)
The entity Pergamon Museum is present in 3 descriptions.
- a close up of a sign with Pergamon Museum in the background
- a sign on a wall with Pergamon Museum in the background
- close up of a sign with Pergamon Museum in the background
Robert Ford (3)
The entity Robert Ford is present in 3 descriptions.
- a vintage photo of Robert Ford et al. posing for a picture
- a vintage photo of Robert Ford et al. posing for the camera
- a vintage photo of Robert Ford et al. sitting posing for the camera
Salisbury Cathedral (3)
The entity Salisbury Cathedral is present in 3 descriptions.
- a vintage photo of a church with Salisbury Cathedral in the background
- a vintage photo of an old building with Salisbury Cathedral in the background
- a vintage photo of an old church with Salisbury Cathedral in the background
Trinity Church (6)
The entity Trinity Church is present in 6 descriptions.
- a close up of a tall building in Trinity Church
- a tall building in Trinity Church
- a vintage photo of a castle with Trinity Church in the background
- a vintage photo of an old building with Trinity Church in the background
- an old photo of a castle with Trinity Church in the background
- an old photo of Trinity Church
Wallace Ford (3)
The entity Wallace Ford is present in 3 descriptions.
- a vintage photo of Wallace Ford et al. posing for a photo
- a vintage photo of Wallace Ford et al. posing for a picture
- a vintage photo of Wallace Ford et al. posing for the camera
red cross (1)
The entity red cross is present in 1 descriptions.
- The image shows a metal can or container with a red cross symbol on it, and a screwdriver with a wooden handle.
1 Vocabularies are slow to compile so they are built about once a month. The current vocabulary was built on 2024-11-14.
2 Named entities are extracted using Compromise, a javascript NLP library. It's not exact, but pretty close.
