Statistics
Total Volume of Data
The database contains 53,422,082 publicly accessible machine-generated annotations covering 378,372 images.
Annotations date from 2018-02-08 to 2024-11-20.
Each annotation indicates an area of interest in an image. An area can be a small portion of an image or it can be the entire image. Small regions typically contain a human face or words. Annotations on the full image are typically tags (e.g. cat, watermelon, rock) and descriptions (e.g. a cow standing in a field).
Access to annotations as data is available via the annotation endpoint on the Harvard Art Museums API.
Sources
Annotations are generated from 5 primary computer vision services.
Each service supplies a variety of functions for detecting and categorizing features. These are the services and features we use.
- AWS Rekognition
- Labels, face detection, text detection
- Clarifai
- Concepts
- Google Vision
- Labels, face detection, text detection, landmark detection
- Imagga
- Tags, categories
- Microsoft Cognitive Sources
- Tags, categories, descriptions, face detection
Annotation counts by source.
The raw data.
| AWS Rekognition | Anthropic | Azure OpenAI Service | Clarifai | Google Vision | Imagga | Microsoft Cognitive Services | |
|---|---|---|---|---|---|---|---|
| category | 0 | 0 | 0 | 0 | 0 | 952,815 | 0 |
| description | 0 | 26,457 | 26,470 | 0 | 0 | 0 | 826,779 |
| face | 319,683 | 0 | 0 | 0 | 107,452 | 0 | 80,368 |
| tag | 4,047,557 | 0 | 0 | 7,443,514 | 4,156,674 | 26,868,279 | 2,333,905 |
| text | 2,858,549 | 0 | 0 | 0 | 3,373,198 | 0 | 0 |
| Total | 7,225,789 | 26,457 | 26,470 | 7,443,514 | 7,637,324 | 27,821,094 | 3,241,052 |
Annotation Types
Annotation types generally describe what the annotation contains or depicts. These are the possible types.
- category
- a broad categorization of the contents of the image
- description
- a full sentence caption of the contents of the image
- face
- a human face is found within the annotation
- tag
- a term or set of terms describing all or part of the image
- text
- some text is found within the annotation
Distribution of sources across annotation types.
The raw data.
| category | description | face | tag | text | |
|---|---|---|---|---|---|
| AWS Rekognition | 0 | 0 | 319,683 | 4,047,557 | 2,858,549 |
| Anthropic | 0 | 26,457 | 0 | 0 | 0 |
| Azure OpenAI Service | 0 | 26,470 | 0 | 0 | 0 |
| Clarifai | 0 | 0 | 0 | 7,443,514 | 0 |
| Google Vision | 0 | 0 | 107,452 | 4,156,674 | 3,373,198 |
| Imagga | 952,815 | 0 | 0 | 26,868,279 | 0 |
| Microsoft Cognitive Services | 0 | 826,779 | 80,368 | 2,333,905 | 0 |
| Total | 952,815 | 879,706 | 507,503 | 44,850,310 | 6,231,748 |
Vocabularies1
The tags and descriptions cluster and breakdown in to distinct terms, descriptive phrases, concepts, and named entities.
Terms
The vocabulary of tags contains 15,133 distinct terms.
The size of vocabulary by source.
The raw data.
| # of terms | |
|---|---|
| AWS Rekognition | 2,724 |
| Clarifai | 4,541 |
| Imagga | 9,582 |
| Google Vision | 8,149 |
| Microsoft Cognitive Services | 3,471 |
Sample set of terms:
- artichoke (36)
- carolina rose (1)
- coffeehouse (11)
- duel (1,372)
- formations (210)
- getting (52)
- home fencing (36)
- homemade (4,044)
- livestock carrier (1)
- lunar (13,309)
- megaphone (309)
- procedure (22)
- reptile (5,080)
- sedan (1,181)
- seppala siberian sleddog (1)
- ski jumping (1)
- teak (9)
- ward (3)
- wig (1,279)
- wise (61)
Descriptive Phrases
The vocabulary of descriptions contains 41,345 descriptive phrases.
Sample set of descriptive phrases:
- a baby sleeping in a dog bed (2)
- a couple of people that are looking at each other (1)
- a couple of young men standing next to a building (1)
- a green cake with a piece of luggage (1)
- a herd of cattle standing on top of a grass covered field (107)
- a horse drawn carriage traveling down a dirt road (14)
- a large blue vase sitting in a bowl (1)
- a person sitting on the side of a window (2)
- a person standing next to a blender (1)
- a person wearing a yellow shirt (1)
- a screenshot of a social media photo of a person (5)
- a view of a white wall (5)
- a white sign with black text in front of a store (3)
- a woman sitting in a car (1)
- a woman wearing a green dress standing in front of a curtain (1)
- an old photo of a dirty field (19)
- close up of a half moon in the dark (5)
- john l lewis frances perkins posing for a picture (1)
- romare bearden standing posing for the camera (1)
- sharon hayes et al standing in front of a sign (8)
Named Entities2
Descriptions can be parsed further in to clusters of named entities.
The vocabulary of named entities contains 1,754 people.
Sample set of people:
Arthur Hunnicutt (3)
The entity Arthur Hunnicutt is present in 3 descriptions.
- a vintage photo of Arthur Hunnicutt et al. posing for a photo
- a vintage photo of Arthur Hunnicutt et al. posing for a picture
- a vintage photo of Arthur Hunnicutt et al. posing for the camera
Benjamin Peirce (5)
The entity Benjamin Peirce is present in 5 descriptions.
- a vintage photo of Benjamin Peirce
- Benjamin Peirce posing for the camera
- Benjamin Peirce wearing a suit
- Benjamin Peirce wearing a suit and tie
- Benjamin Peirce wearing a suit and tie looking at the camera
David Eisenhower (3)
The entity David Eisenhower is present in 3 descriptions.
- David Eisenhower, Richard Nixon, Julie Nixon Eisenhower, Tricia Nixon Cox, Pat Nixon, Edward Cox posing for a photo
- David Eisenhower, Richard Nixon, Julie Nixon Eisenhower, Tricia Nixon Cox, Pat Nixon, Edward Cox posing for the camera
- David Eisenhower, Richard Nixon, Julie Nixon Eisenhower, Tricia Nixon Cox, Pat Nixon, Edward Cox sitting posing for the camera
Deborah Kerr (6)
The entity Deborah Kerr is present in 6 descriptions.
- Deborah Kerr and woman sitting next to a window
- Deborah Kerr and woman sitting on a bench
- Deborah Kerr et al. sitting on a bench
- Peter Viertel, Deborah Kerr are posing for a picture
- Peter Viertel, Deborah Kerr sitting at a table
- Peter Viertel, Deborah Kerr sitting on a table
Elisabeth Rethberg (3)
The entity Elisabeth Rethberg is present in 3 descriptions.
- a vintage photo of Elisabeth Rethberg and woman posing for a picture
- a vintage photo of Elisabeth Rethberg et al. posing for a picture
- a vintage photo of Elisabeth Rethberg et al. posing for the camera
George Henry Williams (2)
The entity George Henry Williams is present in 2 descriptions.
- a vintage photo of George Henry Williams
- an old photo of George Henry Williams
George Ward Hunt (4)
The entity George Ward Hunt is present in 4 descriptions.
- a book with a picture of George Ward Hunt
- a close up of George Ward Hunt holding a book
- an old photo of George Ward Hunt
- an old photo of George Ward Hunt holding a book
Henry Wilson (18)
The entity Henry Wilson is present in 18 descriptions.
- a black and white photo of Henry Wilson
- a vintage photo of Henry Wilson
- a vintage photo of Henry Wilson holding a book
- a vintage photo of Henry Wilson holding a sign
- a vintage photo of Henry Wilson holding a sign posing for the camera
- a vintage photo of Henry Wilson in a suit
- a vintage photo of Henry Wilson in a suit and tie
- a vintage photo of Henry Wilson in a suit posing for a picture
- a vintage photo of Henry Wilson in a suit posing for the camera
- a vintage photo of Henry Wilson wearing a suit and tie
- an old photo of Henry Wilson
- Henry Wilson holding a book
- Henry Wilson holding a book posing for the camera
- Henry Wilson holding a sign
- Henry Wilson posing for a photo
- Henry Wilson standing in front of a mirror posing for the camera
- Henry Wilson standing next to a book
- old photo of Henry Wilson
John Milton (2)
The entity John Milton is present in 2 descriptions.
- a black and white photo of John Milton
- an old photo of John Milton
Katharine Hepburn (3)
The entity Katharine Hepburn is present in 3 descriptions.
- Katharine Hepburn in a white shirt
- Katharine Hepburn in a white shirt and black hair
- Katharine Hepburn looking at the camera
Louis Lumiere (3)
The entity Louis Lumiere is present in 3 descriptions.
- a vintage photo of Auguste and Louis Lumiere, Maud Wood Park posing for a picture
- a vintage photo of Auguste and Louis Lumiere, Maud Wood Park posing for the camera
- a vintage photo of Auguste and Louis Lumiere, Maud Wood Park sitting posing for the camera
Mahatma Gandhi (2)
The entity Mahatma Gandhi is present in 2 descriptions.
- Mahatma Gandhi posing for a picture
- Mahatma Gandhi posing for the camera
Maisie Williams (3)
The entity Maisie Williams is present in 3 descriptions.
- Maisie Williams standing in a dark room
- Maisie Williams standing in front of a door
- Maisie Williams standing in the dark
Pedro Calderon de la Barca (3)
The entity Pedro Calderon de la Barca is present in 3 descriptions.
- a screen shot of Pedro Calderon de la Barca
- a screen shot of Pedro Calderon de la Barca with a beard looking at the camera
- Pedro Calderon de la Barca looking at the camera
Richard Hart (3)
The entity Richard Hart is present in 3 descriptions.
- Richard Hart et al. posing for a photo
- Richard Hart et al. posing for a picture
- Richard Hart et al. posing for the camera
Rufus King (3)
The entity Rufus King is present in 3 descriptions.
- a black and white photo of Rufus King
- a vintage photo of Rufus King holding a book
- an old black and white photo of Rufus King
Samuel Pepys (5)
The entity Samuel Pepys is present in 5 descriptions.
- an old photo of Samuel Pepys
- old photo of Samuel Pepys
- Samuel Pepys in a newspaper
- Samuel Pepys sitting in a newspaper
- Samuel Pepys sitting on a newspaper
Theo Sommer (3)
The entity Theo Sommer is present in 3 descriptions.
- Theo Sommer sitting at a desk
- Theo Sommer sitting on a desk
- Theo Sommer sitting on a table
Walter Seymour Allward (2)
The entity Walter Seymour Allward is present in 2 descriptions.
- Walter Seymour Allward et al. in uniform posing for a photo
- Walter Seymour Allward et al. posing for a photo
Wendell Phillips (6)
The entity Wendell Phillips is present in 6 descriptions.
- a black and white photo of Wendell Phillips
- a vintage photo of Wendell Phillips
- a vintage photo of Wendell Phillips holding a book
- a vintage photo of Wendell Phillips in a suit and tie
- an old photo of Wendell Phillips holding a book
- Wendell Phillips holding a book
The vocabulary of named entities contains 54 places.
Sample set of places:
289 Washington St. (1)
The entity 289 Washington St. is present in 1 descriptions.
- The image appears to be a vintage business card or calling card for a photographer named S. Masury, who was a "Photographic Artist" located at 289 Washington St. in Boston.
Belgium (10)
The entity Belgium is present in 10 descriptions.
- a close up of Leopold II of Belgium
- a vintage photo of Leopold II of Belgium
- a vintage photo of Leopold II of Belgium holding a book
- a vintage photo of Leopold II of Belgium standing in front of a book
- an old photo of Leopold II of Belgium
- Leopold II of Belgium holding a book
- Leopold II of Belgium posing for a photo
- Leopold II of Belgium standing in front of a book
- Leopold II of Belgium taking a selfie
- old photo of Leopold II of Belgium
Boston. (1)
The entity Boston. is present in 1 descriptions.
- The image appears to be a vintage business card or calling card for a photographer named S. Masury, who was a "Photographic Artist" located at 289 Washington St. in Boston.
Egypt (2)
The entity Egypt is present in 2 descriptions.
- Abbas II of Egypt in a box
- Abbas II of Egypt sitting in a box
England (1)
The entity England is present in 1 descriptions.
- an old photo of Henry III of England
France (6)
The entity France is present in 6 descriptions.
- a vintage photo of Louis XIV of France
- a vintage photo of Louis XIV of France holding a book
- an old photo of Louis XIV of France
- an old photo of Louis XVI of France
- an old photo of Louis XVI of France sitting on a bed
- Louis XVI of France sitting on a bed
Georgia (3)
The entity Georgia is present in 3 descriptions.
- an old photo of Georgia O'Keeffe
- Georgia O'Keeffe looking at the camera
- the face of Georgia O'Keeffe
Greece (15)
The entity Greece is present in 15 descriptions.
- a close up of George I of Greece holding a sign
- a old photo of George I of Greece
- a vintage photo of George I of Greece holding a book
- a vintage photo of George I of Greece in a suit and tie
- a vintage photo of George I of Greece wearing a suit and tie
- an old photo of George I of Greece
- an old photo of Princess Cecilie of Greece and Denmark
- an old photo of Princess Cecilie of Greece and Denmark and woman posing for a picture
- George I of Greece holding a sign
- George I of Greece standing in front of a sign
- old photo of George I of Greece
- old photo of Princess Cecilie of Greece and Denmark et al. standing in front of a window
- Princess Cecilie of Greece and Denmark, Bobby Jordan posing for a photo
- Princess Cecilie of Greece and Denmark, Bobby Jordan posing for a picture
- Princess Cecilie of Greece and Denmark, Bobby Jordan posing for the camera
Hawaii (4)
The entity Hawaii is present in 4 descriptions.
- a vintage photo of Queen Emma of Hawaii and woman posing for a picture
- a vintage photo of Queen Emma of Hawaii et al. posing for a picture
- a vintage photo of Queen Emma of Hawaii et al. posing for the camera
- a vintage photo of Queen Emma of Hawaii holding a book
Ho Chi Minh (3)
The entity Ho Chi Minh is present in 3 descriptions.
- a vintage photo of Ho Chi Minh
- a vintage photo of Ho Chi Minh et al. posing for the camera
- a vintage photo of Ho Chi Minh in a newspaper
Milan (3)
The entity Milan is present in 3 descriptions.
- an old photo of a large building with Milan Cathedral in the background
- an old photo of Milan Cathedral
- an old photo of Milan Cathedral street
Montenegro (5)
The entity Montenegro is present in 5 descriptions.
- a vintage photo of Elena of Montenegro
- a vintage photo of Elena of Montenegro et al. posing for a photo
- a vintage photo of Elena of Montenegro et al. posing for a picture
- a vintage photo of Elena of Montenegro et al. posing for the camera
- an old photo of Elena of Montenegro
Oregon Trail (3)
The entity Oregon Trail is present in 3 descriptions.
- a vintage photo of a horse with Oregon Trail in the background
- a vintage photo of a person on a horse with Oregon Trail in the background
- a vintage photo of a person riding a horse with Oregon Trail in the background
Rio (3)
The entity Rio is present in 3 descriptions.
- Rio Reiser sitting in a chair
- Rio Reiser wearing a black hat
- Rio Reiser wearing a hat
Rome (12)
The entity Rome is present in 12 descriptions.
- a close up of an old building with Pantheon, Rome in the background
- a vintage photo of a busy city street with Pantheon, Rome in the background
- a vintage photo of a church with Pantheon, Rome in the background
- a vintage photo of a horse drawn carriage on Pantheon, Rome street
- a vintage photo of an old building in the background with Pantheon, Rome in the background
- a vintage photo of an old building with Pantheon, Rome in the background
- a vintage photo of an old church with Pantheon, Rome in the background
- a vintage photo of an old stone building with Pantheon, Rome in the background
- a vintage photo of Pantheon, Rome
- a vintage photo of Pantheon, Rome street
- an old photo of a large building with Pantheon, Rome in the background
- an old photo of Pantheon, Rome
Ross Hill (3)
The entity Ross Hill is present in 3 descriptions.
- a black and white photo of Ross Hill
- a vintage photo of Ross Hill
- Ross Hill sitting in front of a window
Texas (3)
The entity Texas is present in 3 descriptions.
- a black and white photo of Texas Jack Omohundro wearing a suit and tie
- a vintage photo of Texas Jack Omohundro wearing a suit and tie
- Texas Jack Omohundro wearing a suit and tie
United Kingdom (5)
The entity United Kingdom is present in 5 descriptions.
- a vintage photo of Princess Alice of the United Kingdom holding a book
- a vintage photo of Princess Alice of the United Kingdom holding a box posing for the camera
- Princess Alice of the United Kingdom et al. posing for a photo in front of a window
- Princess Alice of the United Kingdom et al. standing in front of a mirror posing for the camera
- Princess Alice of the United Kingdom et al. standing in front of a window
Virginia (5)
The entity Virginia is present in 5 descriptions.
- Virginia Mayo in front of a mirror posing for the camera
- Virginia Mayo standing in front of a mirror posing for the camera
- Virginia Weidler et al. posing for a photo
- Virginia Weidler et al. posing for a picture
- Virginia Weidler et al. posing for the camera
london (2)
The entity london is present in 2 descriptions.
- a large clock tower towering over the city of london
- a tall clock tower towering over the city of london
The vocabulary of named entities contains 11 organizations.
Sample set of organizations:
Andy Warhol Museum (1)
The entity Andy Warhol Museum is present in 1 descriptions.
- a close up of a book with The Andy Warhol Museum in the background
Bruton Parish Church (3)
The entity Bruton Parish Church is present in 3 descriptions.
- a house that has a sign on the side of Bruton Parish Church
- a sign in front of a house with Bruton Parish Church in the background
- a sign on the side of a house with Bruton Parish Church in the background
Lincoln Memorial (21)
The entity Lincoln Memorial is present in 21 descriptions.
- a group of people posing for a photo with Lincoln Memorial in the background
- a group of people posing for a picture with Lincoln Memorial in the background
- a group of people posing for the camera with Lincoln Memorial in the background
- a man and a woman standing in front of a window with Lincoln Memorial in the background
- a man sitting on a bench with Lincoln Memorial in the background
- a man standing in front of a mirror with Lincoln Memorial in the background
- a man standing in front of a window with Lincoln Memorial in the background
- a vintage photo of a group of people posing for a picture with Lincoln Memorial in the background
- a vintage photo of a group of people posing for the camera with Lincoln Memorial in the background
- a vintage photo of a man holding a book with Lincoln Memorial in the background
- a vintage photo of a man sitting on a bench with Lincoln Memorial in the background
- a vintage photo of a man standing in front of a book with Lincoln Memorial in the background
- a vintage photo of a man with Lincoln Memorial in the background
- a vintage photo of a person holding a book with Lincoln Memorial in the background
- a vintage photo of a person standing in front of Lincoln Memorial
- a vintage photo of a person with Lincoln Memorial in the background
- a vintage photo of an old man standing in front of a book with Lincoln Memorial in the background
- an old photo of a cake with Lincoln Memorial in the background
- an old photo of a man with Lincoln Memorial in the background
- an old photo of a person with Lincoln Memorial in the background
- old photo of a person with Lincoln Memorial in the background
Luther King Jr. Memorial (3)
The entity Luther King Jr. Memorial is present in 3 descriptions.
- a statue of a person with Martin Luther King Jr. Memorial in the background
- a stone statue of a person with Martin Luther King Jr. Memorial in the background
- an old photo of a person with Martin Luther King Jr. Memorial in the background
Notre (6)
The entity Notre is present in 6 descriptions.
- a close up of a church with Notre Dame de Paris in the background
- a close up of an old building with Notre Dame de Paris in the background
- a close up of an old church with Notre Dame de Paris in the background
- an old photo of a large building with Notre Dame de Paris in the background
- an old photo of Notre Dame de Paris
- old photo of Notre Dame de Paris
Pergamon Museum (3)
The entity Pergamon Museum is present in 3 descriptions.
- a close up of a sign with Pergamon Museum in the background
- a sign on a wall with Pergamon Museum in the background
- close up of a sign with Pergamon Museum in the background
Robert Ford (3)
The entity Robert Ford is present in 3 descriptions.
- a vintage photo of Robert Ford et al. posing for a picture
- a vintage photo of Robert Ford et al. posing for the camera
- a vintage photo of Robert Ford et al. sitting posing for the camera
Salisbury Cathedral (3)
The entity Salisbury Cathedral is present in 3 descriptions.
- a vintage photo of a church with Salisbury Cathedral in the background
- a vintage photo of an old building with Salisbury Cathedral in the background
- a vintage photo of an old church with Salisbury Cathedral in the background
Trinity Church (6)
The entity Trinity Church is present in 6 descriptions.
- a close up of a tall building in Trinity Church
- a tall building in Trinity Church
- a vintage photo of a castle with Trinity Church in the background
- a vintage photo of an old building with Trinity Church in the background
- an old photo of a castle with Trinity Church in the background
- an old photo of Trinity Church
Wallace Ford (3)
The entity Wallace Ford is present in 3 descriptions.
- a vintage photo of Wallace Ford et al. posing for a photo
- a vintage photo of Wallace Ford et al. posing for a picture
- a vintage photo of Wallace Ford et al. posing for the camera
red cross (1)
The entity red cross is present in 1 descriptions.
- The image shows a metal can or container with a red cross symbol on it, and a screwdriver with a wooden handle.
1 Vocabularies are slow to compile so they are built about once a month. The current vocabulary was built on 2024-11-14.
2 Named entities are extracted using Compromise, a javascript NLP library. It's not exact, but pretty close.
