Statistics

Total Volume of Data

The database contains 53,298,779 publicly accessible machine-generated annotations covering 377,449 images.

Annotations date from 2018-02-08 to 2024-04-12.

Each annotation indicates an area of interest in an image. An area can be a small portion of an image or it can be the entire image. Small regions typically contain a human face or words. Annotations on the full image are typically tags (e.g. cat, watermelon, rock) and descriptions (e.g. a cow standing in a field).

Access to annotations as data is available via the annotation endpoint on the Harvard Art Museums API.

Sources

Annotations are generated from 5 primary computer vision services.

Each service supplies a variety of functions for detecting and categorizing features. These are the services and features we use.

AWS Rekognition
Labels, face detection, text detection
Clarifai
Concepts
Google Vision
Labels, face detection, text detection, landmark detection
Imagga
Tags, categories
Microsoft Cognitive Sources
Tags, categories, descriptions, face detection

Annotation counts by source.

The raw data.

AWS Rekognition Anthropic Azure OpenAI Service Clarifai Google Vision Imagga Microsoft Cognitive Services
category 0 0 0 0 0 951,433 0
description 0 10,605 10,639 0 0 0 826,345
face 319,146 0 0 0 107,193 0 80,288
tag 4,038,804 0 0 7,433,234 4,147,338 26,825,047 2,330,890
text 2,853,942 0 0 0 3,363,493 0 0
Total 7,211,892 10,605 10,639 7,433,234 7,618,024 27,776,480 3,237,523

Annotation Types

Annotation types generally describe what the annotation contains or depicts. These are the possible types.

category
a broad categorization of the contents of the image
description
a full sentence caption of the contents of the image
face
a human face is found within the annotation
tag
a term or set of terms describing all or part of the image
text
some text is found within the annotation

Distribution of sources across annotation types.

The raw data.

category description face tag text
AWS Rekognition 0 0 319,146 4,038,804 2,853,942
Anthropic 0 10,605 0 0 0
Azure OpenAI Service 0 10,639 0 0 0
Clarifai 0 0 0 7,433,234 0
Google Vision 0 0 107,193 4,147,338 3,363,493
Imagga 951,433 0 0 26,825,047 0
Microsoft Cognitive Services 0 826,345 80,288 2,330,890 0
Total 951,433 847,589 506,627 44,775,694 6,217,436

Vocabularies1

The tags and descriptions cluster and breakdown in to distinct terms, descriptive phrases, concepts, and named entities.

Terms

The vocabulary of tags contains 15,125 distinct terms.

The size of vocabulary by source.

The raw data.

# of terms
AWS Rekognition 2,721
Clarifai 4,539
Imagga 9,578
Google Vision 8,147
Microsoft Cognitive Services 3,456

Sample set of terms:

Descriptive Phrases

The vocabulary of descriptions contains 41,085 descriptive phrases.

Sample set of descriptive phrases:

Named Entities2

Descriptions can be parsed further in to clusters of named entities.

The vocabulary of named entities contains 1,753 people.

Sample set of people:

Abram Garfields (2)

The entity Abram Garfields is present in 2 descriptions.

  • a close up of Abram Garfields face
  • close up of Abram Garfields face
Ali Sami Yen (3)

The entity Ali Sami Yen is present in 3 descriptions.

  • a vintage photo of Ali Sami Yen
  • a vintage photo of Ali Sami Yen in a suit and tie
  • a vintage photo of Ali Sami Yen wearing a suit and tie
Arthur Goldberg (3)

The entity Arthur Goldberg is present in 3 descriptions.

  • a man standing in front of Arthur Goldberg, David Dubinsky posing for a photo
  • Arthur Goldberg, David Dubinsky posing for a photo
  • Arthur Goldberg, David Dubinsky posing for the camera
Basil W. Duke (3)

The entity Basil W. Duke is present in 3 descriptions.

  • a black and white photo of Basil W. Duke
  • a vintage photo of Basil W. Duke
  • an old black and white photo of Basil W. Duke
Billy Bowlegs (3)

The entity Billy Bowlegs is present in 3 descriptions.

  • an old photo of Billy Bowlegs
  • an old photo of Billy Bowlegs holding a book
  • old photo of Billy Bowlegs
Clarence Brown (3)

The entity Clarence Brown is present in 3 descriptions.

  • an old photo of Clarence Brown et al. posing for the camera
  • Clarence Brown et al. posing for a photo
  • Clarence Brown et al. sitting posing for the camera
Errol Flynn (5)

The entity Errol Flynn is present in 5 descriptions.

  • a vintage photo of Errol Flynn
  • a vintage photo of Errol Flynn holding a book
  • an old photo of Errol Flynn
  • Errol Flynn posing for a photo
  • Errol Flynn posing for the camera
George Law (6)

The entity George Law is present in 6 descriptions.

  • a black and white photo of George Law
  • a black and white photo of George Law in a suit and tie
  • a vintage photo of George Law in a suit and tie
  • an old black and white photo of George Law
  • George Law standing in front of a door
  • George Law standing in front of a mirror posing for the camera
Henry Aldrich (3)

The entity Henry Aldrich is present in 3 descriptions.

  • an old photo of Henry Aldrich
  • Henry Aldrich looking at a book
  • old photo of Henry Aldrich
Henry Wadsworth Longfellow (6)

The entity Henry Wadsworth Longfellow is present in 6 descriptions.

  • a vintage photo of Henry Wadsworth Longfellow et al. posing for a picture
  • a vintage photo of Henry Wadsworth Longfellow standing in front of a door
  • a vintage photo of Henry Wadsworth Longfellow standing posing for the camera
  • Henry Wadsworth Longfellow wearing a suit
  • Henry Wadsworth Longfellow wearing a suit and tie
  • Henry Wadsworth Longfellow wearing a suit and tie looking at the camera
John Houston Savage (2)

The entity John Houston Savage is present in 2 descriptions.

  • a vintage photo of John Houston Savage
  • a vintage photo of John Houston Savage holding a book
John Van Buren (9)

The entity John Van Buren is present in 9 descriptions.

  • a vintage photo of John Van Buren
  • a vintage photo of John Van Buren holding a book
  • a vintage photo of John Van Buren in a suit and tie
  • a vintage photo of John Van Buren wearing a suit and tie
  • a vintage photo of John Van Buren with an open door
  • an old photo of John Van Buren
  • an old photo of John Van Buren in a suit and tie
  • an old photo of John Van Buren wearing a suit and tie
  • John Van Buren in a suit and tie
Lizzie Borden (4)

The entity Lizzie Borden is present in 4 descriptions.

  • a black and white photo of Lizzie Borden
  • a vintage photo of Lizzie Borden
  • a vintage photo of Lizzie Borden wearing a suit and tie
  • an old black and white photo of Lizzie Borden
Mark Sykes (3)

The entity Mark Sykes is present in 3 descriptions.

  • Mark Sykes wearing a suit and tie
  • Mark Sykes wearing a suit and tie posing for a photo
  • Mark Sykes wearing a suit and tie posing for a picture
Nicholas II (10)

The entity Nicholas II is present in 10 descriptions.

  • a black and white photo of Nicholas II of Russia
  • a vintage photo of Nicholas II of Russia
  • Nicholas II of Russia, Alexandra Feodorovna posing for a photo
  • Nicholas II of Russia, Alexandra Feodorovna posing for a picture
  • Nicholas II of Russia, Alexandra Feodorovna posing for the camera
  • Nicholas II of Russia et al. posing for a photo
  • Nicholas II of Russia et al. posing for the camera
  • Nicholas II of Russia holding a box posing for the camera
  • Nicholas II of Russia holding a sign posing for the camera
  • Nicholas II of Russia standing in front of a mirror posing for the camera
Richard Cobden (11)

The entity Richard Cobden is present in 11 descriptions.

  • a vintage photo of Richard Cobden
  • a vintage photo of Richard Cobden et al. sitting in a box
  • a vintage photo of Richard Cobden holding a book
  • a vintage photo of Richard Cobden in a suit and tie
  • a vintage photo of Richard Cobden sitting in a box
  • a vintage photo of Richard Cobden sitting on a book
  • a vintage photo of Richard Cobden sitting on top of a book
  • a vintage photo of Richard Cobden wearing a suit and tie
  • an old photo of Richard Cobden in a suit and tie
  • old photo of Richard Cobden in a suit and tie
  • Richard Cobden wearing a suit and tie
Richard Throssel (2)

The entity Richard Throssel is present in 2 descriptions.

  • a vintage photo of Richard Throssel
  • a vintage photo of Richard Throssel holding a book
Tony Ward (1)

The entity Tony Ward is present in 1 descriptions.

  • a cat sitting on a statue of Tony Ward
William Hayden English (16)

The entity William Hayden English is present in 16 descriptions.

  • a black and white photo of William Hayden English
  • a close up of William Hayden English holding a book
  • a vintage photo of William Hayden English wearing a suit and tie
  • an old photo of William Hayden English
  • an old photo of William Hayden English in a suit and tie
  • an old photo of William Hayden English wearing a suit and tie
  • close up of William Hayden English holding a book
  • old photo of William Hayden English
  • old photo of William Hayden English in a suit and tie
  • William Hayden English holding a book
  • William Hayden English holding a book posing for the camera
  • William Hayden English holding a sign posing for the camera
  • William Hayden English in a suit and tie
  • William Hayden English in a suit standing in front of a sign
  • William Hayden English posing for a photo in front of a book
  • William Hayden English wearing a suit and tie
Zhou (3)

The entity Zhou is present in 3 descriptions.

  • a black and white photo of Zhou Enlai
  • a vintage photo of Zhou Enlai
  • an old black and white photo of Zhou Enlai

The vocabulary of named entities contains 52 places.

Sample set of places:

Albania (5)

The entity Albania is present in 5 descriptions.

  • a vintage photo of Zog I of Albania et al. posing for a picture
  • a vintage photo of Zog I of Albania et al. posing for the camera
  • a vintage photo of Zog I of Albania standing in front of a building
  • Zog I of Albania and woman posing for a photo
  • Zog I of Albania et al. posing for a photo
Austria (18)

The entity Austria is present in 18 descriptions.

  • a painting of Archduke Ferdinand Karl of Austria
  • a painting of Archduke Ferdinand Karl of Austria holding a sign
  • a vintage photo of Archduchess Maria Theresa of Austria
  • a vintage photo of Archduke Franz Ferdinand of Austria
  • a vintage photo of Archduke Franz Ferdinand of Austria holding a book
  • a vintage photo of Archduke Franz Ferdinand of Austria standing in front of a building
  • a vintage photo of Archduke Franz Ferdinand of Austria wearing a black dress
  • a vintage photo of Empress Elisabeth of Austria
  • a vintage photo of Empress Elisabeth of Austria holding a book
  • a vintage photo of Empress Elisabeth of Austria with a book
  • an old black and white photo of Archduke Franz Ferdinand of Austria
  • an old photo of Empress Elisabeth of Austria
  • Archduchess Maria Theresa of Austria posing for a photo
  • Archduchess Maria Theresa of Austria posing for the camera
  • Archduke Ferdinand Karl of Austria holding a sign posing for the camera
  • Empress Elisabeth of Austria posing for a photo
  • Empress Elisabeth of Austria posing for a photo in front of a book
  • old photo of Empress Elisabeth of Austria
Bavaria (25)

The entity Bavaria is present in 25 descriptions.

  • a black and white photo of King Ludwig II of Bavaria
  • a black and white photo of King Ludwig II of Bavaria holding a book
  • a screen shot of a photo of King Ludwig II of Bavaria
  • a screen shot of King Ludwig II of Bavaria
  • a screen shot of King Ludwig II of Bavaria in a suit and tie
  • a vintage photo of King Ludwig II of Bavaria
  • a vintage photo of King Ludwig II of Bavaria and woman posing for a picture
  • a vintage photo of King Ludwig II of Bavaria et al. posing for a picture
  • a vintage photo of King Ludwig II of Bavaria holding a book
  • a vintage photo of King Ludwig II of Bavaria in a suit and tie
  • a vintage photo of King Ludwig II of Bavaria in a white shirt and black text
  • a vintage photo of King Ludwig II of Bavaria making a face for the camera
  • an old photo of King Ludwig II of Bavaria
  • an old photo of King Ludwig II of Bavaria holding a book
  • an old photo of King Ludwig II of Bavaria in a suit and tie
  • an old photo of Otto of Bavaria
  • King Ludwig II of Bavaria, Pope Pius IX are posing for a picture
  • King Ludwig II of Bavaria, Pope Pius IX posing for a photo
  • King Ludwig II of Bavaria posing for a photo
  • King Ludwig II of Bavaria posing for a photo in front of a book
  • King Ludwig II of Bavaria posing for the camera
  • King Ludwig II of Bavaria wearing a suit and tie
  • old photo of King Ludwig II of Bavaria
  • Otto of Bavaria posing for a photo
  • Otto of Bavaria posing for a picture
California (3)

The entity California is present in 3 descriptions.

  • J. California Cooper et al. posing for a photo
  • J. California Cooper et al. sitting on a bench posing for the camera
  • J. California Cooper et al. sitting posing for the camera
Cleveland (11)

The entity Cleveland is present in 11 descriptions.

  • a black and white photo of Frances Folsom Cleveland Preston
  • a close up of Frances Folsom Cleveland Preston holding a sign
  • a vintage photo of Frances Folsom Cleveland Preston
  • a vintage photo of Frances Folsom Cleveland Preston et al. posing for a photo
  • a vintage photo of Frances Folsom Cleveland Preston et al. posing for a picture
  • a vintage photo of Frances Folsom Cleveland Preston et al. posing for the camera
  • a vintage photo of Frances Folsom Cleveland Preston et al. sitting posing for the camera
  • an old photo of Frances Folsom Cleveland Preston
  • Frances Folsom Cleveland Preston and woman posing for a photo
  • Frances Folsom Cleveland Preston et al. posing for a photo
  • Frances Folsom Cleveland Preston posing for the camera
Egypt (2)

The entity Egypt is present in 2 descriptions.

  • Abbas II of Egypt in a box
  • Abbas II of Egypt sitting in a box
England (1)

The entity England is present in 1 descriptions.

  • an old photo of Henry III of England
George Hill (3)

The entity George Hill is present in 3 descriptions.

  • George Hill Hodel sitting in uniform
  • George Hill Hodel sitting on a bed
  • George Hill Hodel sitting on a table
Italy (3)

The entity Italy is present in 3 descriptions.

  • a black and white photo of Victor Emmanuel III of Italy
  • a vintage photo of Victor Emmanuel III of Italy
  • a vintage photo of Victor Emmanuel III of Italy in a black shirt
Jerusalem (3)

The entity Jerusalem is present in 3 descriptions.

  • a close up of a book cover with Temple in Jerusalem in the background
  • a close up of a book with Temple in Jerusalem in the background
  • close up of a book with Temple in Jerusalem in the background
Joe Hill (3)

The entity Joe Hill is present in 3 descriptions.

  • a black and white photo of Joe Hill
  • a vintage photo of Joe Hill in a suit and tie
  • a vintage photo of Joe Hill wearing a suit and tie
London (8)

The entity London is present in 8 descriptions.

  • a group of people at a beach with London Bridge in the background
  • a group of people on a beach with London Bridge in the background
  • a vintage photo of Charmian London
  • a vintage photo of Charmian London and a child posing for the camera
  • a vintage photo of Charmian London et al. posing for a photo
  • a vintage photo of Charmian London et al. posing for a picture
  • a vintage photo of Charmian London et al. posing for the camera
  • an old photo of a bird with London Bridge in the background
New Mexico (9)

The entity New Mexico is present in 9 descriptions.

  • a close up of an orange with Ohkay Owingeh, New Mexico in the background
  • a piece of bread with Ohkay Owingeh, New Mexico in the background
  • a piece of cake on a table with Ohkay Owingeh, New Mexico in the background
  • a piece of food on a table with Ohkay Owingeh, New Mexico in the background
  • a piece of food with Ohkay Owingeh, New Mexico in the background
  • an orange cut in half and sitting on a table with Ohkay Owingeh, New Mexico in the background
  • an orange cut in half on a plate with Ohkay Owingeh, New Mexico in the background
  • an orange cut in half with Ohkay Owingeh, New Mexico in the background
  • an orange sitting on a table with Ohkay Owingeh, New Mexico in the background
Romania (19)

The entity Romania is present in 19 descriptions.

  • a vintage photo of Carol I of Romania holding a book
  • a vintage photo of Carol I of Romania holding a book posing for the camera
  • a vintage photo of Carol I of Romania standing in front of a book
  • a vintage photo of Carol I of Romania wearing a suit and tie
  • a vintage photo of Ferdinand I of Romania
  • a vintage photo of Marie of Romania et al. posing for the camera
  • a vintage photo of Marie of Romania et al. sitting in a box
  • a vintage photo of Marie of Romania, Ferdinand I of Romania posing for a picture
  • a vintage photo of Marie of Romania, Ferdinand I of Romania posing for a picture
  • a vintage photo of Marie of Romania sitting in a box
  • an old photo of Elisabeth of Romania
  • Elisabeth of Romania posing for a photo
  • Elisabeth of Romania posing for a picture
  • Marie of Romania, Ferdinand I of Romania are posing for a picture
  • Marie of Romania, Ferdinand I of Romania are posing for a picture
  • Marie of Romania in a white box
  • Marie of Romania posing for a photo
  • Marie of Romania sitting in a box
  • Marie of Romania with collar shirt
Rome (12)

The entity Rome is present in 12 descriptions.

  • a close up of an old building with Pantheon, Rome in the background
  • a vintage photo of a busy city street with Pantheon, Rome in the background
  • a vintage photo of a church with Pantheon, Rome in the background
  • a vintage photo of a horse drawn carriage on Pantheon, Rome street
  • a vintage photo of an old building in the background with Pantheon, Rome in the background
  • a vintage photo of an old building with Pantheon, Rome in the background
  • a vintage photo of an old church with Pantheon, Rome in the background
  • a vintage photo of an old stone building with Pantheon, Rome in the background
  • a vintage photo of Pantheon, Rome
  • a vintage photo of Pantheon, Rome street
  • an old photo of a large building with Pantheon, Rome in the background
  • an old photo of Pantheon, Rome
Serbia (3)

The entity Serbia is present in 3 descriptions.

  • Alexander I of Serbia holding a sign
  • Alexander I of Serbia posing for a photo
  • an old photo of Alexander I of Serbia
St (6)

The entity St is present in 6 descriptions.

  • a close up of a church with St Mark's Basilica in the background
  • a group of people riding on the back of a church with St Mark's Basilica in the background
  • a large building with St Mark's Basilica in the background
  • a large old building with many windows with St Mark's Basilica in the background
  • an old church with St Mark's Basilica in the background
  • an old photo of a church with St Mark's Basilica in the background
Sweden (5)

The entity Sweden is present in 5 descriptions.

  • a vintage photo of Oscar II of Sweden
  • a vintage photo of Oscar II of Sweden in a white shirt and black text
  • a vintage photo of Oscar II of Sweden in white shirt and black text
  • an old photo of Oscar II of Sweden
  • old photo of Oscar II of Sweden
United Kingdom (5)

The entity United Kingdom is present in 5 descriptions.

  • a vintage photo of Princess Alice of the United Kingdom holding a book
  • a vintage photo of Princess Alice of the United Kingdom holding a box posing for the camera
  • Princess Alice of the United Kingdom et al. posing for a photo in front of a window
  • Princess Alice of the United Kingdom et al. standing in front of a mirror posing for the camera
  • Princess Alice of the United Kingdom et al. standing in front of a window
Washington (14)

The entity Washington is present in 14 descriptions.

  • a black and white photo of Washington Irving
  • a large statue in front of Washington Square Park
  • a large stone statue in front of Washington Square Park
  • a statue in front of Washington Square Park
  • a vintage photo of Washington Irving
  • a vintage photo of Washington Irving holding a book
  • a vintage photo of Washington Irving in a suit and tie
  • a vintage photo of Washington Irving in a white shirt and black text
  • a vintage photo of Washington Irving wearing a suit and tie
  • an old black and white photo of Washington Irving
  • an old photo of Washington Irving
  • Washington Irving posing for a photo
  • Washington Irving posing for a picture
  • Washington Irving wearing a suit and tie

The vocabulary of named entities contains 10 organizations.

Sample set of organizations:

Andy Warhol Museum (1)

The entity Andy Warhol Museum is present in 1 descriptions.

  • a close up of a book with The Andy Warhol Museum in the background
Bruton Parish Church (3)

The entity Bruton Parish Church is present in 3 descriptions.

  • a house that has a sign on the side of Bruton Parish Church
  • a sign in front of a house with Bruton Parish Church in the background
  • a sign on the side of a house with Bruton Parish Church in the background
Lincoln Memorial (21)

The entity Lincoln Memorial is present in 21 descriptions.

  • a group of people posing for a photo with Lincoln Memorial in the background
  • a group of people posing for a picture with Lincoln Memorial in the background
  • a group of people posing for the camera with Lincoln Memorial in the background
  • a man and a woman standing in front of a window with Lincoln Memorial in the background
  • a man sitting on a bench with Lincoln Memorial in the background
  • a man standing in front of a mirror with Lincoln Memorial in the background
  • a man standing in front of a window with Lincoln Memorial in the background
  • a vintage photo of a group of people posing for a picture with Lincoln Memorial in the background
  • a vintage photo of a group of people posing for the camera with Lincoln Memorial in the background
  • a vintage photo of a man holding a book with Lincoln Memorial in the background
  • a vintage photo of a man sitting on a bench with Lincoln Memorial in the background
  • a vintage photo of a man standing in front of a book with Lincoln Memorial in the background
  • a vintage photo of a man with Lincoln Memorial in the background
  • a vintage photo of a person holding a book with Lincoln Memorial in the background
  • a vintage photo of a person standing in front of Lincoln Memorial
  • a vintage photo of a person with Lincoln Memorial in the background
  • a vintage photo of an old man standing in front of a book with Lincoln Memorial in the background
  • an old photo of a cake with Lincoln Memorial in the background
  • an old photo of a man with Lincoln Memorial in the background
  • an old photo of a person with Lincoln Memorial in the background
  • old photo of a person with Lincoln Memorial in the background
Luther King Jr. Memorial (3)

The entity Luther King Jr. Memorial is present in 3 descriptions.

  • a statue of a person with Martin Luther King Jr. Memorial in the background
  • a stone statue of a person with Martin Luther King Jr. Memorial in the background
  • an old photo of a person with Martin Luther King Jr. Memorial in the background
Notre (6)

The entity Notre is present in 6 descriptions.

  • a close up of a church with Notre Dame de Paris in the background
  • a close up of an old building with Notre Dame de Paris in the background
  • a close up of an old church with Notre Dame de Paris in the background
  • an old photo of a large building with Notre Dame de Paris in the background
  • an old photo of Notre Dame de Paris
  • old photo of Notre Dame de Paris
Pergamon Museum (3)

The entity Pergamon Museum is present in 3 descriptions.

  • a close up of a sign with Pergamon Museum in the background
  • a sign on a wall with Pergamon Museum in the background
  • close up of a sign with Pergamon Museum in the background
Robert Ford (3)

The entity Robert Ford is present in 3 descriptions.

  • a vintage photo of Robert Ford et al. posing for a picture
  • a vintage photo of Robert Ford et al. posing for the camera
  • a vintage photo of Robert Ford et al. sitting posing for the camera
Salisbury Cathedral (3)

The entity Salisbury Cathedral is present in 3 descriptions.

  • a vintage photo of a church with Salisbury Cathedral in the background
  • a vintage photo of an old building with Salisbury Cathedral in the background
  • a vintage photo of an old church with Salisbury Cathedral in the background
Trinity Church (6)

The entity Trinity Church is present in 6 descriptions.

  • a close up of a tall building in Trinity Church
  • a tall building in Trinity Church
  • a vintage photo of a castle with Trinity Church in the background
  • a vintage photo of an old building with Trinity Church in the background
  • an old photo of a castle with Trinity Church in the background
  • an old photo of Trinity Church
Wallace Ford (3)

The entity Wallace Ford is present in 3 descriptions.

  • a vintage photo of Wallace Ford et al. posing for a photo
  • a vintage photo of Wallace Ford et al. posing for a picture
  • a vintage photo of Wallace Ford et al. posing for the camera

1 Vocabularies are slow to compile so they are built about once a month. The current vocabulary was built on 2024-02-23.
2 Named entities are extracted using Compromise, a javascript NLP library. It's not exact, but pretty close.