Statistics

Total Volume of Data

The database contains 53,305,076 publicly accessible machine-generated annotations covering 377,818 images.

Annotations date from 2018-02-08 to 2024-05-10.

Each annotation indicates an area of interest in an image. An area can be a small portion of an image or it can be the entire image. Small regions typically contain a human face or words. Annotations on the full image are typically tags (e.g. cat, watermelon, rock) and descriptions (e.g. a cow standing in a field).

Access to annotations as data is available via the annotation endpoint on the Harvard Art Museums API.

Sources

Annotations are generated from 5 primary computer vision services.

Each service supplies a variety of functions for detecting and categorizing features. These are the services and features we use.

AWS Rekognition: Labels, face detection, text detection
Clarifai: Concepts
Google Vision: Labels, face detection, text detection, landmark detection
Imagga: Tags, categories
Microsoft Cognitive Sources: Tags, categories, descriptions, face detection

Annotation counts by source.

The raw data.

	AWS Rekognition	Anthropic	Azure OpenAI Service	Clarifai	Google Vision	Imagga	Microsoft Cognitive Services
category	0	0	0	0	0	951,541	0
description	0	10,605	10,639	0	0	0	826,384
face	319,208	0	0	0	107,193	0	80,293
tag	4,039,338	0	0	7,434,014	4,148,026	26,828,195	2,331,109
text	2,854,290	0	0	0	3,363,859	0	0
Total	7,212,836	10,605	10,639	7,434,014	7,619,078	27,779,736	3,237,786

Annotation Types

Annotation types generally describe what the annotation contains or depicts. These are the possible types.

category: a broad categorization of the contents of the image
description: a full sentence caption of the contents of the image
face: a human face is found within the annotation
tag: a term or set of terms describing all or part of the image
text: some text is found within the annotation

Distribution of sources across annotation types.

The raw data.

	category	description	face	tag	text
AWS Rekognition	0	0	319,208	4,039,338	2,854,290
Anthropic	0	10,605	0	0	0
Azure OpenAI Service	0	10,639	0	0	0
Clarifai	0	0	0	7,434,014	0
Google Vision	0	0	107,193	4,148,026	3,363,859
Imagga	951,541	0	0	26,828,195	0
Microsoft Cognitive Services	0	826,384	80,293	2,331,109	0
Total	951,541	847,628	506,694	44,781,063	6,218,150

Vocabularies¹

The tags and descriptions cluster and breakdown in to distinct terms, descriptive phrases, concepts, and named entities.

Terms

The vocabulary of tags contains 15,125 distinct terms.

The size of vocabulary by source.

The raw data.

	# of terms
AWS Rekognition	2,721
Clarifai	4,539
Imagga	9,578
Google Vision	8,147
Microsoft Cognitive Services	3,456

Sample set of terms:

Descriptive Phrases

The vocabulary of descriptions contains 41,085 descriptive phrases.

Sample set of descriptive phrases:

Named Entities²

Descriptions can be parsed further in to clusters of named entities.

The vocabulary of named entities contains 1,753 people.

Sample set of people:

Amos Adams Lawrence (6)

The entity Amos Adams Lawrence is present in 6 descriptions.

a vintage photo of Amos Adams Lawrence wearing a black hat
a vintage photo of Amos Adams Lawrence wearing a hat
Amos Adams Lawrence wearing a black hat
Amos Adams Lawrence wearing a hat
Amos Adams Lawrence wearing a suit and hat
Amos Adams Lawrence wearing a suit and tie

Edith Evans (3)

The entity Edith Evans is present in 3 descriptions.

Edith Evans et al. posing for a photo
Edith Evans et al. posing for a photo in front of a window
Edith Evans et al. posing for the camera

Francis Drake (2)

The entity Francis Drake is present in 2 descriptions.

Francis Drake posing for the camera
Francis Drake standing in front of a mirror posing for the camera

Grover Cleveland (7)

The entity Grover Cleveland is present in 7 descriptions.

a old photo of Grover Cleveland
a vintage photo of Grover Cleveland
a vintage photo of Grover Cleveland in a suit and tie
an old photo of Grover Cleveland
Grover Cleveland holding a sign
Grover Cleveland posing for a photo
old photo of Grover Cleveland

Hamilton (12)

The entity Hamilton is present in 12 descriptions.

a black and white photo of Hamilton Fish
a old photo of Hamilton Fish
a vintage photo of Hamilton Fish
a vintage photo of Hamilton Fish holding a book
a vintage photo of Hamilton Fish in a suit and tie
a vintage photo of Hamilton Fish wearing a suit and tie
an old black and white photo of Hamilton Fish
an old photo of Hamilton Fish
an old photo of Hamilton Fish wearing a suit and tie
Hamilton Fish, Rufus Choate, Domingo Faustino Sarmiento, Daniel Drew, James Russell Lowell, Ulysses S. Grant are posing for a picture
Hamilton Fish wearing a suit and tie
old photo of Hamilton Fish

Isaac Smith Kalloch (2)

The entity Isaac Smith Kalloch is present in 2 descriptions.

Isaac Smith Kalloch holding a book
Isaac Smith Kalloch holding a sign

James Buchanan (10)

The entity James Buchanan is present in 10 descriptions.

a black and white photo of James Buchanan
a vintage photo of James Buchanan
a vintage photo of James Buchanan et al. posing for a picture
a vintage photo of James Buchanan holding a book
a vintage photo of James Buchanan in a suit and tie
a vintage photo of James Buchanan in black suit
a vintage photo of James Buchanan wearing a suit and tie
an old black and white photo of James Buchanan
an old photo of James Buchanan
James Buchanan with a book

James Garner (3)

The entity James Garner is present in 3 descriptions.

a vintage photo of James Garner in a suit and tie
a vintage photo of James Garner wearing a suit and tie
an old photo of James Garner in a suit and tie

James Walker (3)

The entity James Walker is present in 3 descriptions.

a vintage photo of James Walker
a vintage photo of James Walker in a suit and tie
a vintage photo of James Walker wearing a suit and tie

John J. Crittenden (5)

The entity John J. Crittenden is present in 5 descriptions.

an old photo of John J. Crittenden
an old photo of John J. Crittenden in a suit and tie
an old photo of John J. Crittenden wearing a suit and tie
old photo of John J. Crittenden
old photo of John J. Crittenden in a suit standing in front of a building

John Wilkes Booth (10)

The entity John Wilkes Booth is present in 10 descriptions.

a black and white photo of John Wilkes Booth
a black and white photo of John Wilkes Booth et al. posing for a picture
a black and white photo of John Wilkes Booth et al. posing for the camera
a vintage photo of John Wilkes Booth
a vintage photo of John Wilkes Booth and woman posing for a picture
a vintage photo of John Wilkes Booth et al. posing for a picture
a vintage photo of John Wilkes Booth et al. posing for the camera
a vintage photo of John Wilkes Booth holding a book
an old black and white photo of John Wilkes Booth
John Wilkes Booth et al. posing for a photo

Leonard Nimoy (3)

The entity Leonard Nimoy is present in 3 descriptions.

a vintage photo of Leonard Nimoy et al. posing for a picture
a vintage photo of Leonard Nimoy et al. posing for the camera
an old photo of Leonard Nimoy et al. posing for the camera

Lois Moran (2)

The entity Lois Moran is present in 2 descriptions.

Lois Moran posing for the camera
Lois Moran wearing a hat

Michael Corcoran (3)

The entity Michael Corcoran is present in 3 descriptions.

Michael Corcoran standing in front of a mirror posing for the camera
Michael Corcoran standing posing for the camera
Michael Corcoran wearing a suit and tie posing for a photo

Mimi Aguglia (1)

The entity Mimi Aguglia is present in 1 descriptions.

an old photo of Mimi Aguglia

Robert Schumann (2)

The entity Robert Schumann is present in 2 descriptions.

an old photo of Robert Schumann
old photo of Robert Schumann

Vincent Van Gogh (5)

The entity Vincent Van Gogh is present in 5 descriptions.

a painting of Vincent Van Gogh
a painting of Vincent Van Gogh in front of a mirror
a painting of Vincent Van Gogh in front of a mirror posing for the camera
a painting of Vincent Van Gogh smiling for the camera
Vincent Van Gogh looking at the camera

Wallace (2)

The entity Wallace is present in 2 descriptions.

Foy E. Wallace et al. sitting at a table
Foy E. Wallace et al. sitting on a table

William Harcourt (4)

The entity William Harcourt is present in 4 descriptions.

a vintage photo of William Harcourt
a vintage photo of William Harcourt holding a book
a vintage photo of William Harcourt in a suit and tie
a vintage photo of William Harcourt wearing a suit and tie

William M. Gwin (4)

The entity William M. Gwin is present in 4 descriptions.

a vintage photo of William M. Gwin
a vintage photo of William M. Gwin holding a book
a vintage photo of William M. Gwin in a suit and tie
a vintage photo of William M. Gwin sitting in front of a book

The vocabulary of named entities contains 52 places.

Sample set of places:

Albania (5)

The entity Albania is present in 5 descriptions.

a vintage photo of Zog I of Albania et al. posing for a picture
a vintage photo of Zog I of Albania et al. posing for the camera
a vintage photo of Zog I of Albania standing in front of a building
Zog I of Albania and woman posing for a photo
Zog I of Albania et al. posing for a photo

Austria (18)

The entity Austria is present in 18 descriptions.

a painting of Archduke Ferdinand Karl of Austria
a painting of Archduke Ferdinand Karl of Austria holding a sign
a vintage photo of Archduchess Maria Theresa of Austria
a vintage photo of Archduke Franz Ferdinand of Austria
a vintage photo of Archduke Franz Ferdinand of Austria holding a book
a vintage photo of Archduke Franz Ferdinand of Austria standing in front of a building
a vintage photo of Archduke Franz Ferdinand of Austria wearing a black dress
a vintage photo of Empress Elisabeth of Austria
a vintage photo of Empress Elisabeth of Austria holding a book
a vintage photo of Empress Elisabeth of Austria with a book
an old black and white photo of Archduke Franz Ferdinand of Austria
an old photo of Empress Elisabeth of Austria
Archduchess Maria Theresa of Austria posing for a photo
Archduchess Maria Theresa of Austria posing for the camera
Archduke Ferdinand Karl of Austria holding a sign posing for the camera
Empress Elisabeth of Austria posing for a photo
Empress Elisabeth of Austria posing for a photo in front of a book
old photo of Empress Elisabeth of Austria

Bavaria (25)

The entity Bavaria is present in 25 descriptions.

a black and white photo of King Ludwig II of Bavaria
a black and white photo of King Ludwig II of Bavaria holding a book
a screen shot of a photo of King Ludwig II of Bavaria
a screen shot of King Ludwig II of Bavaria
a screen shot of King Ludwig II of Bavaria in a suit and tie
a vintage photo of King Ludwig II of Bavaria
a vintage photo of King Ludwig II of Bavaria and woman posing for a picture
a vintage photo of King Ludwig II of Bavaria et al. posing for a picture
a vintage photo of King Ludwig II of Bavaria holding a book
a vintage photo of King Ludwig II of Bavaria in a suit and tie
a vintage photo of King Ludwig II of Bavaria in a white shirt and black text
a vintage photo of King Ludwig II of Bavaria making a face for the camera
an old photo of King Ludwig II of Bavaria
an old photo of King Ludwig II of Bavaria holding a book
an old photo of King Ludwig II of Bavaria in a suit and tie
an old photo of Otto of Bavaria
King Ludwig II of Bavaria, Pope Pius IX are posing for a picture
King Ludwig II of Bavaria, Pope Pius IX posing for a photo
King Ludwig II of Bavaria posing for a photo
King Ludwig II of Bavaria posing for a photo in front of a book
King Ludwig II of Bavaria posing for the camera
King Ludwig II of Bavaria wearing a suit and tie
old photo of King Ludwig II of Bavaria
Otto of Bavaria posing for a photo
Otto of Bavaria posing for a picture

Belgium (10)

The entity Belgium is present in 10 descriptions.

a close up of Leopold II of Belgium
a vintage photo of Leopold II of Belgium
a vintage photo of Leopold II of Belgium holding a book
a vintage photo of Leopold II of Belgium standing in front of a book
an old photo of Leopold II of Belgium
Leopold II of Belgium holding a book
Leopold II of Belgium posing for a photo
Leopold II of Belgium standing in front of a book
Leopold II of Belgium taking a selfie
old photo of Leopold II of Belgium

California (3)

The entity California is present in 3 descriptions.

J. California Cooper et al. posing for a photo
J. California Cooper et al. sitting on a bench posing for the camera
J. California Cooper et al. sitting posing for the camera

Denmark (27)

The entity Denmark is present in 27 descriptions.

a statue of Alexandra of Denmark
a statue of Alexandra of Denmark in a suit and tie
a statue of Alexandra of Denmark wearing a suit and tie
a vintage photo of Alexandra of Denmark
a vintage photo of Alexandra of Denmark et al. posing for a photo
a vintage photo of Alexandra of Denmark et al. posing for a picture
a vintage photo of Alexandra of Denmark et al. posing for the camera
a vintage photo of Alexandra of Denmark et al. sitting on a bench
a vintage photo of Alexandra of Denmark holding a book
a vintage photo of Alexandra of Denmark holding a sign posing for the camera
a vintage photo of Alexandra of Denmark in a suit and tie
a vintage photo of Princess Thyra of Denmark
a vintage photo of Princess Thyra of Denmark holding a book
Alexandra of Denmark holding a book
Alexandra of Denmark holding a book posing for the camera
Alexandra of Denmark posing for a photo in front of a book
Alexandra of Denmark posing for the camera
an old photo of Princess Cecilie of Greece and Denmark
an old photo of Princess Cecilie of Greece and Denmark and woman posing for a picture
an old photo of Princess Thyra of Denmark
old photo of Princess Cecilie of Greece and Denmark et al. standing in front of a window
Prince Harald of Denmark posing for the camera
Prince Harald of Denmark standing posing for the camera
Prince Harald of Denmark wearing a suit and tie
Princess Cecilie of Greece and Denmark, Bobby Jordan posing for a photo
Princess Cecilie of Greece and Denmark, Bobby Jordan posing for a picture
Princess Cecilie of Greece and Denmark, Bobby Jordan posing for the camera

England (1)

The entity England is present in 1 descriptions.

an old photo of Henry III of England

George Perkins Marsh (3)

The entity George Perkins Marsh is present in 3 descriptions.

an old photo of George Perkins Marsh
George Perkins Marsh posing for a photo
old photo of George Perkins Marsh

Hawaii (4)

The entity Hawaii is present in 4 descriptions.

a vintage photo of Queen Emma of Hawaii and woman posing for a picture
a vintage photo of Queen Emma of Hawaii et al. posing for a picture
a vintage photo of Queen Emma of Hawaii et al. posing for the camera
a vintage photo of Queen Emma of Hawaii holding a book

Israel (2)

The entity Israel is present in 2 descriptions.

a vintage photo of Israel Silvestre
a vintage photo of Israel Silvestre holding a book

Joe Hill (3)

The entity Joe Hill is present in 3 descriptions.

a black and white photo of Joe Hill
a vintage photo of Joe Hill in a suit and tie
a vintage photo of Joe Hill wearing a suit and tie

Mexico (7)

The entity Mexico is present in 7 descriptions.

a black and white photo of Maximilian I of Mexico
a vintage photo of Maximilian I of Mexico in a suit and tie
a vintage photo of Maximilian I of Mexico in a suit posing for a picture
a vintage photo of Maximilian I of Mexico wearing a suit and tie
an old black and white photo of Maximilian I of Mexico
Maximilian I of Mexico holding a sign
Maximilian I of Mexico wearing a suit and tie

Morocco (3)

The entity Morocco is present in 3 descriptions.

an old photo of Princess Lalla Aicha of Morocco et al. in a room
an old photo of Princess Lalla Aicha of Morocco et al. standing in a room
Princess Lalla Aicha of Morocco et al. standing in a room

New Mexico (9)

The entity New Mexico is present in 9 descriptions.

a close up of an orange with Ohkay Owingeh, New Mexico in the background
a piece of bread with Ohkay Owingeh, New Mexico in the background
a piece of cake on a table with Ohkay Owingeh, New Mexico in the background
a piece of food on a table with Ohkay Owingeh, New Mexico in the background
a piece of food with Ohkay Owingeh, New Mexico in the background
an orange cut in half and sitting on a table with Ohkay Owingeh, New Mexico in the background
an orange cut in half on a plate with Ohkay Owingeh, New Mexico in the background
an orange cut in half with Ohkay Owingeh, New Mexico in the background
an orange sitting on a table with Ohkay Owingeh, New Mexico in the background

Old Tuscaloosa County (3)

The entity Old Tuscaloosa County is present in 3 descriptions.

a man wearing a hat and looking at the camera with Old Tuscaloosa County Jail in the background
a man wearing a hat looking at the camera with Old Tuscaloosa County Jail in the background
a man wearing a hat with Old Tuscaloosa County Jail in the background

Rome (12)

The entity Rome is present in 12 descriptions.

a close up of an old building with Pantheon, Rome in the background
a vintage photo of a busy city street with Pantheon, Rome in the background
a vintage photo of a church with Pantheon, Rome in the background
a vintage photo of a horse drawn carriage on Pantheon, Rome street
a vintage photo of an old building in the background with Pantheon, Rome in the background
a vintage photo of an old building with Pantheon, Rome in the background
a vintage photo of an old church with Pantheon, Rome in the background
a vintage photo of an old stone building with Pantheon, Rome in the background
a vintage photo of Pantheon, Rome
a vintage photo of Pantheon, Rome street
an old photo of a large building with Pantheon, Rome in the background
an old photo of Pantheon, Rome

St (6)

The entity St is present in 6 descriptions.

a close up of a church with St Mark's Basilica in the background
a group of people riding on the back of a church with St Mark's Basilica in the background
a large building with St Mark's Basilica in the background
a large old building with many windows with St Mark's Basilica in the background
an old church with St Mark's Basilica in the background
an old photo of a church with St Mark's Basilica in the background

Sweden (5)

The entity Sweden is present in 5 descriptions.

a vintage photo of Oscar II of Sweden
a vintage photo of Oscar II of Sweden in a white shirt and black text
a vintage photo of Oscar II of Sweden in white shirt and black text
an old photo of Oscar II of Sweden
old photo of Oscar II of Sweden

Tennessee (3)

The entity Tennessee is present in 3 descriptions.

a vintage photo of Tennessee Williams in a suit and tie
a vintage photo of Tennessee Williams wearing a suit and tie
an old photo of Tennessee Williams wearing a suit and tie

Tex (6)

The entity Tex is present in 6 descriptions.

Tex Hughson et al. posing for a photo
Tex Hughson et al. posing for a photo in front of a building
Tex Hughson et al. posing for the camera
Tex Ritter et al. posing for a photo
Tex Ritter et al. sitting in front of a mirror posing for the camera
Tex Ritter et al. standing in front of a mirror posing for the camera

The vocabulary of named entities contains 10 organizations.

Sample set of organizations:

Andy Warhol Museum (1)

The entity Andy Warhol Museum is present in 1 descriptions.

a close up of a book with The Andy Warhol Museum in the background

Bruton Parish Church (3)

The entity Bruton Parish Church is present in 3 descriptions.

a house that has a sign on the side of Bruton Parish Church
a sign in front of a house with Bruton Parish Church in the background
a sign on the side of a house with Bruton Parish Church in the background

Lincoln Memorial (21)

The entity Lincoln Memorial is present in 21 descriptions.

a group of people posing for a photo with Lincoln Memorial in the background
a group of people posing for a picture with Lincoln Memorial in the background
a group of people posing for the camera with Lincoln Memorial in the background
a man and a woman standing in front of a window with Lincoln Memorial in the background
a man sitting on a bench with Lincoln Memorial in the background
a man standing in front of a mirror with Lincoln Memorial in the background
a man standing in front of a window with Lincoln Memorial in the background
a vintage photo of a group of people posing for a picture with Lincoln Memorial in the background
a vintage photo of a group of people posing for the camera with Lincoln Memorial in the background
a vintage photo of a man holding a book with Lincoln Memorial in the background
a vintage photo of a man sitting on a bench with Lincoln Memorial in the background
a vintage photo of a man standing in front of a book with Lincoln Memorial in the background
a vintage photo of a man with Lincoln Memorial in the background
a vintage photo of a person holding a book with Lincoln Memorial in the background
a vintage photo of a person standing in front of Lincoln Memorial
a vintage photo of a person with Lincoln Memorial in the background
a vintage photo of an old man standing in front of a book with Lincoln Memorial in the background
an old photo of a cake with Lincoln Memorial in the background
an old photo of a man with Lincoln Memorial in the background
an old photo of a person with Lincoln Memorial in the background
old photo of a person with Lincoln Memorial in the background

Luther King Jr. Memorial (3)

The entity Luther King Jr. Memorial is present in 3 descriptions.

a statue of a person with Martin Luther King Jr. Memorial in the background
a stone statue of a person with Martin Luther King Jr. Memorial in the background
an old photo of a person with Martin Luther King Jr. Memorial in the background

Notre (6)

The entity Notre is present in 6 descriptions.

a close up of a church with Notre Dame de Paris in the background
a close up of an old building with Notre Dame de Paris in the background
a close up of an old church with Notre Dame de Paris in the background
an old photo of a large building with Notre Dame de Paris in the background
an old photo of Notre Dame de Paris
old photo of Notre Dame de Paris

Pergamon Museum (3)

The entity Pergamon Museum is present in 3 descriptions.

a close up of a sign with Pergamon Museum in the background
a sign on a wall with Pergamon Museum in the background
close up of a sign with Pergamon Museum in the background

Robert Ford (3)

The entity Robert Ford is present in 3 descriptions.

a vintage photo of Robert Ford et al. posing for a picture
a vintage photo of Robert Ford et al. posing for the camera
a vintage photo of Robert Ford et al. sitting posing for the camera

Salisbury Cathedral (3)

The entity Salisbury Cathedral is present in 3 descriptions.

a vintage photo of a church with Salisbury Cathedral in the background
a vintage photo of an old building with Salisbury Cathedral in the background
a vintage photo of an old church with Salisbury Cathedral in the background

Trinity Church (6)

The entity Trinity Church is present in 6 descriptions.

a close up of a tall building in Trinity Church
a tall building in Trinity Church
a vintage photo of a castle with Trinity Church in the background
a vintage photo of an old building with Trinity Church in the background
an old photo of a castle with Trinity Church in the background
an old photo of Trinity Church

Wallace Ford (3)

The entity Wallace Ford is present in 3 descriptions.

a vintage photo of Wallace Ford et al. posing for a photo
a vintage photo of Wallace Ford et al. posing for a picture
a vintage photo of Wallace Ford et al. posing for the camera

¹ Vocabularies are slow to compile so they are built about once a month. The current vocabulary was built on 2024-02-23.
² Named entities are extracted using Compromise, a javascript NLP library. It's not exact, but pretty close.

Menu

Menu

Statistics

Total Volume of Data

Sources

Annotation Types

Vocabularies1

Terms

Descriptive Phrases

Named Entities2

Vocabularies¹

Named Entities²