CLIP-ViT-B-16-EntityNet-33M

A CLIP ViT-B/16 model trained from scratch on EntityNet-33M.

EntityNet is small enough to train a CLIP model from scratch on a single 8-GPU machine in about two days, which makes it useful for research that needs full control over the training data. It is especially strong on animals and plants.

Paper · Code · Dataset · Project page

Results

Zero-shot top-1 accuracy (%), all models evaluated with the same protocol from the paper, so numbers can differ slightly from those on other model cards. Retrieval is the average image-text recall@1 over COCO, Flickr30k and XM3600. Rare Species tests 400 species that were removed from our training data. Following the paper, it is only reported for models whose training data is known to exclude them ("–" otherwise).

Model	ImageNet	iNat 2021	CUB	Rare Species	Retrieval
OpenAI CLIP ViT-B/16 (400M images)	68.3	9.2	56.1	–	52.1
BioCLIP ViT-B/16 (TreeOfLife-10M)	18.6	–	78.1	38.1	0.8
This model	66.2	32.0	85.3	47.1	39.8

Usage

import torch
import open_clip
from PIL import Image

model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:lmb-freiburg/CLIP-ViT-B-16-EntityNet-33M")
model.eval()
tokenizer = open_clip.get_tokenizer("ViT-B-16", context_length=32)  # trained with 32 text tokens

image = preprocess(Image.open("image.jpg")).unsqueeze(0)
labels = ["a dog", "a cat", "a rabbit"]
text = tokenizer(labels)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print(labels[probs.argmax().item()])

Species names work in English ("red fox") and Latin ("Vulpes vulpes"); the paper reports the better of the two.

Training

The model has seen ~0.6B images at a batch size of 8k. Text labels were sampled 50/50 from the images' web alt texts and from the knowledge-graph text behind the image (the search query, or the entity's aliases or descriptions). See the paper for details.

All EntityNet models

Model	Training	Best for
CLIP-ViT-B-16-EntityNet-33M	from scratch, all 33M images	studying CLIP training with full control over the data
CLIP-ViT-B-32-EntityNet-33M	from scratch, all 33M images	same, smaller and faster
CLIP-ViT-B-16-EntityNet-33M-Finetuned	DataComp-1B finetuned on 33M	general use with a boost on animals and plants
CLIP-ViT-B-32-EntityNet-33M-Finetuned	DataComp-1B finetuned on 33M	same, smaller and faster
CLIP-ViT-B-16-LivingThings-10M-Finetuned	DataComp-1B finetuned on 10M organisms	best species recognition
CLIP-ViT-B-32-LivingThings-10M-Finetuned	DataComp-1B finetuned on 10M organisms	species recognition, smaller and faster
CLIP-ViT-B-32-LivingThings-10M	from scratch, 10M organisms	a species expert trained from scratch on organism images only

Citation

@inproceedings{ging2026entitynet,
  author    = {Simon Ging and Sebastian Walter and Jelena Bratuli{\'c} and Johannes Dienert and Hannah Bast and Thomas Brox},
  title     = {Using Knowledge Graphs to Harvest Datasets for Efficient {CLIP} Model Training},
  booktitle = {Pattern Recognition, 47th {DAGM} German Conference, {DAGM} {GCPR} 2025, Freiburg, Germany, September 23--26, 2025, Proceedings},
  series    = {Lecture Notes in Computer Science},
  publisher = {Springer Nature Switzerland},
  year      = {2026},
  pages     = {287--302},
  isbn      = {978-3-032-12840-9},
  doi       = {10.1007/978-3-032-12840-9_19}
}