davanstrien/hub-task-tagger-gliner2.5-base

Hub task tagger (GLiNER2.5-base)

Suggests task tags for a Hugging Face dataset, such as text-generation, robotics or automatic-speech-recognition, from its column names and first row. It picks from the 52 task tags the Hub offers, and gives a probability for each.

TypeSafe AI's Jev has got people interested in "System One" models: models that don't generate text, but read a state and return typed answers with probabilities. This is a small, open example of the same idea for one narrow job, fine-tuned in 17 minutes on Hugging Face Jobs.

Try it: davanstrien/hub-task-tagger. Paste a dataset id and see the suggested tags, the owner's tags, and the exact text the model read.

How to use

The model reads a short text built from the dataset viewer's preview: a Columns: line with each column's name and type, then the first row, cut to 370 tokens. It must be built exactly as in training, so this repo ships the builder, tagger_core.py, next to the weights. The snippet below downloads it with huggingface_hub and uses it.

import json, math, os, sys
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
from gliner2.classification import Classifier, ClassificationSchema, ClassificationConfig

repo = "davanstrien/hub-task-tagger-gliner2.5-base"

# 1. Build the model's input for a dataset, with the code that built the training data.
path = hf_hub_download(repo, "tagger_core.py")
sys.path.insert(0, os.path.dirname(path))  # make the downloaded file importable
import tagger_core as core

builder = core.StateBuilder(Tokenizer.from_file(hf_hub_download("Qwen/Qwen3.5-4B-Base", "tokenizer.json")), 370)

first_rows, config, split, error = core.fetch_first_rows("fancyzhx/ag_news")
state, truncated = builder.build(core.render_first_rows(first_rows))
print(state)

# 2. Score all 52 tags.
labels = json.load(open(hf_hub_download(repo, "classification_schema.json")))["tasks"][0]["labels"]
model = Classifier.from_pretrained(repo).eval()
schema = ClassificationSchema().multi("labels", labels)
logits = model.batch_score([state[:2000]], schema, config=ClassificationConfig(batch_size=1))[0].tasks["labels"]

# 3. Calibrated probabilities: softmax over the 52 logits at temperature 1.30 (fitted on held-out data).
z = [logits[label] / 1.30 for label in labels]
m = max(z)
e = [math.exp(v - m) for v in z]
probs = sorted(zip(labels, (v / sum(e) for v in e)), key=lambda x: -x[1])
print(probs[:3])

# A set of tags: everything at 0.225 or above, and always the top tag.
suggested = [label for label, p in probs if p >= 0.225] or [probs[0][0]]
print(suggested)

The Qwen tokenizer is only used to measure and cut the input to the length the model was trained on. fetch_first_rows returns an error instead of rows when a dataset has no viewer preview.

Install with pip install "gliner2[local]==2.0.0". Weights are fp32. On a free 2-vCPU Space one prediction takes about 0.7–1 s; on an L4 GPU in fp16, about 20 ms.

Training

• Recipe: train-gliner2.py from uv-scripts/classification, run as one hf jobs uv run command on an rtx-pro-6000: --base-model fastino/gliner2.5-base-v1 --labels-file labels.json --label-column labels --label-augmentation off, 5 epochs, seed 0, bf16. About 17 minutes of training, roughly $1.50.
• Data: 16,000 public Hub datasets with owner-declared task_categories. Each example is the dataset's column names and first row (from the dataset viewer), cut to about 370 tokens, and its declared tags (multi-label). Retired tags were mapped to current ones where the task is the same (for example text2text-generation → text-generation); others were dropped.
• Split by time and owner: the evaluation sets are 1,000 (calibration) and 3,000 (development) datasets created in the 60 days before 2026-09-22, by owners who have no dataset in the training data. Near-duplicate datasets from the same owner cannot leak between training and evaluation.

Evaluation

On the 3,000 development datasets. A prediction counts as correct if it is one of the owner's tags.

	Top-1	Top-3	Macro recall (24 tags)
This model	0.695	0.881	0.525
Same recipe, second seed	0.686	0.879	0.511
GLiNER2.5-small, same recipe	0.653	0.855	0.458
GLiNER2.5-base, zero-shot	0.102	0.213	0.099
Always "text-generation"	0.320	–	–

• The owners' tags are a noisy target. In a hand-checked sample, about 1 dataset in 10 was missing a tag that fits. Counting 74 card-checked extra tags as correct raises top-1 to 0.720.
• As a set (all tags at probability ≥ 0.225), micro-F1 against the owner's tags is 0.623, with 1.21 tags per dataset on average against 1.32 declared.
• Confidence: the temperature and the threshold were fitted on the calibration set. Thresholds chosen there held on development within a few points of error, but not exactly: treat the probabilities as a good ranking, not as exact rates.

Limitations

• It only sees the first row. Datasets whose purpose is not visible from the columns and one example (for example, generic tables) are hard.
• Tabular tags are weak. It often confuses tabular-regression with tabular-classification, because the input does not say which column is the target.
• Rare tags (17 of the 52, such as text-to-3d or voice-activity-detection) had too few training examples to measure.
• It learned from owners' choices, including their habits: for example, question-answering datasets tagged text-generation, or the reverse.
• Trained and evaluated mostly on English-language metadata.

Credits

Built on GLiNER2 by Fastino. Training, evaluation and the demo were run on Hugging Face Jobs and Spaces, with much of the work done by coding agents (Claude) and checked by a person. More detail: GLINER2-NOTES.md.