README.md

---
language:
- hu
license: apache-2.0
library_name: tokenizers
tags:
- picur
- tokenizer
- hungarian
- magyar
- morphology
- sentencepiece
- unigram
---

# picur-tokenizer

A 49,152-piece SentencePiece Unigram tokenizer for Hungarian,
fitted so that it segments on real morpheme boundaries,
rather than on whatever the compression objective prefers.

```python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("picur/picur-tokenizer")
tok.tokenize("a házakban lakó emberekkel")
# ['▁a', '▁ház', 'ak', 'ban', '▁lakó', '▁ember', 'ek', 'kel']
```

## Why another tokenizer

> - Nevezz meg három állatot.
> - Kutyácska, lovacska, verebecske.
> - Jó, de kicsinyítő képző nélkül.
> - Kecs, macs, szar.

Hungarian is agglutinative: one stem carries a long tail of suffixes, so `ház` (house) surfaces as `házban`, `házakban`, `házaimban`.
A tokenizer fitted on compression alone shatters that paradigm across unrelated rows,
and the model then has to learn `ház` several times over - once per spelling of the same word.
Training effort that should have gone into knowledge goes into re-learning morphology instead.

This vocabulary is fitted to keep a stem on one row and let the suffixes be their own pieces,
so a paradigm shares its knowledge instead of spreading it.

## How it segments

| word | pieces | |
|---|---|---|
| `házakban` | `▁ház` `ak` `ban` | stem + plural + inessive |
| `könyvekkel` | `▁könyv` `ek` `kel` | stem + plural + instrumental |
| `emberekkel` | `▁ember` `ek` `kel` | the same two suffixes, the same rows |
| `magyarokat` | `▁magyar` `ok` `at` | back-vowel harmony of the pair above |
| `csapadékvíz` | `▁csapadék` `víz` | compound, cut at the declared seam |
| `tavak` | `▁tav` `ak` | v-stem alternation (`tó` -> `tav`) |

Vowel harmony is why `ban`/`ben`, `kel`/`kal` and `hoz`/`hez`/`höz` all exist as separate rows:
they are the same grammatical function wearing different vowels.
See [Tied rows](#tied-rows) for what is shipped to help a model learn them as one.

## Measured

Every number below is reproducible from public artifacts by `benchmark.py` in this repository:

```
pip install transformers datasets
python benchmark.py
```

It reads a pinned UniMorph commit, five Hugging Face datasets at a fixed document count, and five published tokenizers.
Nothing from the pipeline that produced this vocabulary is involved.

### Paradigm sharing

The property this vocabulary is built for.
On [UniMorph Hungarian](https://github.com/unimorph/hun) first 20,000 lemmas in file order: 19,424 lemmas, 1,014,882 lemma/form pairs:

| tokenizer | vocab | **one stem row + suffixes** | prefix kept | lemma is one token | tokens/form |
|---|---|---|---|---|---|
| **picur/picur-tokenizer** | 49,152 | **45.4%** | **73.1%** | **55.1%** | 3.04 |
| NYTK/PULI-GPT-3SX | 50,000 | 24.5% | 54.9% | 38.5% | 2.98 |
| SZTAKI-HLT/hubert-base-cc | 32,001 | 23.6% | 55.5% | 28.1% | 3.07 |
| FacebookAI/xlm-roberta-base | 250,002 | 10.8% | 60.4% | 12.8% | 3.66 |
| Qwen/Qwen3-8B | 151,643 | 2.0% | 54.9% | 3.1% | 5.29 |

**one stem row + suffixes** asks both halves of the question at once:
the lemma is a SINGLE row, and its inflected form begins with that row.
So every mention of the word teaches one embedding.

Neither half can be won on its own. Cutting everything small satisfies `prefix kept` and destroys `lemma is one token`.
Swallowing whole inflected forms does the reverse.
Only the conjunction says a paradigm shares a row, and it does so at the same token cost per form as the closest baseline.

### Fertility

Characters a token buys, on the first 2,000 documents of each dataset:

| corpus | picur | PULI-GPT-3SX | huBERT | XLM-R | Qwen3 |
|---|---|---|---|---|---|
| `lazos/gutenberg-hu` | 3.740 | **4.050** | 3.915 | 3.381 | 2.417 |
| `lazos/gazette-hu` | 2.960 | **3.808** | 3.665 | 3.392 | 1.997 |
| `SZTAKI-HLT/HunSum-2-abstractive` | 4.001 | **4.544** | 4.487 | 3.803 | 2.367 |
| `open-web-math/open-web-math` | **2.953** | 2.394 | 2.385 | 3.412 | 3.669 |
| `bigcode/starcoderdata` (python) | **2.498** | 1.721 | 2.454 | 3.205 | 4.121 |

**On Hungarian prose the two Hungarian baselines beat this tokenizer, and that is the trade, not a defect.**
PULI buys more characters per token by cutting `házakban` into `ház` `akban`, keeping stem and endings fused;
this vocabulary spends one more token to cut it into `▁ház` `ak` `ban`, and gets the paradigm table above in return.

Where the comparison is not about Hungarian morphology it inverts:
on mathematics and code this vocabulary beats both Hungarian baselines,
because a slice of each was in the fit corpus so technical text does not fall back to bytes.
Qwen wins those outright, as an English-first tokenizer should.

### Special tokens

| id | token | |
|---|---|---|
| 0 | `<\|unk\|>` | |
| 1 | `<\|bos\|>` | one per document |
| 2 | `<\|eos\|>` | |
| 3 | `<\|pad\|>` | |
| 4 | `<\|im_start\|>` | ChatML turn open |
| 5 | `<\|im_end\|>` | ChatML turn close; the stop token for chat |
| 6-21 | `<\|reserved_0..15\|>` | unused, for later extension |

Tool-call markers (`<tools>`, `<tool_call>`, `<tool_response>` and their closers) are present as ordinary pieces, following the Hermes convention.

Note the ChatML split: `<|im_end|>` terminates a *turn*, `<|eos|>` terminates a *sequence*. A chat model should stop on `<|im_end|>`.

### Tied rows

`ties.json` maps 1,357 allomorph rows onto the one row a model should learn first: `ban` -> `ben`, `höz` -> `hoz`, across 100 suffix families.
Vowel harmony makes these variants of one morpheme,
so initialising their embeddings from a shared base gives the model the paradigm for free rather than making it discover the relationship from data.

This is an initialisation aid, not part of tokenization, nothing reads it automatically.

## How it was fitted

The vocabulary is overfitted well past its target size and then **selected down**,
with a declared Hungarian lexicon deciding what survives:

1. **Seed** - 32,888 pieces required up front: 28,000 frequent stems, 1,000 named entities,
   1,000 capitalised forms, 432 suffixes, 2,005 alternating stems and 448 programming keywords.
2. **Overfit** - a large Unigram vocabulary is trained on the corpus.
3. **Evict** - any piece that is a regular inflection of a declared headword is removed. `magyarokat` becomes `▁magyar` `ok` `at`:
   the whole ending `okat` is not a row, while the plural `ok` and the accusative `at` each are, and every other paradigm reuses them.
4. **Spare** - an eviction is reversed if the word would then segment worse than its morphology says it should.
5. **Prune** - rows the corpus never reaches are dropped.

Corpus: 700 Mchar drawn across 57 Hungarian sources:
encyclopedic, news, legal, parliamentary, academic, literary, subtitles, web and forum.
Plus small code and mathematics slices, so that technical text does not fall back to bytes.

## Limitations

- **Hungarian-first.** English and code are handled but not optimized; other languages fall back to bytes.
- **Coverage, not rules, sets the floor.** A stem outside the 28,000-stem seed can still be mis-split,
  and an alternating stem can lose on score: `művek` cuts as `mű` + `vek` where morphology says `műv` + `ek`.
- **Compound seams come from a declared lexicon**, so a compound absent from it gets no guaranteed seam.
- **Cased entity handling is partial.** 1,000 capitalised forms are seeded; beyond those, capitalisation costs a row.
- **Two losses are declared, not accidental.** Metaspace writes the word mark as
  U+2581, so a literal U+2581 in the input is indistinguishable from a space. The
  tokenizer declares a `Replace(U+2581, " ")` normalizer ahead of Metaspace, which
  makes the collision part of the contract instead of a surprise - text carrying
  block-drawing characters (a sparkline, say) round-trips with that one character
  turned into a space. Leading whitespace on an input is dropped for the same
  reason: Metaspace prepends a mark unconditionally. Everything else round-trips
  byte-exactly, including every script outside Latin, via byte fallback.

## Attribution

This tokenizer is a derivative work of the following resouces, see `NOTICE`.

- **[morphdb.hu](https://github.com/morphdb/morphdb.hu)** - CC-BY 2.5, (c) Nemeth
  Laszlo and Godo Ferenc. The declared lexicon: 351,917 headwords, 37,042 compound
  seams, alternation classes and vowel harmony. This decided what the fit protects
  and what it evicts.
- **[hunmorph-foma](https://github.com/r0ller/hunmorph-foma)** - BSD-2-Clause,
  (c) 2016 r0ller. Finite-state transducers that analysed corpus words into lemma
  plus tags, bootstrapping the segmentation this vocabulary is scored against.
- **[UniMorph Hungarian](https://github.com/unimorph/hun)** - CC BY-SA 3.0, derived
  from English Wiktionary. Used by `benchmark.py` only: it is scored against and
  never built from, which is what makes the paradigm table above independent of
  everything that produced the vocabulary.

The tokenizer itself is Apache-2.0.

## Citation

```bibtex
@misc{picur-tokenizer,
  title  = {picur-tokenizer: a morphology-aware Hungarian tokenizer},
  author = {Bencz, Lajos},
  year   = {2026},
  url    = {https://huggingface.co/picur/picur-tokenizer}
}
```