--- language: - uk library_name: pytorch pipeline_tag: image-to-text license: cc-by-sa-4.0 tags: - ocr - htr - handwritten-text-recognition - parseq - ukrainian - cyrillic - image-to-text datasets: - pumb-ai/synthetic-cyrillic-large - ai-forever/school_notebooks_RU - nastyboget/synthetic_cyrillic_large metrics: - cer - wer model-index: - name: parseq-b-cyrillic-handwritten results: - task: type: image-to-text name: Handwritten Text Recognition dataset: type: other name: Cyrillic handwriting + synthetic mix, held-out val split: validation metrics: - type: cer value: 0.0118 name: CER - type: wer value: 0.0476 name: WER --- # parseq-b-cyrillic-handwritten A PARSeq-B (permuted autoregressive sequence) line recognizer for single-line handwritten Ukrainian/Cyrillic text — a larger ViT-B encoder variant (~96M parameters, ~4× the PARSeq-S model). ## TL;DR | | value | |---|---| | Architecture | PARSeq — ViT-B encoder + 1-layer permutation-LM decoder (ECCV 2022) | | Parameters | ~96M | | Stage | mixed pretrain (general handwriting reader, from scratch; pre-fine-tune) | | Languages | Ukrainian | | Decode | AR greedy + 1 refinement iteration | | Best CER / WER | **0.0118 / 0.0476** (greedy+refine, seed 42) | | Input | a single cropped text line, BGR (numpy) — resized to 48×512 | | Output | the transcribed string (232-char frozen Cyrillic charset) | ## Intended use Recognizing single cropped lines of handwritten Ukrainian text, downstream of a line/region detector or on already-segmented lines. Useful directly as a general handwriting reader, or as an init for fine-tuning on a target domain. A companion ViT-S variant (~24M params) trained with the same recipe is available at [`Hukyl/parseq-s-cyrillic-handwritten`](https://huggingface.co/Hukyl/parseq-s-cyrillic-handwritten). ## How to use Note: the checkpoint is a plain `torch.save` archive, not transformers-compatible. The payload is a dict: ```python import torch payload = torch.load("best.pt", map_location="cpu", weights_only=True) # keys: "model_state" (state_dict), "charset" (str), "config" (dict), # "metrics" (dict), "epoch" (int) print(payload["config"]) # {'img_height': 48, 'img_width': 512, 'patch_size': (4, 8), 'embed_dim': 768, # 'enc_num_heads': 12, 'enc_mlp_ratio': 4, 'enc_depth': 12, 'dec_num_heads': 12, # 'dec_mlp_ratio': 4, 'dec_depth': 1, 'max_label_length': 100, 'dropout': 0.1, # 'decode_ar': True, 'refine_iters': 1, 'drop_path_rate': 0.1} ``` To run it: construct a PARSeq model (ViT encoder + 1-layer two-stream permutation decoder; see [`baudm/parseq`](https://github.com/baudm/parseq), Apache-2.0) with the embedded `config`, build a tokenizer over the embedded `charset` (token order is `[E]` + the charset string + `[B]` + `[P]`; `[E]` is id 0), then `model.load_state_dict(payload["model_state"])`. The input geometry ships in the checkpoint's `config`: each crop is resized unconditionally to 48×512 RGB (no aspect preservation; `INTER_AREA` on downscale) and normalized `(x − 0.5) / 0.5`. Output confidence is the mean per-step max-softmax over the decoded sequence. ## Architecture PARSeq, after Bautista & Atienza, "Scene Text Recognition with Permuted Autoregressive Sequence Models" (ECCV 2022, [arXiv:2207.06966](https://arxiv.org/abs/2207.06966)). Model code is derived from [`baudm/parseq`](https://github.com/baudm/parseq) (Apache-2.0). Geometry, decoder, and charset are identical to the ViT-S variant; only the encoder width and head count are scaled up. | component | value | |---|---| | encoder | ViT-B — 12 layers, dim 768, 12 heads, MLP ratio 4, patch 4×8 | | decoder | 1-layer two-stream permutation-LM, 12 heads, MLP ratio 4 | | training | K=6 permutations (permuted AR sequence modeling) | | decode | autoregressive greedy + 1 refinement iteration | | dropout / drop-path | 0.1 / 0.1 | | input | 48×512 RGB, unconditional resize, `(x−0.5)/0.5` | | max label length | 100 | | parameters | ~96M | ### Charset A frozen 232-character Cyrillic charset (`sha256 d1b9161e…3ff976`). Token IDs follow string order; specials are `[E]` (id 0) first and `[B]`, `[P]` last. Text is not NFKD-normalized. The set covers the full Ukrainian Cyrillic block (incl. і ї є ґ and the apostrophe), the Russian-only capitals Э Ё and letters ъ ы ё, digits, punctuation, and a small set of Greek/math symbols. Latin lookalikes that are visually identical to Cyrillic letters (`a c e i o y`; `A B C E H K M O P T X`) are deliberately excluded — the target output space is Cyrillic-normalized, so those are folded to their Cyrillic counterparts. ## Training Trained from scratch on ~0.65M train crops materialized over a real + synthetic Cyrillic mix; synthetic:real ≈ 1.15:1. All sources are charset-filtered to the frozen vocab; every crop is treated as a handwritten line. The mix matches the ViT-S variant except for a smaller synthetic cap, to keep the heavier model's wall-time budget comparable. Training was halted at epoch 24 of a planned 35; the published weights are the best-val-CER epoch reached (24), and the planned SWA weight-averaging phase (final 25% of epochs) did not run. | component | language | setting | ≈ contribution | |---|---|---|---| | `ukrhandwritten` | uk | ×3 | ~111k real lines | | `cyrillic_kaggle` (Cyrillic Handwriting Dataset) | ru | ×1 | ~74k real word/phrase crops | | `school_notebooks_RU` | ru | cap 120k | 120k real word crops | | `pumb-ai/synthetic-cyrillic-large` | uk+ru | cap 250k | 250k synthetic lines | | `nastyboget/synthetic_cyrillic_large` | ru | cap 100k | 100k synthetic lines | ### Hyperparameters (as launched) | hyperparameter | value | |---|---| | epochs | 35 planned (halted at 24) | | effective batch size | 512 (micro-batch 128 × accum 4) | | learning rate | 5e-4 peak | | schedule | OneCycleLR, warmup 10% | | optimizer | AdamW, weight_decay 0.05 | | label smoothing | 0.1 | | drop-path rate | 0.1 | | permutations (K) | 6 | | gradient clipping | 20 | | precision | bf16 AMP | | val fraction | 0.02 (hash-stable holdout) | | seed | 42 | ### Online data augmentation Online augmentation was applied during training (`default` profile). Only training crops were augmented — the held-out 2% was never augmented — and every train crop was augmented (`identity_prob = 0.0`), at the 48×512 working resolution (downscaled before augmenting). Each crop was transformed once per epoch by _one geometric_ + _one or two photometric_ operations at random. | pool | transforms (handwritten line crops) | |---|---| | geometric (pick 1) | margin pad 0.02–0.15; width-adaptive rotation (≤±8°); **horizontal shear ±5°**; elastic distortion (α30, σ4.5); baseline warp (amp 4–11% of height, wavelength 250–900 px) | | photometric (pick 1–2) | paper-colour shift; Gaussian noise σ5–15; JPEG q25–65; contrast 0.7–1.2 / gamma 0.6–1.3; morphological erode/dilate (k≤4/3); bleed-through (α0.05–0.12) | ## Results In-mix held-out validation (41,136 crops), greedy + 1 refinement iteration: | metric | value | |---|---| | CER | 0.0118 | | WER | 0.0476 | | exact-match accuracy | 0.9270 | | n_samples | 41,136 | ## Limitations & biases - Single-line handwriting crops only; a general handwriting reader, not tuned to any specific document domain — expect higher error on noisy document scans until fine-tuned. - The charset is Cyrillic-normalized: Latin lookalikes are folded to their Cyrillic twins, so the model never emits the Latin forms. - The training run was halted early (see *Training*), so the checkpoint may be short of the recipe's full potential. - Single seed and validation split — no across-run variance estimate. ## Training data & attribution | dataset | source | license | role | |---|---|---|---| | UkrHandwritten | Kaggle [`annyhnatiuk/ukrainian-handwritten-text`](https://www.kaggle.com/datasets/annyhnatiuk/ukrainian-handwritten-text) | CC BY-SA 4.0 | real Ukrainian lines (×3) | | Cyrillic Handwriting Dataset | Kaggle [`constantinwerner/cyrillic-handwriting-dataset`](https://www.kaggle.com/datasets/constantinwerner/cyrillic-handwriting-dataset) | CC0 | real Russian word/phrase crops | | school_notebooks_RU | HF [`ai-forever/school_notebooks_RU`](https://huggingface.co/datasets/ai-forever/school_notebooks_RU) | MIT | real Russian word crops (cap 120k) | | pumb-ai/synthetic-cyrillic-large | HF [`pumb-ai/synthetic-cyrillic-large`](https://huggingface.co/datasets/pumb-ai/synthetic-cyrillic-large) | Apache-2.0 | synthetic uk+ru lines (cap 250k) | | nastyboget/synthetic_cyrillic_large | HF [`nastyboget/synthetic_cyrillic_large`](https://huggingface.co/datasets/nastyboget/synthetic_cyrillic_large) | MIT | synthetic Russian lines (cap 100k) | **Attribution is mandatory** for UkrHandwritten (CC BY-SA 4.0, share-alike) — please keep this credit if you redistribute or build on this model. Review each dataset's card for its own terms before redistribution. ## License & lineage Model weights are released under CC BY-SA 4.0, inherited from the UkrHandwritten training data (CC BY-SA 4.0, attribution + share-alike — the most restrictive of the training inputs). The PARSeq model code is Apache-2.0 ([`baudm/parseq`](https://github.com/baudm/parseq)).