StenToken

StenToken is a fork-trained BPE tokenizer family with separate English and English+code bundles across vocabulary targets from 1,024 to 32,768 tokens. These deliberately small vocabularies explore compact embedding tables and text segmentation; current comparison results do not show StenToken as competitive with the alternatives listed below.

This is a tokenizer repository, not a language-model repository. It contains tokenizer artifacts, the separately scoped StenTokenRuntime-rs inference runtime, and this model card. It does not include model weights, a training script, or the original custom Python source code.

---

Basic Information

StenToken is a compact tokenizer family that uses Capcode to represent capitalization.

Current status and intended use

This first version of StenToken is a stepping stone in StentorLabs’ tokenizer work, not a recommended tokenizer for training a new language model or for production deployment. Use it for tokenizer research, inspection, and learning while better StentorLabs tokenizers are developed.

Important Unicode round-trip limitation

StenToken does not fully guarantee exact Unicode-code-point reversibility. Python case conversion used by its Capcode path can change a character's code-point composition, so decoding may not reproduce the original code-point sequence in every case. If your application requires strict code-point-for-code-point round trips, treat this as a material limitation and validate your own text before adopting StenToken.

Use the tokenizer

Install the Hugging Face Hub client:

pip install huggingface_hub

Download the English or EnglishCode bundle your compatible model requires:

from huggingface_hub import hf_hub_download

repo_id = "StentorLabs/StenToken"
subfolder = "Stentoken/English/8k"

files = {
    name: hf_hub_download(repo_id=repo_id, filename=f"{subfolder}/{name}")
    for name in ("vocab.json", "mergeable_ranks.json", "sten_tokenizer.pt")
}

Load those three files with the matching StenToken Capcode runtime. The repository includes the separately scoped Rust inference runtime in StenTokenRuntime-rs; its compatibility requirements are summarized in the Token-ID Contract.

Choosing a size

These estimates use a model’s core parameter count, excluding embedding parameters, and assume tied input/output embeddings. They were calculated from Chinchilla scaling laws and the filtered-C4 StenToken bytes-per-token results reported below.

Vocabulary	Estimated best core range at 20×	Estimated best core range at 600×
1,024	Below 1.5M	Below 8.03M
2,048	1.5M–5M	8M–25M
4,096	5M–17M	25M–80M
8,192	17M–70M	80M–335M
16,384	70M–290M	335M–≈1.4B
24,576	290M–≈485M	≈1.4B–≈2.4B
32,768	Above ≈485M	Above ≈2.4B

These are exploratory StentorLabs estimates, not established tokenizer-selection rules.

Pick the right bundle

Choose English for English text and EnglishCode for English plus code. For example, the 8K English bundle is in Stentoken/English/8k/; the 8K English-and-code bundle is in Stentoken/EnglishCode/8k/.

Overall assessment

Many StenToken tokens are ordinary subwords, complete words, Capcode forms, technical vocabulary, and code patterns. A small portion is less useful, including memorized numeric identifiers, project-specific code fragments, repeated boilerplate, and awkward incomplete word fragments.

Examples of relatively ordinary tokens:

the
ing
tion
information
classification
conversation
instructions
confidence
dataframe
import
return
array([
strip()
readline()
from django.
db import models
<|tool_call|>
<|fim_prefix|>

Examples of the worst outlier tokens:

8077683550237, 417.
8077683550243, 417.
",CcrackCcontrolCvars(id
pos= CcrackCcontrolCbase
Csection= "Cm2hastCizqWr
0,steelCstress= -4.
slha', 'condition': 0.
of the Clicense at\n#\n#
setChorizontalCstret
http://ww
iloc[i,2]+
CqCsize(45, 15))\n

These are rare, worst-case outliers and are not representative of the vocabulary overall.

Reported bytes-per-token (BPT) comparison

The following table gives the comparison. BPT means bytes per token: the average number of input bytes represented by one token. The table reports the measured values for each corpus.

Tokenizer	Vocab size	C4 filtered bytes/token	C4 unfiltered bytes/token	Python bytes/token
TokenMonster english-1024-balanced-v1	1,024	2.488	1.868	1.932
TokenMonster english-2048-balanced-v1	2,048	3.006	2.163	2.270
TokenMonster english-4096-balanced-v1	4,096	3.582	2.526	2.579
TokenMonster english-8000-balanced-v1	8,000	4.145	2.800	2.885
TokenMonster english-16000-balanced-v1	16,000	4.754	3.164	3.220
TokenMonster english-24000-balanced-v1	24,000	5.107	3.370	3.412
TokenMonster english-32000-balanced-v1	32,000	5.354	3.510	3.531
StenToken English	1,024	1.629	1.418	1.300
StenToken English	2,048	1.805	1.537	1.382
StenToken English	4,096	1.999	1.689	1.455
StenToken English	8,192	2.205	1.851	1.545
StenToken English	16,384	2.391	2.026	1.668
StenToken English	24,576	2.473	2.126	1.741
StenToken English	32,768	2.534	2.202	1.772
StenToken EnglishCode	1,024	1.614	1.417	1.428
StenToken EnglishCode	2,048	1.790	1.542	1.536
StenToken EnglishCode	4,096	1.982	1.692	1.658
StenToken EnglishCode	8,192	2.184	1.858	1.756
StenToken EnglishCode	16,384	2.374	2.044	1.843
StenToken EnglishCode	24,576	2.457	2.132	1.880
StenToken EnglishCode	32,768	2.520	2.206	1.917
Mistral-7B-v0.1	32,000	4.159	3.000	3.093
Llama-2-7b	32,000	4.002	2.918	3.057
Phi-3-mini-4k-instruct	32,000	4.002	2.918	3.057
GPT-2	50,257	4.542	3.504	2.180

The reported BPT results are tokenizer-level measurements. They do not establish downstream model quality, accuracy, or usefulness for training.

Why it is called StenToken

StenToken is named after Stentor, the tiny but smart single-celled organism.

---

Detailed Overview

Everything below is reference material for people comparing tokenizer designs or investigating exact behavior.

Tokenizer Architecture

Component	Value	Notes
Tokenizer family	BPE	Rank-based byte-pair encoding at inference
Published fork targets	1,024–32,768	Seven artifact sizes; trained separately by family
Merge objective	Length-MAX	Frequency weighted by combined merge length
Length exponent	Scheduled, mix-aware	Starts at 1.2; rises to the active family’s endpoint
Semantic gate	Scheduled NPMI	Rejects low-association candidate merges
Subword phase	First	Learns primarily within pre-token units
Superword phase	Second	Adds cross-boundary structure after subword learning
Case representation	Capcode-only	Required throughout the published 1,024–32,768 family
Inference	Rank-based BPE	Applies learned merges in rank order
Stochastic mode	BPE-dropout	Optional training-time segmentation variation

Token-ID Contract

The ID layout is deliberately explicit. A compatible model must preserve these assignments exactly.

ID range	Meaning	Stability requirement
0	Padding	Fixed
1–256	Byte fallback tokens for byte values 0x00–0xFF	Fixed
257+	Control tokens and learned BPE token surfaces	Bundle-specific
Learned token range	BPE vocabulary learned from the training corpus	Must match the exported vocab and merge ranks
vocab.json token IDs	Embedding rows and output logits are indexed by these IDs	Must remain unchanged
mergeable_ranks.json ordering	Changes segmentation and therefore input IDs	Must remain unchanged
sten_tokenizer.pt state	Must match the vocabulary and merge ranks	Bundle-specific
Capcode preprocessing	Mandatory text representation before BPE	Must remain enabled
Special/control token IDs	Changes sequence structure and padding semantics	Must remain unchanged
Padding ID (0)	Affects attention masks and loss masking	Fixed
Artifact revision	Small tokenizer changes are model-breaking changes	Record the exact revision

Reserved control tokens

The tokenizer reserves control-token IDs separately from learned BPE surfaces. Raw user text should be encoded with the ordinary text encoder. Applications that need to insert controls should do so structurally, using their known IDs, rather than relying on a textual occurrence being interpreted as a control token.

This separation prevents a string in user text from silently changing sequence structure.

---

Core Design Choices

1. Length-MAX merge selection

Vanilla BPE selects candidate merges mainly by frequency. StenToken uses a length-weighted objective:

score(a, b) = frequency(a, b) × (length(a) + length(b))^1.8

The exponent is not fixed at 1.8 during the fork pipeline. It starts at 1.2 for the first 65% of merge progress, then rises linearly toward the active family’s mix-aware endpoint. With the default sources, the endpoints are approximately 1.769 for English and 1.725 for English+code; 1.8 is the global ceiling.

The intent is to give a recurring longer unit more credit than a very short pair with the same frequency. That may affect segmentation at compact vocabulary sizes, where spending a learned slot on a longer unit matters more.

2. NPMI semantic merge gate

Candidate pairs must satisfy a normalized pointwise mutual information gate before they can be considered for merging. NPMI is bounded to the interval from -1 to 1, which makes it less dominated by very rare coincidences than raw PMI.

Gate	Default threshold	Role
Subword NPMI	0.4 → 0.2	Relaxes linearly as the vocabulary fills
Superword NPMI	0.5 → 0.2	Starts stricter, then relaxes over merge progress
Fork minimum pair frequency	50	Effective floor for every default target below 65,536
Dynamic frequency floor	2e-7 of corpus scale	Raises the effective floor when corpus scale requires it

The gate is intended to screen out one class of undesirable token: a frequent-looking sequence that has little stable semantic or structural relationship. It cannot prove that a token is linguistically meaningful.

3. Two-phase BPE

StenToken does not open all possible cross-boundary merges immediately.

Target range	Subword allocation	Superword allocation
1,024–16,384	93.75%	6.25%
24,576–32,768	87.5%	12.5%

The subword allocation learns word-internal morphology, common spellings, and code fragments. The remaining slots are reserved for cross-boundary superword units. Delaying cross-boundary merges is a guardrail: if they are available from the first merge, broad fragments such as a leading space plus a common word can absorb capacity too early and crowd out internal structure.

4. Picky-BPE and LiteToken cleanup

The training pipeline includes two vocabulary-management mechanisms:

• Picky-BPE pruning: a scaffold token can be removed during training when a merge explains nearly all of its usage.
• LiteToken residue handling: tokens frequently created as merge targets but rarely used in final segmentation can be identified as residues.

These mechanisms are intended to manage vocabulary allocation. They do not guarantee exact reversibility.

5. Rank-based BPE inference

At encoding time, StenToken applies the lowest-ranked available merge repeatedly within each pre-tokenized chunk. This is ordinary rank-based BPE behavior rather than a greedy search over arbitrary learned strings.

---

Vocabulary Training

Streaming-first corpus collection

The training system streams source datasets rather than downloading the full corpus to local disk. It uses bounded retention and checkpointing so a large run can continue after a runtime interruption.

Training control	Default	Why it exists
Shared corpus target	150 GB of text	Normal fork-pipeline default
English family word types	1,200,000 → 900,000	Effective cap and prune target
English+code family word types	1,500,000 → 1,200,000	Effective cap and prune target
Checkpoint cadence	5 GB or 20 min	Makes long runs resumable
Sentence-cache cap	150,000 types	Bounds cross-boundary statistics
BPE dropout protected ranks	First 500	Keeps foundational merges stable

The published training process derives both family variants, then branches each family to its target sizes. Checkpoints are training artifacts, not runtime dependencies for deployment.

Training objective details

Candidate merge scoring combines frequency, length, and association quality. The system tracks document/source sketches, pair counts, and pre-token statistics in an attempt to reduce the chance that a small number of repeated documents dominates candidate selection.

The implementation uses indexed mutable BPE state and periodically rescored pair heaps to avoid rebuilding the full candidate table after every merge. This is intended to reduce work at larger corpus sizes, but it does not establish practical performance or eliminate resource requirements.

Reproducibility boundaries

The default seed is 42. Exact reproduction can still differ when dataset snapshots, streaming order, library versions, source availability, and interruption/resume points differ. Tokenizer training should be considered reproducible only when the full artifact set, source revisions, configuration, and environment are controlled.

---

Data Pipeline

The shared input mixture is English-heavy and includes Python code:

Source	Configuration	Weight	Intended contribution
HuggingFaceFW/fineweb	sample-10BT, train	50%	General English web text
wikimedia/wikipedia	20231101.en, train	20%	Encyclopedic English
fddemarco/pushshift-reddit-comments	train	10%	Informal conversational text
Cyrile/dataset-the-stack-v2-dedup-sub	Python, train	20%	Python code

The percentages are configured shared-stream weights, not a guarantee of exact final byte proportions. The normal fork pipeline derives two output families from this stream:

Output family	Sources used	Effective family weights
English	FineWeb, Wikipedia, Reddit	62.5% / 25% / 12.5% after normalization
English+code	All four shared sources	50% / 20% / 10% / 20%

Streaming sources can contain unavailable shards, changed revisions, formatting variation, duplicate content, and text that fails quality gates.

Quality and normalization controls

Before vocabulary statistics are accumulated, the pipeline can apply lightweight checks and normalization:

• Unicode normalization with punctuation normalization.
• Source-specific code handling; general web text preserves Unicode by default.
• Bounded control-character ratio.
• Repeated-character-run handling.
• Text-length and allowed-character checks.
• Rolling document deduplication.
• Code-aware splitting for identifiers such as snake_case and CamelCase.
• Preservation of whitespace in pre-tokenization.

These filters are pragmatic data engineering. They are not a safety filter and they do not guarantee data quality, legality, representativeness, or absence of personal information in upstream corpora.

---

Text Processing

Unicode and byte fallback

Input text is represented through UTF-8 bytes internally. The learned vocabulary contains byte-space surfaces, while byte fallback preserves coverage for anything not represented by a learned merge.

This means StenToken can encode non-English text, emoji, unusual punctuation, and arbitrary byte sequences. It does not mean it is optimized for them.

Capcode

StenToken is a Capcode tokenizer. Capcode is the capitalization representation used throughout the published 1,024–32,768 family. It is applied before StenToken’s pre-tokenization and ranked BPE stages.

• It reduces duplicated vocabulary capacity across lowercase, title-case, and uppercase forms.
• It preserves casing rather than simply lowercasing input.
• It gives the tokenizer a consistent representation for all-caps spans, identifiers, and ordinary words.

Inspiration and attribution

Thank you to Alasdair Forsythe for the Capcode tokenizer inspiration. StenToken’s custom Capcode system is a tweaked version of the TokenMonster approach: it keeps the central idea of sharing lowercase, title-case, and uppercase vocabulary capacity while adding behavior for mixed prose, source code, identifiers, acronyms, and technical text.

Code-aware identifier decomposition

StenToken recognizes common identifier structure, including CamelCase, PascalCase, snake_case, acronym transitions, and letter-and-number boundaries. This lets an identifier such as the following expose component pieces instead of becoming one opaque unit:

getHTTPResponse

That can create overlap across related identifiers:

getHTTPResponse
parseHTTPResponse
HTTPResponseCode
http_response

This identifier decomposition is an additional StenToken capability for code-heavy data; it goes beyond capitalization encoding alone.

Consecutive uppercase spans

StenToken adds a block-capitalization representation using B and E markers. A whitespace-separated span such as the following can use one uppercase block rather than a separate capitalization instruction before every word:

UNITED STATES NAVY
→ Bunited states navyE

This is intended to reduce capitalization-marker overhead for headings, acronyms, constants, legal text, and technical labels before BPE begins. It does not by itself guarantee a lower final token count, because learned BPE merges can segment either representation differently.

Honest limits

StenToken’s Capcode is not inherently better merely because it is newer or more elaborate. It has concrete limits:

• Its custom C, W, B, and E marker language requires the StenToken decoder; it is not interchangeable with TokenMonster’s native decoder.
• Uppercase-block detection is optimized for whitespace-separated words. Punctuation-separated spans such as NASA-USA-API do not receive the same block treatment.

Future improvements include formally lossless Unicode handling and explicit validation or constraints for valid C, W, B, and E sequences.

Pretokenization

The pre-tokenizer preserves whitespace and handles several text/code patterns deliberately:

Pattern	Intended handling
Whitespace	Preserved rather than discarded
Contractions	Kept as structured text units
Python identifiers	Can be split around meaningful case/underscore components
Operators	Common multi-character operators treated atomically
Keywords with trailing space	Recognized to support code structure
Repeated-character runs	Isolated or bounded to avoid pathological learned units

No pre-tokenization scheme is neutral. These choices prioritize English prose and Python-like code, and may be less appropriate for languages or data formats with different boundaries.

---

---

Ethical Considerations & Societal Impact

Inherited data biases

The tokenizer’s vocabulary is shaped by its source mixture. Its web and code sources can overrepresent some dialects, topics, technical conventions, and online communities while underrepresenting others.

Tokenizer bias can appear as uneven compression: a form of text that uses fewer tokens may receive more effective context capacity in a fixed-window model than a form that fragments into many byte-level pieces.

Content and privacy

Upstream web-scale sources can include harmful, incorrect, sensitive, or personal content. A tokenizer does not retain documents as a retrieval database, but its vocabulary-learning process is still derived from corpus statistics. Use appropriate data governance, licensing review, and privacy practices when retraining or extending it.

---

Performance and Resource Considerations

Runtime tokenization cost depends on text length, merge-table size, backend, batching, and hardware. The artifact itself is much smaller than a language model, but tokenizer throughput can still matter in high-volume preprocessing.

---

Related Work

Work	Relationship
Byte Pair Encoding (BPE)	Base merge-based vocabulary method
BPE-dropout	Stochastic merge dropping for segmentation augmentation
PMI/NPMI association measures	Used to assess merge association strength
Picky BPE	Influences vocabulary-pruning strategy
Subword regularization	Broader family of stochastic segmentation approaches

---

Environmental Impact

Tokenizer training consumes compute through streaming, statistics collection, pair scoring, checkpointing, and merge selection. This repository does not publish a verified energy or carbon measurement for a particular completed tokenizer run.

---

Citation

If you use StenToken in research or a project, please cite the tokenizer repository and record the exact artifact revision:

@software{stentoken,
  title        = {StenToken: Streaming Length-MAX Byte-Fallback BPE Tokenizer},
  author       = {StentorLabs},
  year         = {2026},
  url          = {https://huggingface.co/StentorLabs/StenToken},
  note         = {Tokenizer artifacts; record the exact revision used}
}

---

StenToken Card Contact

Questions, benchmarks, or feedback: StentorLabs@gmail.com or open a discussion.

Made with ❤️ by StentorLabs

Democratizing AI through accessible, efficient models — trained on free compute, shared with everyone.