README.md

---
license: apache-2.0
language:
- ar
base_model: oddadmix/50M-2048-Emhotob
pipeline_tag: text-classification
library_name: transformers
tags:
- arabic
- rule-checking
- text-classification
- compliance
- moderation
- tiny-model
- on-device
- llama
datasets:
- oddadmix/arabic-rule-checking
---

# Nawah-RuleCheck-v2 — مدقّق قواعد عربي بـ 207.2 ميجابايت

A **51,787,264-parameter** Arabic rule checker; weights file **207,161,320 bytes
(207.16 MB)**. Give it a text and a rule written in ordinary Arabic and it answers
**مطابق** (the text satisfies the rule) or **مخالف** (it does not).

> **بالعربية:** نموذج عربي صغير (51,787,264 معامل) يأخذ نصًا وقاعدة مكتوبة بلغة طبيعية
> ويقرّر إن كان النص مطابقًا للقاعدة أم مخالفًا لها.

12 transformer layers, hidden size 512. It is a sequence classifier
(`LlamaForSequenceClassification`), not a generator — the answer is two-valued, so a classification
head fits better at this size and is directly scorable.

**Where the parameters are.** 35,403,264 of the 51,787,264 parameters are
the transformer body; 16,384,000 (31.6%) are the 32,000 × 512
embedding table. Small Arabic models are mostly vocabulary — worth knowing before reading the
results as reasoning capacity.

## The size ladder

This model is one rung of a four-point ladder trained on identical data and scored by one harness
in one session ([`eval_all.py`](https://huggingface.co/oddadmix/Nawah-RuleCheck-v2/blob/main/eval_all.py)). Nothing below
is carried over from an earlier run or another card.

| model | params | weights | seen wording | unseen wording | minimal pairs | hand-written wording | CPU ms |
|---|---:|---:|---:|---:|---:|---:|---:|
| [`Nawah-RuleCheck-500K`](https://huggingface.co/oddadmix/Nawah-RuleCheck-500K) | 518,256 | 2.08 MB | 0.9866 | 0.9778 | 0.9926 | 0.5558 | 0.33 |
| [`Nawah-RuleCheck-1M`](https://huggingface.co/oddadmix/Nawah-RuleCheck-1M) | 1,073,504 | 4.30 MB | 0.9952 | 0.9864 | 0.9991 | 0.7708 | 0.55 |
| [`Nawah-RuleCheck-5M`](https://huggingface.co/oddadmix/Nawah-RuleCheck-5M) | 5,080,704 | 20.33 MB | 0.9979 | 0.9949 | 0.9926 | 0.8642 | 1.06 |
| **this model** | **51,787,264** | **207.16 MB** | **0.9980** | **0.9980** | **0.9991** | **0.8883** | **16.58** |
| majority baseline | — | — | 0.6401 | 0.6401 | 0.5000 | 0.5925 | — |

*CPU ms = single example, batch 1, float32, 2 threads, median of 200 runs, same input for every
model. On an RTX 5090 this model is 3.74 ms.*

### What the four columns mean

- **seen wording** (6,624 pairs) — held-out texts (`task_id` split, zero text overlap
  with training), rules phrased the way training phrased them. Text generalisation.
- **unseen wording** (6,624 pairs) — the same held-out texts, rule phrasings held out of
  training entirely, drawn from the same 631-paraphrase pool.
- **minimal pairs** (1,080 pairs) — a real text and a surgically edited copy whose
  verdict flips, verified by `rules_common.py`. Any cue that merely correlates with the label
  inside the corpus dies here.
- **hand-written wording** (1,200 pairs) — 10 rules restated in terse, colloquial Arabic
  written from scratch, sharing almost no vocabulary with the paraphrase pool. This is the honest
  hard column, and it is where model size actually buys something.

## Per-rule accuracy, unseen wording

| rule | acc |
|---|---:|
| `ends_question` | 1.000 |
| `has_date` | 1.000 |
| `has_number` | 1.000 |
| `has_phone` | 1.000 |
| `has_price` | 1.000 |
| `max_words_30` | 1.000 |
| `max_words_50` | 1.000 |
| `min_words_30` | 1.000 |
| `no_email` | 1.000 |
| `no_latin` | 1.000 |
| `no_url` | 1.000 |
| `has_city` | 0.999 |
| `no_excess_punct` | 0.999 |
| `no_phone` | 0.996 |
| `min_words_25` | 0.975 |
| `min_words_15` | 0.952 |
| `max_words_25` | 0.950 |
| `min_words_20` | 0.943 |
| `max_words_40` | 0.870 |

## Per-rule accuracy, hand-written wording

| rule | acc |
|---|---:|
| `no_latin` | 1.000 |
| `no_url` | 1.000 |
| `has_price` | 1.000 |
| `no_excess_punct` | 1.000 |
| `ends_question` | 1.000 |
| `no_phone` | 0.992 |
| `has_date` | 0.992 |
| `has_number` | 0.933 |
| `has_phone` | 0.525 |
| `no_email` | 0.442 |

## Limitations

Read the **hand-written wording** column, not the headline. Inside the paraphrase distribution the
whole ladder is bunched between 0.9778 and 0.9980 — a 2.0-point spread across a 100x size
range. On rule wordings written from scratch the same four models spread from 0.5558 to 0.8883,
a 33.2-point gap, and the 500K rung falls *below* the 0.5925 majority-class baseline. Almost all of
what size buys on this task is robustness to wording you did not train on. If your rules are a fixed
catalogue you can phrase in the training register, the small rungs are close to free. If users will
phrase rules in wording you do not control, size matters and **no rung on this ladder is finished**.

Trained on 19 rules decidable from the string itself. Rules needing world knowledge, judgement or
multi-step inference are out of distribution. Texts are 1–3 line Arabic business documents
(classified ads, support tickets, job posts, complaints, rental listings) across nine regions;
longer or very different text is untested. Word-count rules are the weakest family — counting is
the one operation here that cannot be pattern-matched.

## Usage

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

M = "oddadmix/Nawah-RuleCheck-v2"
tok = AutoTokenizer.from_pretrained(M)
model = AutoModelForSequenceClassification.from_pretrained(M).eval()

def check(text, rule):
    x = tok(f"النص: {text}\nالقاعدة: {rule}", return_tensors="pt", add_special_tokens=False)
    with torch.no_grad():
        p = model(**x).logits.softmax(-1)[0]
    return model.config.id2label[int(p.argmax())], float(p.max())

check("للبيع سيارة نظيفة، للتواصل على الرقم 0551234567",
      "يُمنع ظهور أي رقم هاتف (8-15 خانة) في النص")
```

Input format is `النص: {text}\nالقاعدة: {rule}` with `add_special_tokens=False`. Text first, rule
second: the head pools the last non-pad token, so under causal attention only the trailing rule
tokens can attend to the whole text.

## Training

Base [`oddadmix/50M-2048-Emhotob`](https://huggingface.co/oddadmix/50M-2048-Emhotob) — Llama,
12 layers, hidden 512, vocab 32,000, tied embeddings, 2048 context.
159,240 (text, rule) pairs, **LR 1e-4, 10 epochs**, cosine schedule,
500 warmup steps, effective batch 64, bf16, `max_length` 224. Best checkpoint
selected on **unseen-wording macro-F1**.

### Learning rate across the ladder

"Smaller models need a higher learning rate" is true here — but **only below 5M**. Each rung was
swept independently on the same data and the same eval:

| rung | params | chosen LR | epochs | tuned? |
|---|---:|---:|---:|:--|
| `Nawah-RuleCheck-500K` | 518,256 | **6e-3** | 15 | ✅ |
| `Nawah-RuleCheck-1M` | 1,073,504 | **1e-3** | 10 | ✅ |
| `Nawah-RuleCheck-5M` | 5,080,704 | **1e-4** | 10 | ✅ |
| `Nawah-RuleCheck-v2` | 51,787,264 | **1e-4** | 10 | ✅ |

The optimum climbs **60×** between the 5M and the 500K (1e-4 → 6e-3), and getting it wrong is
expensive: on the 500K base the big model's recipe scores **0.9179** unseen against 0.9780 tuned —
six points that look like a capacity limit and are not.

Above 5M the effect simply stops. The 5M and the 51.8M want the **same 1e-4**, and pushing the 5M
up toward the 500K's learning rate destroys it (3e-3 → 0.88, 6e-3 → 0.77 unseen). So neither
recipe transfers in either direction: the tiny rungs need their own LR, and that LR must not be
carried back up the ladder.

### This rung's sweep

Same data, same eval splits, same everything — only LR and epoch budget move:

| LR | epochs | seen | unseen |
|---:|---:|---:|---:|
| 3e-4 | 3 | 0.9929 | 0.9799 |
| 3e-5 | 10 | 0.9973 | 0.9964 |
| 1e-4 | 10 | 0.9980 | 0.9980 |  ← these weights
| 3e-4 | 10 | 0.9962 | 0.9852 |
| 1e-4 | 15 | 0.9985 | 0.9959 |

This rung originally shipped at 3e-4 / 3 epochs, carried over from the Emhotob SFT ladder without a sweep, and it was leaving real accuracy on the table: 0.9799 unseen against 0.9980 here, and 0.8667 against 0.8883 on hand-written wording. 1e-4 is bracketed on both sides (3e-5 -> 0.9964, 3e-4 -> 0.9852) and a longer budget does not help (15 epochs -> 0.9959). These weights are the retuned ones.

## How the labels were made

Labels are **computed, not model-judged**. Each rule is a deterministic predicate over the raw
string (`rules_common.py`, shipped here), so ground truth is exact by construction. That also makes
paraphrasing free supervision: rewording a rule cannot change its verdict.

## Siblings

- [`oddadmix/Nawah-RuleCheck-500K`](https://huggingface.co/oddadmix/Nawah-RuleCheck-500K) — 518,256 params
- [`oddadmix/Nawah-RuleCheck-1M`](https://huggingface.co/oddadmix/Nawah-RuleCheck-1M) — 1,073,504 params
- [`oddadmix/Nawah-RuleCheck-5M`](https://huggingface.co/oddadmix/Nawah-RuleCheck-5M) — 5,080,704 params
- [`oddadmix/Nawah-RuleCheck-v2`](https://huggingface.co/oddadmix/Nawah-RuleCheck-v2) — 51,787,264 params

## Reproducing every number on this page

```bash
python eval_all.py oddadmix/Nawah-RuleCheck-500K oddadmix/Nawah-RuleCheck-1M \
                   oddadmix/Nawah-RuleCheck-5M  oddadmix/Nawah-RuleCheck-v2
```

`eval_all.py` is in this repo, along with the frozen case sets it scores against
(`cf_cases.json`, `ood_cases.json`) so the minimal pairs and the hand-written wordings are
byte-identical for every rung and every re-run. Also here: `prepare_rules_cls.py`,
`train_rules_cls.py`, `gen_rule_paraphrases.py`, `counterfactual_eval.py`, `rules_common.py`
(which *defines* the labels), and the 631 rule paraphrases in `rule_paraphrases.json`.