Stoicheia_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stoicheia — a character-level model for Ancient Greek\n",
    "\n",
    "Stoicheia is a 405M-parameter character-level masked-diffusion encoder for Ancient Greek.\n",
    "Its input is factored into five aligned planes — letters, word/sentence boundaries,\n",
    "diacritics, capitalization, punctuation — and **any of them can be set to *unknown* at\n",
    "inference**. One model therefore reads an edited text, bare *scriptio continua*, and a\n",
    "lacuna of unknown length, changing nothing but its input.\n",
    "\n",
    "This notebook runs the whole release end to end on a free Colab GPU (CPU works too, slower):\n",
    "\n",
    "1. restore a lacuna whose width is *not known* in advance\n",
    "2. pick the restoration model that has provably **never read** your document\n",
    "3. tag and parse a verse of Homer\n",
    "4. macronize and scan a line of verse\n",
    "5. score the macronizer against a hand-annotated benchmark\n",
    "\n",
    "Every model and dataset used below is public.\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "%pip install -q --upgrade transformers huggingface_hub safetensors torch datasets\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Restoring a lacuna of unknown width\n",
    "\n",
    "The hard case in epigraphy and papyrology is a break whose extent is uncertain, in text that\n",
    "carries no accents and no word division. Write `[N±M]` and the model scores every width in\n",
    "`N-M … N+M` by its own confidence, restoring the letters, the accents and the word boundaries\n",
    "together.\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import sys, torch\n",
    "from transformers import AutoModel\n",
    "from huggingface_hub import hf_hub_download, snapshot_download\n",
    "\n",
    "REPO = \"Ericu950/Stoicheia-doc_clean\"   # zero exposure to inscriptions or papyri\n",
    "model = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()\n",
    "\n",
    "# the processor is a plain helper module, not part of the model classes: fetch it first\n",
    "hf_hub_download(repo_id=REPO, filename=\"processing_char_bert.py\", local_dir=\".\")\n",
    "from processing_char_bert import CharBertProcessor\n",
    "proc = CharBertProcessor()\n",
    "\n",
    "# John 1:1 as it would reach us on a damaged, unaccented, unspaced witness\n",
    "damaged = \"εναρχηηνο[5±3]καιολογοςηνπροστονθεον\"\n",
    "best, width, candidates = proc.restore_elastic(model, damaged, mask_dia_boundary=True)\n",
    "print(\"restored :\", best)\n",
    "print(\"width    :\", width, \"characters\")\n",
    "for c in candidates[:5]:\n",
    "    print(\"   \", c)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. The model that has never read your document\n",
    "\n",
    "A single fixed train/test split makes a model useless for exactly the documents an editor\n",
    "cares about. Ten restoration checkpoints are released instead, one per held-out final digit\n",
    "of the PHI/TM identifier: whatever inscription or papyrus you are working on, one of the ten\n",
    "has provably never seen it during fine-tuning, and its backbone never saw a documentary text\n",
    "at all. A reading proposed by *that* model cannot be a memory of the edition you are checking.\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "def model_that_never_read(document_id: str) -> str:\n",
    "    \"\"\"Pick the released checkpoint whose held-out digit matches this document.\"\"\"\n",
    "    digit = str(document_id).strip()[-1]\n",
    "    return f\"Ericu950/Stoicheia-restoration-test{digit}\"\n",
    "\n",
    "for phi in [\"PHI 12345\", \"PHI 293\", \"TM 8100\"]:\n",
    "    print(f\"{phi:12s} -> {model_that_never_read(phi)}\")\n",
    "\n",
    "# use it exactly like the backbone above\n",
    "REPO = model_that_never_read(\"PHI 293\")\n",
    "local = snapshot_download(REPO, allow_patterns=[\"*.py\", \"*.json\"])\n",
    "sys.path.insert(0, local)\n",
    "restorer = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()\n",
    "\n",
    "# note the inputs: τηβου and στεφα...ετης carry no accents and no word division.\n",
    "# accents and spacing are predictions here, not requirements, so the model fills\n",
    "# the gap and decides where the words end in the same pass.\n",
    "print(\"\\n\", proc.restore_respaced(restorer, \"ἔδοξεν τηβου-- καὶ τῷ δήμῳ\"))\n",
    "# -> ἔδοξεν τῇ βουλῇ καὶ τῷ δήμῳ\n",
    "print(proc.restore_respaced(restorer, \"στεφανῶσαι αὐτὸν χρυσῷ στεφα[3±1]ετης ἕνεκα\"))\n",
    "# -> στεφανῶσαι αὐτὸν χρυσῷ στεφάνῳ ἀρετῆς ἕνεκα\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Tagging and parsing\n",
    "\n",
    "Four heads on one shared backbone — factored XPOS, an edit-script lemmatizer, a UPOS\n",
    "auxiliary and a biaffine dependency parser — all from a single forward pass.\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from huggingface_hub import snapshot_download\n",
    "REPO = \"Ericu950/Stoicheia-tagger-parser\"\n",
    "local = snapshot_download(REPO, allow_patterns=[\"*.json\", \"*.txt\", \"*.py\", \"*.model\"])\n",
    "sys.path.insert(0, local)\n",
    "from processing_char_bert_joint import CharBertJointProcessor\n",
    "\n",
    "parser_model = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()\n",
    "jproc = CharBertJointProcessor.from_pretrained(local)\n",
    "\n",
    "words = \"μῆνιν ἄειδε θεὰ Πηληϊάδεω Ἀχιλῆος\".split()\n",
    "batch = jproc([words])\n",
    "with torch.no_grad():\n",
    "    out = parser_model(**batch)\n",
    "rows = jproc.decode(out, batch, ud=True)\n",
    "\n",
    "sent = rows[0] if rows and not isinstance(rows[0], dict) else rows\n",
    "hdr = ('id', 'form', 'lemma', 'upos', 'head', 'deprel')\n",
    "print('%3s %-12s%-12s%-8s%4s %s' % hdr)\n",
    "for i, w in enumerate(sent, 1):\n",
    "    print('%3d %-12s%-12s%-8s%4s %s' % (i, w['form'], w['lemma'], w['upos'], w['head'], w['deprel']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Vowel length and metre\n",
    "\n",
    "Greek orthography never marks vowel length: α, ι and υ — the *dichrona* — are ambiguous.\n",
    "Recovering it (*macronization*) is lexical knowledge, and it is the prerequisite for scanning\n",
    "verse. `Stoicheia-meter` does both at once; `Stoicheia-macronizer` does vowel length alone,\n",
    "slightly better.\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "REPO = \"Ericu950/Stoicheia-meter\"\n",
    "local = snapshot_download(REPO, allow_patterns=[\"*.json\", \"*.txt\", \"*.py\", \"*.model\"])\n",
    "sys.path.insert(0, local)\n",
    "from processing_char_bert_meter import CharBertMeterProcessor\n",
    "\n",
    "meter_model = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()\n",
    "mproc = CharBertMeterProcessor()\n",
    "\n",
    "line = \"ἄνδρα μοι ἔννεπε, μοῦσα, πολύτροπον, ὃς μάλα πολλὰ\"\n",
    "batch = mproc(line)\n",
    "with torch.no_grad():\n",
    "    out = meter_model(**{k: v for k, v in batch.items() if not k.startswith(\"_\")})\n",
    "print(\"macronized:\", mproc.decode_macronization(out, batch))   # _ long, ^ short\n",
    "print(\"scanned   :\", mproc.decode_scansion(out, batch))        # [heavy] {light}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Scoring against the benchmark\n",
    "\n",
    "*Norma Syllabarum Graecarum* is a hand-annotated benchmark of macronization and\n",
    "syllabification. Here we score the dedicated macronizer on its test split — every ambiguous\n",
    "α/ι/υ position, compared against the gold mark.\n"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import json, re\n",
    "from huggingface_hub import hf_hub_download\n",
    "\n",
    "REPO = \"Ericu950/Stoicheia-macronizer\"\n",
    "local = snapshot_download(REPO, allow_patterns=[\"*.json\", \"*.txt\", \"*.py\", \"*.model\"])\n",
    "sys.path.insert(0, local)\n",
    "mac_model = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()\n",
    "\n",
    "path = hf_hub_download(\"Ericu950/norma\", \"data/test.jsonl\", repo_type=\"dataset\")\n",
    "rows = [json.loads(l) for l in open(path, encoding=\"utf-8\")]\n",
    "rows = [r for r in rows if r[\"task\"] == \"macronize\"][:120]   # raise for the full set\n",
    "\n",
    "MARKS = re.compile(r\"[_^]\")\n",
    "n = correct = 0\n",
    "for r in rows:\n",
    "    gold = r[\"text\"]\n",
    "    raw = MARKS.sub(\"\", gold)\n",
    "    batch = mproc(raw)\n",
    "    with torch.no_grad():\n",
    "        out = mac_model(**{k: v for k, v in batch.items() if not k.startswith(\"_\")})\n",
    "    pred = mproc.decode_macronization(out, batch)\n",
    "    for g, p in zip(gold, pred):\n",
    "        pass\n",
    "    # compare mark-by-mark at the positions the gold marks\n",
    "    gi = pi = 0\n",
    "    while gi < len(gold) and pi < len(pred):\n",
    "        if gold[gi] in \"_^\" and pred[pi] in \"_^\":\n",
    "            n += 1; correct += (gold[gi] == pred[pi]); gi += 1; pi += 1\n",
    "        elif gold[gi] in \"_^\":\n",
    "            n += 1; gi += 1\n",
    "        elif pred[pi] in \"_^\":\n",
    "            pi += 1\n",
    "        else:\n",
    "            gi += 1; pi += 1\n",
    "print(f\"macronization accuracy on {len(rows)} lines: {100*correct/max(n,1):.2f}%  ({n} scored positions)\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Everything in the release**\n",
    "\n",
    "| | |\n",
    "|---|---|\n",
    "| 11 pretrained backbones | ten rotated literary folds + one documentary-clean |\n",
    "| 10 restoration checkpoints | one per held-out PHI/TM digit |\n",
    "| tagger-parser, meter, macronizer | fine-tuned from the documentary-clean backbone |\n",
    "| 5 datasets | pretraining corpus, synthetic augmentation, inscriptions, meter silver, benchmark |\n",
    "\n",
    "Training and evaluation code, including the split pipeline that produces the decontamination\n",
    "guarantee, is in the accompanying code repository.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  },
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "accelerator": "GPU"
 },
 "nbformat": 4,
 "nbformat_minor": 0
}