| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - text2text-generation |
| - flan-t5 |
| - bible |
| - simplification |
| - readability |
| - difficulty-scoring |
| - multi-task |
| - seq2seq |
| datasets: |
| - LoveJesus/passage-difficulty-simplifier-dataset-chirho |
| pipeline_tag: text2text-generation |
| base_model: google/flan-t5-base |
| model-index: |
| - name: passage-difficulty-simplifier-chirho |
| results: |
| - task: |
| type: text2text-generation |
| name: Text Generation |
| metrics: |
| - name: Eval Loss |
| type: eval_loss |
| value: 2.228 |
| - name: Difficulty Accuracy |
| type: accuracy |
| value: 0.9377 |
| - name: Combined Score |
| type: combined_score |
| value: 0.3781 |
| --- |
| |
| <!-- For God so loved the world that he gave his only begotten Son, |
| that whoever believes in him should not perish but have eternal life. - John 3:16 --> |
|
|
| # Passage Difficulty Scorer & Plain-Language Simplifier (Model 8) |
|
|
| A fine-tuned **google/flan-t5-base** (248M parameters) for dual-task Bible passage processing: (1) reading difficulty scoring and (2) archaic-to-modern English simplification. Both tasks are learned jointly through multi-task training on the same model. Upgraded from flan-t5-small (80M) for improved accuracy. |
|
|
| ## Model Description |
|
|
| This model takes Bible passages as input and performs one of two tasks, selected by a natural language prefix: |
|
|
| ### Task 1: Difficulty Scoring |
|
|
| Analyzes a Bible passage and produces a structured difficulty assessment. |
|
|
| - **Prefix**: `rate difficulty:` |
| - **Output format**: `reading_level: [1-12] | vocab_complexity: [low/medium/high] | archaic_forms: [count] | difficulty: [easy/medium/hard]` |
|
|
| ### Task 2: Simplification |
|
|
| Converts archaic or complex Bible passages into plain modern English. |
|
|
| - **Prefix**: `simplify:` |
| - **Output**: Plain-language paraphrase of the input verse |
|
|
| ## Training Details |
|
|
| | Parameter | Value | |
| |---|---| |
| | **Base model** | `google/flan-t5-base` (248M params) | |
| | **Architecture** | Encoder-Decoder (T5) | |
| | **Training approach** | Full fine-tuning, multi-task | |
| | **Trainer** | `Seq2SeqTrainer` with `DataCollatorForSeq2Seq` | |
| | **Epochs** | 5 | |
| | **Batch size** | 32 (H200 GPU) | |
| | **Effective batch size** | 32 (gradient accumulation = 1 on H200) | |
| | **Learning rate** | 2e-4 | |
| | **LR scheduler** | Cosine with 10% warmup | |
| | **Weight decay** | 0.01 | |
| | **Label smoothing** | 0.1 | |
| | **Mixed precision** | bf16 (H200) | |
| | **Max input length** | 256 tokens | |
| | **Max target length** | 256 tokens | |
| | **Early stopping** | Patience = 2, monitoring `eval_loss` | |
| | **Best model selection** | Lowest `eval_loss` | |
| | **Generation (eval)** | `predict_with_generate=True`, beam search | |
|
|
| ### Dataset |
|
|
| Trained on approximately **120K+ examples** combining both tasks, split by Bible book to prevent verse-level leakage (80/10/10 by book): |
|
|
| | Task | Target Count | Description | |
| |---|---|---| |
| | Difficulty scoring | ~64K | Verses from 6 translations with algorithmically computed labels | |
| | Simplification | ~96K | Cross-translation pairs mapping complex to simple English | |
|
|
| #### Translations Used |
|
|
| | Translation | Style | Role | |
| |---|---|---| |
| | KJV (King James Version) | Formal, archaic | Complex source | |
| | ASV (American Standard Version) | Formal, dated | Complex source | |
| | YLT (Young's Literal Translation) | Ultra-literal | Complex source | |
| | Darby Bible | Literal, dated | Complex source / Difficulty scoring | |
| | BBE (Bible in Basic English) | 850-word vocabulary, ~Grade 4 | Simple target | |
| | OEB (Open English Bible) | Modern, public domain | Simple target | |
|
|
| #### Simplification Pairs |
|
|
| | Complex Source | Simple Target | |
| |---|---| |
| | KJV | BBE | |
| | KJV | OEB | |
| | ASV | BBE | |
| | YLT | OEB | |
|
|
| #### Data Source |
|
|
| Bible text sourced from **ScrollMapper Bible Databases** (public domain translations on GitHub). |
|
|
| #### Difficulty Scoring Labels |
|
|
| Labels are computed algorithmically from textual features: |
|
|
| - **Reading level** (1-12): Approximate Flesch-Kincaid grade level analog, adjusted for archaic vocabulary and uncommon word ratio |
| - **Vocabulary complexity** (low/medium/high): Ratio of words outside a ~3,000-word common English vocabulary |
| - **Archaic forms** (count): Number of archaic English words detected (thee, thou, hath, doth, -eth/-est verb endings, etc.) |
| - **Difficulty** (easy/medium/hard): Composite score from reading level, vocabulary complexity, and archaic form count |
|
|
| ## Usage |
|
|
| ### Quick Start: Simplification |
|
|
| ```python |
| # For God so loved the world that he gave his only begotten Son, |
| # that whoever believes in him should not perish but have eternal life. - John 3:16 |
| |
| from transformers import AutoTokenizer, AutoModelForSeq2SeqLM |
| |
| tokenizer_chirho = AutoTokenizer.from_pretrained("LoveJesus/passage-difficulty-simplifier-chirho") |
| model_chirho = AutoModelForSeq2SeqLM.from_pretrained("LoveJesus/passage-difficulty-simplifier-chirho") |
| |
| input_text_chirho = "simplify: And the LORD God formed man of the dust of the ground, and breathed into his nostrils the breath of life; and man became a living soul." |
| |
| inputs_chirho = tokenizer_chirho(input_text_chirho, return_tensors="pt", max_length=256, truncation=True) |
| outputs_chirho = model_chirho.generate(**inputs_chirho, max_length=256, num_beams=4, early_stopping=True) |
| result_chirho = tokenizer_chirho.decode(outputs_chirho[0], skip_special_tokens=True) |
| |
| print(result_chirho) |
| # Expected: A simplified, modern English version of the verse |
| ``` |
|
|
| ### Quick Start: Difficulty Scoring |
|
|
| ```python |
| # For God so loved the world that he gave his only begotten Son, |
| # that whoever believes in him should not perish but have eternal life. - John 3:16 |
| |
| from transformers import AutoTokenizer, AutoModelForSeq2SeqLM |
| import re |
| |
| tokenizer_chirho = AutoTokenizer.from_pretrained("LoveJesus/passage-difficulty-simplifier-chirho") |
| model_chirho = AutoModelForSeq2SeqLM.from_pretrained("LoveJesus/passage-difficulty-simplifier-chirho") |
| |
| input_text_chirho = "rate difficulty: For God so loved the world, that he gave his only begotten Son, that whosoever believeth in him should not perish, but have everlasting life." |
| |
| inputs_chirho = tokenizer_chirho(input_text_chirho, return_tensors="pt", max_length=256, truncation=True) |
| outputs_chirho = model_chirho.generate(**inputs_chirho, max_length=256, num_beams=4, early_stopping=True) |
| raw_output_chirho = tokenizer_chirho.decode(outputs_chirho[0], skip_special_tokens=True) |
| |
| print(raw_output_chirho) |
| # Expected: "reading_level: X | vocab_complexity: Y | archaic_forms: Z | difficulty: W" |
| |
| # Parse structured output |
| reading_level_chirho = re.search(r"reading_level:\s*(\d+)", raw_output_chirho) |
| difficulty_chirho = re.search(r"difficulty:\s*(\w+)", raw_output_chirho) |
| vocab_chirho = re.search(r"vocab_complexity:\s*(\w+)", raw_output_chirho) |
| archaic_chirho = re.search(r"archaic_forms:\s*(\d+)", raw_output_chirho) |
| |
| if reading_level_chirho: |
| print(f"Reading Level: Grade {reading_level_chirho.group(1)}") |
| if difficulty_chirho: |
| print(f"Difficulty: {difficulty_chirho.group(1)}") |
| if vocab_chirho: |
| print(f"Vocabulary Complexity: {vocab_chirho.group(1)}") |
| if archaic_chirho: |
| print(f"Archaic Forms: {archaic_chirho.group(1)}") |
| ``` |
|
|
| ### Batch Inference |
|
|
| ```python |
| # For God so loved the world that he gave his only begotten Son, |
| # that whoever believes in him should not perish but have eternal life. - John 3:16 |
| |
| import torch |
| from transformers import AutoTokenizer, AutoModelForSeq2SeqLM |
| |
| tokenizer_chirho = AutoTokenizer.from_pretrained("LoveJesus/passage-difficulty-simplifier-chirho") |
| model_chirho = AutoModelForSeq2SeqLM.from_pretrained("LoveJesus/passage-difficulty-simplifier-chirho") |
| model_chirho.eval() |
| |
| verses_chirho = [ |
| "simplify: Verily, verily, I say unto thee, Except a man be born again, he cannot see the kingdom of God.", |
| "simplify: Wherefore, as by one man sin entered into the world, and death by sin; and so death passed upon all men, for that all have sinned:", |
| "rate difficulty: In the beginning God created the heaven and the earth.", |
| "rate difficulty: Jesus wept.", |
| ] |
| |
| inputs_chirho = tokenizer_chirho(verses_chirho, return_tensors="pt", max_length=256, truncation=True, padding=True) |
| |
| with torch.no_grad(): |
| outputs_chirho = model_chirho.generate(**inputs_chirho, max_length=256, num_beams=4, early_stopping=True) |
| |
| results_chirho = tokenizer_chirho.batch_decode(outputs_chirho, skip_special_tokens=True) |
| |
| for verse_chirho, result_chirho in zip(verses_chirho, results_chirho): |
| print(f"Input: {verse_chirho}") |
| print(f"Output: {result_chirho}\n") |
| ``` |
|
|
| ## Evaluation |
|
|
| ### Metrics |
|
|
| | Task | Metric | Description | |
| |---|---|---| |
| | Difficulty Scoring | `difficulty_accuracy_chirho` | Exact match on easy/medium/hard label | |
| | Difficulty Scoring | Reading level MAE | Mean absolute error on grade level (1-12) | |
| | Difficulty Scoring | Vocab complexity accuracy | Exact match on low/medium/high | |
| | Simplification | BLEU | Corpus-level BLEU score (sacrebleu) | |
| | Simplification | BERTScore F1 | Semantic similarity to reference simplifications | |
| | Simplification | Exact match | Proportion of predictions matching reference exactly | |
| | Combined | `combined_score_chirho` | 0.4 * difficulty_accuracy + 0.6 * simplification_exact_match | |
| |
| ### Results (v2 - flan-t5-base upgrade) |
| |
| | Metric | Score | |
| |---|---| |
| | **Eval loss** | **2.228** (best at epoch 3) | |
| | **Difficulty accuracy** | **93.8%** | |
| | **Simplification exact match** | 0.50% | |
| | **Combined score** | **0.378** | |
| | Train loss | 1.964 | |
| | Hardware | NVIDIA H200 (143GB), ~64 min | |
| |
| ### Training Trajectory |
| |
| | Epoch | Eval Loss | Difficulty Acc | Combined Score | |
| |-------|-----------|----------------|----------------| |
| | 1 | 2.282 | 87.1% | 0.351 | |
| | 2 | 2.244 | 91.9% | 0.370 | |
| | **3** | **2.228** | 93.8% | 0.378 | |
| | 4 | 2.236 | 94.7% | 0.382 | |
| | 5 | 2.241 | 94.8% | 0.382 | |
| |
| Best model selected by lowest eval_loss (epoch 3). Difficulty accuracy continued improving through epoch 5 but loss began increasing at epoch 4, indicating mild overfitting on the simplification task. |
|
|
| ## Try It Live |
|
|
| **[Interactive Demo on HuggingFace Spaces](https://huggingface.co/spaces/LoveJesus/passage-difficulty-simplifier-chirho)** |
|
|
| The Gradio-powered demo provides two tabs: |
| - **Simplify**: Enter any Bible verse and receive a plain-language version |
| - **Difficulty**: Enter a verse and get reading level, vocabulary complexity, archaic form count, and overall difficulty |
|
|
| ## Limitations |
|
|
| - Trained exclusively on Bible text; does not generalize to other literary or domain-specific texts |
| - Simplification quality varies by verse length and complexity; very long passages may be truncated |
| - Difficulty scoring labels are algorithmically generated (not human-annotated), which introduces systematic biases |
| - Base model (248M params) balances accuracy with accessibility |
| - Simplification targets (BBE, OEB) have their own translation biases; outputs reflect those stylistic choices |
| - Archaic form detection relies on a fixed word list and may miss uncommon archaic constructions |
| - The model does not preserve verse references or theological nuance; it is a readability tool, not a study Bible |
|
|
| ## Intended Use |
|
|
| - Bible study tools that need plain-language paraphrasing of archaic translations |
| - Reading level assessment for curriculum planning or children's ministry materials |
| - Accessibility applications that present Bible text at appropriate reading levels |
| - Research into text simplification for historical English |
|
|
| ## Out-of-Scope Use |
|
|
| - Replacing authoritative Bible translations for doctrinal study |
| - General-purpose text simplification outside of biblical literature |
| - Machine translation between languages (this model operates only in English) |
|
|
| ## Model Architecture |
|
|
| ``` |
| google/flan-t5-base (Encoder-Decoder) |
| Encoder: 12 layers, 12 heads, d_model=768 |
| Decoder: 12 layers, 12 heads, d_model=768 |
| Total parameters: ~248M (all trainable, full fine-tuning) |
| Vocabulary: SentencePiece, 32,128 tokens |
| ``` |
|
|
| ## Repository Structure |
|
|
| ``` |
| passage-difficulty-simplifier-chirho/ |
| src-chirho/ |
| train-chirho/train-simplifier-chirho.py # Training script |
| eval-chirho/evaluate-chirho.py # Evaluation script |
| data-chirho/build-simplifier-dataset-chirho.ts # Dataset builder (Bun/TS) |
| data-chirho/download-translations-chirho.ts # Translation downloader |
| upload-hf-chirho.py # HuggingFace upload script |
| space-chirho/ |
| app.py # Gradio demo application |
| data-chirho/ |
| raw-chirho/ # Raw Bible CSVs |
| processed-chirho/ # JSONL train/val/test splits |
| models-chirho/ |
| simplifier-chirho/best-chirho/ # Best checkpoint |
| cards-chirho/ |
| simplifier-card-chirho.md # This model card |
| config-chirho.yaml # Training configuration |
| spec-chirho/ |
| progress-chirho.sqlite # Agent progress log |
| ``` |
|
|
| ## Training Reproducibility |
|
|
| ```bash |
| # 1. Download Bible translations |
| cd passage-difficulty-simplifier-chirho |
| bun run src-chirho/data-chirho/download-translations-chirho.ts |
| |
| # 2. Build dual-task dataset |
| bun run src-chirho/data-chirho/build-simplifier-dataset-chirho.ts |
| |
| # 3. Train model |
| python src-chirho/train-chirho/train-simplifier-chirho.py |
| |
| # 4. Evaluate |
| python src-chirho/eval-chirho/evaluate-chirho.py |
| |
| # 5. Upload to HuggingFace |
| python src-chirho/upload-hf-chirho.py |
| ``` |
|
|
| ## License |
|
|
| MIT |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{lovejesus2026passagedifficultysimplifier, |
| title={Passage Difficulty Scorer & Plain-Language Simplifier: Multi-Task Flan-T5 for Bible Readability}, |
| author={loveJesus}, |
| year={2026}, |
| publisher={HuggingFace}, |
| url={https://huggingface.co/LoveJesus/passage-difficulty-simplifier-chirho} |
| } |
| ``` |
|
|
| --- |
|
|
| Built with love for Jesus. Published by [loveJesus](https://huggingface.co/LoveJesus). |
|
|
