---
license: apache-2.0
library_name: coreml
language:
  - ja
pipeline_tag: image-to-text
tags:
  - manga
  - ocr
  - japanese
  - coreml
  - ios
base_model:
  - kha-white/manga-ocr-base
  - ogkalu/comic-text-and-bubble-detector
---

# nemu on-device manga OCR (Core ML)

These are the Core ML models that [nemu](https://github.com/nemu-pm/nemu)'s iOS app bundles for fully on-device Japanese manga OCR. They power the reader's Japanese-learning plugin. The pipeline has three stages:

1. **Detect.** `ogkalu-ctbd-v4s-fp16.mlpackage` is a text and speech-bubble detector: RT-DETRv4-S from [ogkalu/comic-text-and-bubble-detector](https://huggingface.co/ogkalu/comic-text-and-bubble-detector), with a 640×640 input. It finds the text blocks on a page. It was built from `detector-v4-s_int8.onnx` at revision `16e8a622`: dequantised, then converted to an fp16 ML Program.
2. **Order.** Blocks are put into manga reading order in the app. This step is code, not a model.
3. **Recognise.** `mangaocr-encoder-int8.mlpackage` and `mangaocr-decoder-int8.mlpackage` read each block. They are [kha-white/manga-ocr-base](https://huggingface.co/kha-white/manga-ocr-base) split into a ViT encoder and a 2-layer BERT decoder, which the host runs step by step with greedy decoding. Weights are int8 linear, per channel; activations stay fp16. `vocab.txt` is the upstream tokenizer vocabulary.

| file | size | notes |
|---|---|---|
| `ogkalu-ctbd-v4s-fp16.mlpackage` | ~20 MB | raw per-query outputs: `logits [1,300,3]` (bubble, text_bubble, text_free) and `boxes [1,300,4]` (normalised cx, cy, w, h). Post-processing runs in Swift. |
| `mangaocr-encoder-int8.mlpackage` | ~82 MB | input: a 224×224 RGB crop, preprocessed with an antialiased bilinear resize to match PIL. |
| `mangaocr-decoder-int8.mlpackage` | ~24 MB | flexible `input_ids` length 1–300. |
| `vocab.txt` | 24 KB | |

Every package targets iOS 17 or later and runs on the Neural Engine, GPU or CPU. In the iOS Simulator, use `.cpuOnly`.

## Quality

The benchmark set is 16 raw manga pages with 148 hand-transcribed dialogue and narration blocks. The fully on-device pipeline, measured in the nemu app, scored:

| metric | score |
|---|---|
| Normalised CER | **1.1%** (Apple Vision baseline: 18.7%) |
| Block recall | 98% |
| Exact blocks | 95% |
| Pages in correct reading order | 15/16 |

For recognition, the Swift implementation's output matches the Python Core ML reference on 166/166 crops. The int8 model matches upstream PyTorch greedy decoding on 164/166 crops.

## Reproducing

The scripts in `scripts/` rebuild the packages, using coremltools 9, torch, transformers and onnx2torch:

```sh
export NEMU_OCR_WORKDIR=$PWD/work   # expects work/models/<upstream weights>, writes work/coreml/
python scripts/convert_mangaocr.py fp16          # manga-ocr → encoder/decoder fp16
python scripts/quantize_int8.py                  # fp16 → int8 weights
python scripts/dequantize_int8_onnx.py detector-v4-s_int8.onnx detector-v4-s.fp32.onnx
python scripts/convert_v4s_coreml.py detector-v4-s.fp32.onnx work/coreml/ogkalu-ctbd-v4s
```

The nemu build pins this repository by commit and checks every file against a sha256 hash (`apps/mobile/modules/nemu-japanese-learning/scripts/fetch-ocr-models.ts`). It then compiles the packages with `xcrun coremlcompiler` and bundles them into the app.

## Licence and attribution

Both upstream models are released under Apache-2.0, and so are these converted derivatives. See `NOTICE` for full attribution.

- **manga-ocr-base**: © Maciej Budyś (kha-white). It was trained on Manga109-s (Aizawa-Yamasaki-Matsui Laboratory, The University of Tokyo) and synthetic data. No Manga109 images are included here.
- **comic-text-and-bubble-detector**: © ogkalu. The model card describes ~11k manga, webtoon, manhua and comic training images; it does not publish a full list of image sources.

Changes made by nemu: conversion to Core ML, splitting the encoder and decoder, int8 weight quantisation of the recogniser, and dequantising and cutting the detector's ONNX graph before its post-processor.