Publish bounded OpenWhispr consumer parity and warmup qualification

coreml/CONSUMER-PARITY-20260924.md

@@ -0,0 +1,88 @@
+# OpenWhispr consumer inference parity — September 24, 2026
+
+**Result: the consumer inference calls match the qualified SDK on the frozen 400-clip panel and 89 additional long recordings. The Orukeet warmup preserves all observed transcripts and token timings. The model remains experimental.**
+
+This is a separate Mac reproduction of the app’s native inference sequence. It closes the previously untested call-path comparison at the scope below; it does not rerun all 20,146 recordings through the app or qualify Expo, microphone capture, installation, diarization or a physical iPhone.
+
+## Frozen scope and results
+
+Both runs used the same existing compiled INT8 artifact, pinned FluidAudio source and original 16 kHz mono FLEURS WAV bytes. The cohort was sealed before inference: all 400 previously frozen short clips in original order, every one of the 64 full-corpus recordings longer than 30 seconds, and the first original-metadata recording longer than 15 seconds in each language. Deduplication yields 89 long recordings. The first short recording was repeated after all other recordings, for 490 observations per run.
+
+| Cohort | Observations per variant | Exact consumer/SDK transcripts | WER, all variants | CER, all variants |
+|---|---:|---:|---:|---:|
+| Frozen 25-language panel | 400 | 400/400 | 1,072/7,650 = 14.0131% | 2,713/49,624 = 5.4671% |
+| Long-recording smoke | 89 | 89/89 | 517/3,821 = 13.5305% | 1,350/25,838 = 5.2249% |
+| First recording repeated after the others | 1 | 1/1 | 1/14 = 7.1429% | 1/74 = 1.3514% |
+
+Each variant completed **490/490 successfully**. All converted sample counts matched the original source counts, and the consumer converter’s Float32 PCM hashes matched the SDK converter’s hashes for every recording. All 64 disk-backed long-file cases matched. No manual padding, truncation, duration filtering or retries were applied. The selected WAV hashes also match the completed full-corpus observations.
+
+All 490 results contained token timings with finite confidence and ordered, nonnegative start/end times. Baseline and warmup outputs matched exactly for text, every token-timing field and reported duration. These checks establish preservation; they do not establish alignment accuracy against manually labeled timestamps. Independent Kaldialign word and character edit distances matched the primary scores.
+
+| Language | Short clips | Exact transcripts, each variant | Consumer/SDK WER |
+|---|---:|---:|---:|
+| Bulgarian | 16 | 16/16 | 15.484% |
+| Czech | 16 | 16/16 | 12.605% |
+| Danish | 16 | 16/16 | 23.129% |
+| German | 16 | 16/16 | 5.528% |
+| Greek | 16 | 16/16 | 33.528% |
+| English | 16 | 16/16 | 4.956% |
+| Spanish | 16 | 16/16 | 4.032% |
+| Estonian | 16 | 16/16 | 18.957% |
+| Finnish | 16 | 16/16 | 12.288% |
+| French | 16 | 16/16 | 8.355% |
+| Croatian | 16 | 16/16 | 8.626% |
+| Hungarian | 16 | 16/16 | 19.549% |
+| Italian | 16 | 16/16 | 5.413% |
+| Lithuanian | 16 | 16/16 | 22.535% |
+| Latvian | 16 | 16/16 | 18.504% |
+| Maltese | 16 | 16/16 | 20.270% |
+| Dutch | 16 | 16/16 | 8.046% |
+| Polish | 16 | 16/16 | 11.828% |
+| Portuguese | 16 | 16/16 | 7.808% |
+| Romanian | 16 | 16/16 | 13.764% |
+| Russian | 16 | 16/16 | 7.850% |
+| Slovak | 16 | 16/16 | 14.521% |
+| Slovenian | 16 | 16/16 | 42.222% |
+| Swedish | 16 | 16/16 | 18.855% |
+| Ukrainian | 16 | 16/16 | 5.018% |
+
+The remaining Slovenian script error is preserved, not repaired, by these equivalent inference paths. Full-corpus recognition evidence remains 23/834 INT8 versus 20/834 LUT6 outputs containing Cyrillic, including 16 and 14 majority-Cyrillic outputs. Neither the installer changes nor silent warmup resolves that limitation.
+
+## Actual configuration differences
+
+| Behavior | Qualified SDK Engine | Consumer native module |
+|---|---|---|
+| Model loader | `OrukeetLocalModels.load` | Same loader |
+| Placement | CPU + Neural Engine; CPU preprocessor | Same defaults |
+| Decoder/configuration | v3, unconditioned greedy; 16 kHz; four concurrent chunks | Same defaults; language argument explicitly nil for Orukeet |
+| Preparation | Initializes manager with models and runs 4,800 silent samples | Original `AsrManager()` + `loadModels()` only assigns models; warmup helper adds the same bounded silent inference |
+| Recording input | SDK audio conversion then PCM-array API | File-URL API; disk-backed conversion above 480,000 samples (30 seconds) |
+| Real-recording state | New TDT decoder state for every call | New TDT decoder state for every call |
+| Output | Trims surrounding whitespace | Preserves manager text and token timings; all observed raw text matched |
+
+The manager’s default chunk concurrency is four, streaming is enabled at 480,000 samples, mel chunk context is enabled and dual-decode arbitration is disabled. No default was changed to produce the match. The inference harness calls `AsrManager()`, `loadModels`, and `transcribe(fileURL, decoderState: &state, language: nil)` as the consumer does. Its independent conversion audit does not replace the URL passed to inference.
+
+## Orukeet-only warmup verification
+
+The app helper runs exactly 4,800 zero Float samples (300 ms) after loading models and before exposing the prepared manager. It creates a local decoder state, discards both the result and that state, and checks cancellation before and after inference. The app cleans up and fails preparation if warmup fails. The real recording continues to create fresh state. Other model branches are unchanged. The helper was compiled verbatim from the app worktree for the second 490-observation run.
+
+Helper SHA-256: `6ce47db6c0b49b8d9818f744a4c96d1297205e0d3837c0c129ae2752faae30e9`.
+
+Six additional fresh-process runs, three per mode in a predeclared interleaved order, each transcribed the same frozen recording twice. All 12 real outputs and token timings matched the original baseline. These use existing compiled models and OS caches on Apple M5 Max / macOS 26.4.1; they are small host diagnostics, not cold-install or phone measurements.
+
+| Fresh-process diagnostic, three runs per mode | Without silent warmup | With silent warmup |
+|---|---:|---:|
+| Median first real transcription call | 63.06 ms | 45.69 ms |
+| Median repeated transcription call | 46.30 ms | 43.63 ms |
+
+Warmup moves the first prediction into preparation. It does not remove that computation or establish a general end-to-end speedup. No phone latency, thermal or memory claim follows from this diagnostic.
+
+## Source identity and remaining gates
+
+- Consumer baseline: `OpenWhispr/openwhispr` commit `03734fbb51eca350986a41fd3c4dfabb78675330`. Warmup variant: commit `0d796516c126869e881873e1a93f117be2cac6df`, with the exact helper above and its Orukeet-only prepare call.
+- Consumer SDK pin: `6c37c587fabcef8b0e932584fbf22a79fcb9d89e`; the report harness uses Engine/Audio byte-identical to that pin and the full-corpus qualified `db30e0b1c27ea0c88ae789473fff6704fc3b4445` sources. The locally observed SDK checkout head `decfb6e4b50a41fbddadf8a9cf12beadf06ae2f1` changes documentation only.
+- FluidAudio: `ddc95f4d03d5be12bf84eeb6e4bde5b724356d88`. Dependencies are existing local source checkouts; no model files were downloaded or copied.
+- INT8 portable artifact SHA-256: `24df9ff76f00f86f9ae1fd601cbbcab1d1eac98c7e8107de67444a7858d88b8b`.
+- Source dataset revision: `google/fleurs` `70bb2e84b976b7e960aa89f1c648e09c59f894dd`, original test recordings; CC-BY-4.0. The panel has model-development exposure and is not an unseen generalization test.
+- Detailed sealed manifests, source copies/hashes, raw attempt records, conversion hashes, token timings and independent scoring receipts are retained locally.
+- Keep the release experimental and opt-in. Recognition/script acceptance and physical-iPhone validation remain separate gates. This result supports the consumer call path on 489 unique recordings, not all supported audio formats, continuous dictation or the complete app lifecycle.

coreml/README.md

@@ -31,6 +31,12 @@ Orukeet INT8 14.013%; it is a small descriptive mobile-runtime panel, not a clou
 
 [Full qualification report and per-language tables](QUALIFICATION-20260924.md)
 
+The consumer's native inference sequence was also checked separately: 490/490 observations
+matched the SDK, including all 64 recordings longer than 30 seconds. The Orukeet-only
+preparation warmup preserved every observed transcript, token timestamp and duration.
+[Consumer call-path and warmup report](CONSUMER-PARITY-20260924.md). This bounded Mac
+check does not replace physical-iPhone or complete-app lifecycle acceptance.
+
 Physical iPhone performance, thermal and microphone acceptance remains required. The archive
 itself is immutable; qualification updates are separate metadata, with no weight or decoder changes.
 

coreml/manifest.json

@@ -63,7 +63,18 @@
           "int8_wer_percent": 14.0131,
           "scope": "small_descriptive_mobile_runtime_panel"
         },
-        "status": "experimental_opt_in"
+        "status": "experimental_opt_in",
+        "consumer_parity": {
+          "report": "CONSUMER-PARITY-20260924.md",
+          "consumer_commit": "0d796516c126869e881873e1a93f117be2cac6df",
+          "observations_per_variant": 490,
+          "unique_recordings": 489,
+          "exact_transcripts": 490,
+          "long_recordings": 89,
+          "disk_backed_recordings_over_30_seconds": 64,
+          "warmup_preserves_text_token_timings_duration": true,
+          "scope": "bounded_mac_native_inference_reproduction_not_physical_iphone_acceptance"
+        }
       },
       "source_code_revision": "6c37c587fabcef8b0e932584fbf22a79fcb9d89e",
       "sdk_release_tag": "v0.1.2-coreml.1",