Arm/pocket-tts-en-mix-precision

Pocket TTS English - Mixed Precision and Mixed Frameworks

This repository packages a mixed-framework deployment bundle derived from kyutai/pocket-tts for English text-to-speech. The bundle uses SentencePiece, GGUF, ONNX, and LiteRT artifacts and supports streaming synthesis at 24 kHz.

✨ Key Highlights

• Faster than real-time — using one Arm® CPU core with SME2 (an RTF lower than 1 is faster than real time):
  • RTF is 0.222 on an Android™ Vivo X300 smartphone.
• Low latency — using one Arm® CPU core with SME2 on an Android™ Vivo X300 smartphone:
  • Median end-to-end latency is 0.897 s for a prompt that produces 4.04 seconds of audio.
  • Median time to first audio chunk is 0.107 s.
• Memory efficient — measured on an Android™ Vivo X300 smartphone:
  • Peak process RSS is 820.3 MiB.
• Compact deployment — the packaged model files total 382.9 MB.
• English text-to-speech — generates spoken English from a text prompt.
• Packaged voice profiles — includes 26 voice profiles.
• Mixed framework deployment — combines SentencePiece, llama.cpp, ONNX Runtime, and LiteRT.
• Selective FP16 weights — the audio decoder uses FP16 fully connected weights.
• Arm® CPU deployment — optimized for efficient execution on Arm® CPUs.
• Streaming output — returns synthesized speech at 24 kHz in streaming mode.

📦 Model Details

Model Description

Pocket TTS is a lightweight text-to-speech model developed by Kyutai for efficient CPU execution.

• Developed by: Kyutai
• Model type: English text-to-speech with packaged voice profiles
• License: CC BY 4.0
• Base model: kyutai/pocket-tts
• Package form: SentencePiece, GGUF, ONNX, TFLite, and NumPy artifacts

Model Sources

• Base model: https://huggingface.co/kyutai/pocket-tts
• Upstream repository: https://github.com/kyutai-labs/pocket-tts
• Voice collection: https://huggingface.co/kyutai/tts-voices

🚀 Get Started with the Model

Run with Compute Flow

Follow these steps to run Pocket TTS English - Mixed Precision and Mixed Frameworks on macOS® with Apple Silicon or on Linux® with an Arm®-based CPU, such as a Raspberry Pi® 5.

Step 1: Clone Compute Flow and set up the virtual environment

git clone https://github.com/ARM-software/ComputeFlow.git && cd ComputeFlow

uv sync && source .venv/bin/activate

Step 2: Install the brick and model

compute-flow update
compute-flow model install tts:pocket-tts pocket-tts-en
compute-flow build native

Step 3: Run the Python CLI example

python -m cli_examples.tts_cli \
  -m @manifest/pocket-tts/pocket-tts-en/pocket_tts_manifest.json \
  -p "Hello from Compute Flow." \
  -o output.wav \
  -t 1

The example synthesizes the prompt into output.wav.

📊 Quality evaluation

Accuracy results for this mixed-precision package will be provided in a future update. However, no intelligibility regression was observed against the upstream FP32 implementation.

For upstream quality results, refer to the evaluation in the Pocket TTS technical report, which reports WER and pairwise human evaluations for audio quality and speaker similarity on the LibriSpeech test-clean split.

🎯 Performance evaluation

Performance was measured under the following conditions:

• One Arm® CPU thread.
• 5 warmups followed by 30 consecutive measured runs with no pause between runs.
• The Android™ Vivo X300 smartphone screen was kept on.
• The model remained loaded and its state was reset between runs.

The following methodology and definitions were used:

• Runtime: LiteRT, llama.cpp, ONNX Runtime, and SentencePiece on CPU, with XNNPACK and KleidiAI.
• Input prompt: "Hello everyone. I am Jack and I am your personal assistant."
• Voice profile: alba, the packaged default.
• Generated output: 4.04 seconds of 24 kHz audio per run.
• End-to-end latency is the summed model execution time for the complete utterance and excludes model setup.
• Time to first audio chunk is the elapsed execution time until the first streaming audio chunk is returned.
• Average memory is the mean process RSS sampled throughout setup and inference.
• Peak memory is the maximum sampled process high-water mark.
• Latency and memory were collected in separate executions to prevent memory sampling from affecting latency.

Metric	Android™ Vivo X300: Mixed precision
Model size	382.9 MB
RTF	0.222
End-to-end latency, p50	0.897 s
End-to-end latency, p90	0.904 s
End-to-end latency, p99	0.906 s
Time to first audio chunk	0.107 s
Peak memory	820.3 MiB
Average memory	811.0 MiB

RTF is the total inference time divided by the generated audio duration, so lower values indicate faster processing and values below 1 indicate faster-than-real-time generation.

🛠️ Technical Specifications

Objective

Generate streaming 24 kHz English speech from a text prompt.

Runtime Architecture

Component role	Framework / format
Text tokenization	SentencePiece
Text encoding, projection, and denoising	ONNX Runtime / ONNX
Flow language modeling	llama.cpp / GGUF
Audio decoding	LiteRT / TFLite
Voice conditioning	NumPy profile

Precision and Quantization

The LiteRT audio decoder uses FP16 fully connected weights.

Input Specification

Input	Description
Text prompt	UTF-8 string

Output Specification

The model returns synthesized 24 kHz audio in streaming mode.

Repository Contents

• pocket_tts_manifest.json — model package manifest.
• tokenizer.model — SentencePiece tokenizer.
• *.onnx, *.gguf, and *.tflite — model components.
• assets/*.npy — packaged voice profiles and model assets.
• metadata.yaml and benchmarks/ — model and benchmark metadata.
• SHA256SUMS — model-bundle checksums for reproducibility.

🗂️ Model and Asset Origin

• Pocket TTS model
• Pocket TTS source repository
• Kyutai TTS voice collection and asset licensing

🔐 Checksums

SHA256SUMS was generated by recursively hashing every regular file in the model bundle, including files in subdirectories, except the generated root SHA256SUMS and paths with a dotfile component.

From the model bundle root, verify the checked-out files with:

shasum -a 256 -c SHA256SUMS