foryoung365/Fun-ASR-Nano-2512-int4-onnx

Fun-ASR-Nano-2512 INT4 ONNX for sherpa-onnx

This repository contains a locally quantized INT4 ONNX variant of FunAudioLLM/Fun-ASR-Nano-2512, prepared for sherpa-onnx offline inference.

Important Notes

• This is not an official release from FunAudioLLM, ModelScope, or k2-fsa.
• The INT4 weights were generated locally from the fp32 ONNX package distributed in the k2-fsa/sherpa-onnx ASR model release assets.
• The original upstream model card currently does not declare clear Hugging Face YAML license metadata. Please verify the upstream usage terms before any redistribution or commercial use.

Source and Lineage

• Upstream model: FunAudioLLM/Fun-ASR-Nano-2512
  • https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512
• ONNX export lineage referenced by upstream sherpa-onnx package:
  • https://www.modelscope.cn/models/zengshuishui/FunASR-nano-onnx/files
  • https://github.com/Wasser1462/FunASR-nano-onnx
• fp32 package used as quantization source:
  • sherpa-onnx-funasr-nano-2025-12-30.tar.bz2
  • from https://github.com/k2-fsa/sherpa-onnx/releases/tag/asr-models

Files

• encoder_adaptor.int4.onnx
• embedding.int4.onnx
• llm.int4.onnx
• Qwen3-0.6B/
  • tokenizer.json
  • merges.txt
  • vocab.json

Quantization Method

• Quantizer: onnxruntime.quantization.matmul_nbits_quantizer.MatMulNBitsQuantizer
• Quantization type: weight-only INT4
• Scope: MatMul weights
• Output format: single-file ONNX artifacts compatible with local sherpa-onnx loading

Local Validation

This INT4 variant was validated locally on Windows with CUDA-enabled sherpa-onnx.

Reference smoke test result on rag_math.wav:

• fp32: 对微分形式的积分是微分几何中的基本概念。
• int8: 对微分形式的积分是微分几何中的基本概念。
• int4: 对微分形式的积分是微分几何中的基本概念。

Example Usage

import sherpa_onnx

recognizer = sherpa_onnx.OfflineRecognizer.from_funasr_nano(
    encoder_adaptor="encoder_adaptor.int4.onnx",
    embedding="embedding.int4.onnx",
    llm="llm.int4.onnx",
    tokenizer="Qwen3-0.6B",
    provider="cuda",
    num_threads=1,
)

Compatibility

• Intended for sherpa-onnx offline inference
• Tested locally with sherpa-onnx==1.12.39+cuda12.cudnn9
• Tested locally with onnxruntime-gpu==1.24.4

Repository Purpose

This repository is intended as a convenient packaged INT4 deployment artifact for local or private inference workflows.