NEST FastConformer β€” Self-Supervised Speech Encoder (LibriSpeech-100)

This is a self-supervised (SSL) speech encoder pretrained from scratch with the NEST objective using the NVIDIA NeMo framework (https://github.com/NVIDIA-NeMo/Speech).

Important: This is a pretrained encoder, not a speech-recognition model. It cannot transcribe audio on its own; it has no text decoder and was never trained on transcripts. It is intended as an initialization / starting point for downstream fine-tuning (e.g. CTC or RNN-T ASR).

Model summary

  • Type: Self-supervised speech encoder (masked-prediction pretraining)
  • Objective: NEST; the model masks blocks of the input audio and learns to predict the masked content, learning general speech representations without any labels.
  • Architecture: FastConformer encoder (ConformerEncoder)
  • Encoder config: d_model=512, n_layers=17, n_heads=8, subsampling=dw_striding, subsampling_factor=8
  • Parameters: ~108M (encoder)
  • Framework: NVIDIA NeMo
  • Language: English
  • Format: .nemo

Intended use

A base model for downstream ASR fine-tuning. Typical workflow:

  1. Load this .nemo as the encoder initialization for a CTC or RNN-T ASR model.
  2. Fine-tune on a labeled (audio + transcript) dataset.
  3. Evaluate the fine-tuned model with Word Error Rate (WER).

It is not suitable for direct transcription, and it is a research/learning-scale checkpoint (see Limitations).

Loading (NeMo)

import nemo.collections.asr as nemo_asr
model = nemo_asr.models.ASRModel.restore_from("nest_encoder_best.nemo")

To use only the encoder weights for fine-tuning an ASR model, initialize your CTC/RNN-T model and load the encoder weights from this checkpoint via NeMo's init_from_nemo_model with an include: [encoder] filter, keeping the encoder architecture identical (n_layers=17, d_model=512).

Training details

  • Pretraining method: NEST masked-prediction (self-supervised, no transcripts)
  • Data: LibriSpeech train-clean-100 (~100 hours of English read speech), used as unlabeled audio (transcripts removed)
  • Audio: 16 kHz mono
  • Loss: MultiMLMLoss (masked-token prediction over a random-projection quantizer, 8192 codes)
  • Optimizer: AdamW
  • Schedule: Noam annealing
  • Precision: bf16
  • Best validation loss: ~5.76 (masked-prediction cross-entropy), reached at ~26 epochs (started from ~6.24 at epoch 0; random-guess baseline is about ln(8192), roughly 8.9)

The pretraining loss decreased steadily from ~8.9 (random) to 6.24 to 5.76, indicating the encoder learned meaningful speech structure from unlabeled audio.

Evaluation

Because this is a self-supervised encoder with no text output, standard ASR metrics such as WER do not apply directly. The only intrinsic training metric is the pretraining masked-prediction loss (best ~5.76, above).

The meaningful (extrinsic) evaluation of an SSL encoder is downstream: fine-tune it on a labeled task and measure WER, comparing against a randomly-initialized encoder of the same architecture. A downstream WER for this checkpoint has not been established and is left to the user.

Limitations

  • Learning-scale checkpoint. Pretrained on only ~100 hours for a limited number of effective epochs. Production SSL encoders use thousands to tens of thousands of hours; this checkpoint will not match production-grade representations.
  • Not a transcriber. No decoder, no text; it cannot recognize speech without fine-tuning.
  • Under-trained encoders can underperform random initialization on some downstream fine-tunes. Users should A/B test against random init for their specific task.
  • English read speech only. Pretrained on audiobook-style speech; representations may transfer poorly to other domains.

License and attribution

Released under CC-BY-4.0, inherited from the LibriSpeech dataset it was pretrained on. Attribution is required.

Citation

  • NVIDIA NeMo β€” https://github.com/NVIDIA-NeMo/Speech
  • LibriSpeech β€” V. Panayotov, G. Chen, D. Povey, S. Khudanpur, "Librispeech: An ASR corpus based on public domain audio books," ICASSP 2015.
Downloads last month
15
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train bhaveshsoni0023/nest-fastconformer-librispeech100-ssl