---
license: other
license_name: research-use-only
license_link: LICENSE
library_name: pytorch
pipeline_tag: audio-to-audio
tags:
  - audio-to-motion
  - co-speech-gesture
  - talking-head
  - smpl-x
  - motion-generation
  - decoder-only
base_model:
  - Qwen/Qwen3-1.7B
  - Qwen/Qwen3-0.6B
---

# Language of Motion — Decoder-only 4-part Audio-to-Motion (a2m)

Speech-driven, whole-body **co-speech motion** generation as **next-token prediction**. A pretrained
LLM backbone (Qwen3) is turned into an audio→motion model by **keeping its text vocabulary and
enlarging it** with GLM-4-Voice audio tokens and four motion codebooks, then generating the motion
compositionally in a single causal stream:

```
[BOS] audio-tokens [SEP] [FACE] face [UPPER] upper [LOWER] lower [HAND] hand [EOS]
```

Each motion stream is decoded by its VQ decoder to **SMPL-X** (face = 6D head + 6D jaw + 100 expr;
body = upper / lower / hand), giving a full talking + gesturing avatar from speech alone.

## Checkpoints

| File | Backbone | Params | val loss ↓ | val acc ↑ |
|---|---|---|---|---|
| `qwen3-1.7B_4part.pt` | Qwen3-1.7B | 1.76 B | **0.696** | 0.854 |
| `qwen3-0.6B_4part.pt` | Qwen3-0.6B | 0.61 B | **0.705** | 0.834 |

*(val loss = mean cross-entropy over the 4 motion streams on a held-out split; the two numbers are
comparable to each other since both use the same vocab/targets.)*

Each `.pt` is self-describing: `{model, cfg, vocab, step, val_loss, val_acc}` (`torch.load`).

## Architecture

- **Backbone**: pretrained `Qwen3-1.7B` / `Qwen3-0.6B` (decoder-only). The **text vocab is kept**;
  the embedding table is resized (mean-init) to append audio + motion + delimiter tokens.
- **Audio tokens**: GLM-4-Voice (16384-way, 12.5 tok/s).
- **Motion codebooks**: ViBES face VQ (512-way, 25 tok/s, 112-D) + upper / lower / hand body VQs
  (256-way each, 6.25 tok/s).
- **Generation**: block-by-block (face → upper → lower → hand), each block constrained to its own
  code range; the decoded VQ streams are composited into a single SMPL-X sequence.

## Training data

Trained on a mix of co-speech-gesture and talking-head corpora:

| Dataset | Provides | Notes |
|---|---|---|
| [BEAT2](https://github.com/PantoMatrix/PantoMatrix) | body **+** face | CC BY-NC 4.0 (non-commercial) |
| TFHP (HDTF-derived) | face only (body masked) | talking-head |
| **YouTube_Talking** | face only (body masked) | in-the-wild talking-head |

**For the YouTube_Talking data preparation, follow ViBES.** We do **not** redistribute any raw
video/audio — YouTube_Talking is released **recipe-only** (URL list + downloader + derived
annotations) as part of ViBES:
[`JuzeZhang/YouTube_Talking`](https://huggingface.co/datasets/JuzeZhang/YouTube_Talking) — see its
README / the ViBES `docs/1-data/youtube_talking.md` for the download + annotation recipe. The audio
tokenization and motion-token extraction used here are the same scripts described there.

## Usage / inference

These weights need the inference harness shipped alongside them (`a2f_decoder/`) plus the VQ
decoders and the GLM-4-Voice tokenizer. Minimal flow:

```bash
# 1. audio-tokens -> 4 motion streams -> SMPL-X
python a2f_decoder/infer_4part.py --ckpt qwen3-1.7B_4part.pt --out out/ --save_only
# 2. render (SMPL-X body via Blender + FLAME face)
```

See `a2f_decoder/README.md` (included) for the full dependency list (VQ checkpoints, SMPL-X / FLAME
model files, GLM-4-Voice tokenizer) and end-to-end example.

## License

**Research use only.** These weights are derived from models trained on BEAT2 (CC BY-NC 4.0,
non-commercial), TFHP (HDTF-derived), and YouTube_Talking (source videos remain under YouTube's
terms; none are redistributed). The base models are Qwen3 (Apache-2.0). Use of these checkpoints is
restricted to **non-commercial research** and inherits the terms of the underlying datasets and
base models. See `LICENSE`.

## Citation

```bibtex
@misc{lom_a2m_4part,
  title  = {Language of Motion: Decoder-only 4-part Audio-to-Motion},
  author = {Zhang, Juze and others},
  year   = {2026},
  note   = {Research-use-only checkpoints, https://huggingface.co/JuzeZhang}
}
```