| --- |
| license: other |
| license_name: openmdw-1.1 |
| license_link: LICENSE |
| library_name: pytorch |
| tags: |
| - robotics |
| - world-model |
| - long-horizon |
| - in-context-learning |
| - RoboTwin |
| --- |
| |
| <div align="center"> |
|
|
| # Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation |
|
|
| ViGAR: **Vi**sual **G**oal-conditioned **A**ction **R**easoning |
|
|
| Shukai Gong<sup>1*</sup> · Xuanran Zhai<sup>2*</sup> · Yintianrun Zhang<sup>1*</sup> · Ruopeng Cui<sup>2</sup> · Ye Huang<sup>1</sup> · Yiyang Fu<sup>1</sup> · Dexuan Lyu<sup>2</sup><br> |
| Chaojie Li<sup>2</sup> · Xinyi Song<sup>2</sup> · Peiwen Lin<sup>2</sup> · Chuang Wang<sup>2</sup> · Mingyuan Jia<sup>3</sup> · Yufan Deng<sup>1</sup><br> |
| Jiaxin Fang<sup>3</sup> · Bo Liang<sup>1</sup> · Jiaxin Li<sup>1</sup> · Yuxiang Gao<sup>3†</sup> · Hao Liu<sup>2†</sup> · Daquan Zhou<sup>1†</sup> |
| |
| <sup>1</sup> Peking University <sup>2</sup> AgiBot <sup>3</sup> CocoMatrix |
| |
| <sup>*</sup> Equal contribution <sup>†</sup> Corresponding author |
|
|
| [](https://dagroup-pku.github.io/ViGAR/) |
| [](https://arxiv.org/abs/2610.02368) |
| [](https://huggingface.co/papers/2610.02368) |
| [](https://github.com/DAGroup-PKU/ViGAR) |
| [](https://github.com/DAGroup-PKU/ViGAR/blob/main/docs/installation.md) |
| [](https://github.com/DAGroup-PKU/ViGAR/blob/main/docs/installation.md) |
| [](LICENSE) |
|
|
| </div> |
|
|
|  |
|
|
| ## 🔥 News |
|
|
| - **2026.10.11**: ViGAR ranks #3 on the [RoboTwin 2.0 Leaderboard](https://robotwin-platform.github.io/leaderboard). |
| - **2026.10.05**: ViGAR is released on [arXiv](https://arxiv.org/abs/2610.02368). |
|
|
| ## 📑 Todo List |
|
|
| - [x] Evaluation code and checkpoints on **RoboTwin**. |
| - [x] Training code and data on **RoboTwin**. |
| - [ ] 900-hour real-robot dataset with subtask annotations. |
|
|
| ## 🎬 Demo |
|
|
| <video src="https://huggingface.co/DAGroup-PKU/ViGAR/resolve/main/assets/vigar_demo.mp4" poster="https://huggingface.co/DAGroup-PKU/ViGAR/resolve/main/assets/vigar_demo_poster.jpg" controls width="100%"></video> |
|
|
| ## ⚙️ Getting Started |
|
|
| ```bash |
| git clone https://github.com/DAGroup-PKU/ViGAR.git |
| cd ViGAR |
| ``` |
|
|
| | Step | Guide | |
| | :--- | :--- | |
| | Set up the environments | [Installation](https://github.com/DAGroup-PKU/ViGAR/blob/main/docs/installation.md) | |
| | Download weights and data | [Download](https://github.com/DAGroup-PKU/ViGAR/blob/main/docs/download.md) | |
| | Evaluate on Simulation Benchmark | [Evaluation](https://github.com/DAGroup-PKU/ViGAR/blob/main/docs/evaluation.md) | |
| | Train the policies and subgoal planners in ViGAR | [Training](https://github.com/DAGroup-PKU/ViGAR/blob/main/docs/training.md) | |
|
|
| ## 💪 Repository |
|
|
| ```text |
| vigar/ Policy model, training and serving |
| subgoal_planner/ Planner runtime and training implementation |
| simulation/ RoboTwin tasks and simulator setup |
| data/ Annotation, goal generation and action conversion |
| evaluation/ Policy/planner services and closed-loop evaluation |
| scripts/ Installation and launch commands |
| configs/ Task and inference settings |
| docs/ Installation, download, evaluation and training guides |
| ``` |
|
|
| The policy uses VeOmni with a bundled Cosmos runtime. Planner processes select |
| their own Cosmos runtime through `PYTHONPATH`; they run separately from policy |
| and simulator processes. |
|
|
| ## 🙏 Acknowledgements |
|
|
| Built on [NVIDIA Cosmos](https://github.com/NVIDIA/cosmos), |
| [VeOmni](https://github.com/ByteDance-Seed/VeOmni), |
| [RoboTwin](https://github.com/RoboTwin-Platform/RoboTwin), |
| [Wan2.2](https://github.com/Wan-Video/Wan2.2) and |
| [Qwen3-VL](https://github.com/QwenLM/Qwen3-VL). |
|
|
| ## ✏️ Citation |
|
|
| ```bibtex |
| @article{gong2026rethinking, |
| title={Rethinking World-Action Model for Compositional and In-Context Robotic Manipulation}, |
| author={Gong, Shukai and Zhai, Xuanran and Zhang, Yintianrun and Cui, Ruopeng and Huang, Ye and Fu, Yiyang and Lyu, Dexuan and Li, Chaojie and Song, Xinyi and Lin, Peiwen and others}, |
| journal={arXiv preprint arXiv:2610.02368}, |
| year={2026} |
| } |
| ``` |
|
|
| ## License |
|
|
| Original project code uses the [MIT license](https://github.com/DAGroup-PKU/ViGAR/blob/main/LICENSE). Bundled components retain their upstream licenses. Cosmos-derived model materials use OpenMDW-1.1. |
