CLAP: Direct VLM-to-VLA Adaptation via Language-Action Grounding

Yuri Ishitoya^1,*,†   Jeremy Siburian^2,*,†   Masashi Hamaya³  

Kuniaki Saito³   Cristian C. Beltran-Hernandez³   Mai Nishimura^3,†

¹Ochanomizu University   ²The University of Tokyo   ³OMRON SINIC X Corporation

^†Equal contribution    ^*Work done as an intern at OMRON SINIC X Corporation

---

📄 Abstract

Vision-language-action models (VLAs) inherit semantic capabilities from pretrained VLMs, yet large-scale post-training on robot data and architectural modifications can reshape the backbone so extensively that it becomes difficult to isolate what the VLM contributes to control. Directly converting pretrained VLMs into VLAs with minimal architectural change offers a more transparent path to understanding how VLM capabilities transfer across model scales. The core obstacle is output-distribution mismatch: predicting actions as bare numeric token sequences moves generation away from the VLM's pretrained language distribution, degrading the capabilities we seek to preserve. To address this, we propose CLAP (Causal Language-Action Prediction), which prepends each numeric action sequence with a natural-language action description, causally conditioning precise action-token prediction on a language-action plan without modifying the backbone architecture. With single-epoch fine-tuning alone, 2B CLAP achieves 90.8% on LIBERO (+14.9 pt over VLA-0) and improves robustness on LIBERO-PRO under language, object, and spatial perturbations. We will release CLAP at 0.8B, 2B, and 4B as an open-weight, multi-scale compact VLA family from a single VLM lineage, enabling controlled analysis of VLM-to-VLA capability transfer.

🚧 Code

Code coming soon.