Robotics VLA Skill
Expert guidance for building generalist robot policies using Vision-Language-Action (VLA) flow models, based on the π0 architecture.
Core Architecture
π0 model = VLM backbone + action expert + flow matching
| Component | Detail |
|---|---|
| VLM backbone | PaliGemma (3B) — provides visual + language understanding |
| Action expert | Separate transformer weights (~300M) for robot state + actions |
| Total params | ~3.3B |
| Action output | Chunks of H=50 actions; 50Hz or 20Hz robots |
| Inference speed | ~73ms on RTX 4090 |
See references/architecture.md for full technical details (attention masks, flow matching math, MoE design).
Training Pipeline
Two-phase approach (mirrors LLM training):
- Pre-training → broad physical capabilities + recovery behaviors across many tasks/robots
- Fine-tuning → fluent, task-specific execution on target task
Key rule: combining both phases outperforms either alone. Pre-training gives robustness; fine-tuning gives precision.
See references/training.md for data mixture ratios, loss functions, and fine-tuning dataset sizing.
Action Representation
Use flow matching, not autoregressive discretization.
- Flow matching models continuous action distributions → essential for high-frequency dexterous control
- Autoregressive token prediction (e.g. RT-2 style) cannot produce action chunks efficiently
- Action chunks allow open-loop execution at 50Hz without temporal ensembling
Multi-Embodiment Support
Single model handles 7+ robot configurations via:
- Zero-padding smaller action spaces to match the largest (17-dim)
- Shared VLM backbone; embodiment-specific behavior learned via data
- Weighted task sampling: n^0.43 to handle imbalanced data across robot types
See references/embodiments.md for robot platform specs and action space details.
High-Level Policy Integration
For long-horizon tasks, use a two-tier approach:
- High-level VLM: decomposes task ("bus the table") → subtasks ("pick up napkin")
- Low-level π0: executes each subtask as a language-conditioned action sequence
Analogous to SayCan. Intermediate language commands significantly boost performance vs. flat task descriptions.
Related & Complementary Research (2025)
π0 has been extended and complemented by several key works. See references/related-work.md for the full landscape, including:
- π0-FAST / π0.5 / π0.6 — direct successors with faster training, open-world generalization, and RL fine-tuning
- RTC — async action chunking to eliminate inference pauses (plug-in, no retraining)
- UniVLA — unsupervised action extraction from raw video (no action labels needed)
- ManiFlow / Streaming Flow — smoother action generation
- GR00T N1, Helix, OpenVLA-OFT, DiVLA, RDT-1B — parallel approaches from NVIDIA, Figure AI, and academia
Evaluation Checklist
When evaluating a robot manipulation policy:
- Out-of-box generalization (no fine-tuning) vs. baselines
- Language following accuracy with flat / human-guided / HL commands
- Fine-tuning efficiency (success rate vs. hours of data)
- Complex multi-stage tasks (5–20 min, recovery from failure)
- Compare: OpenVLA, Octo, ACT, Diffusion Policy as baselines