VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models
Institute for AI, Peking University
VLA-Arena is an open-source framework for benchmarking Vision-Language-Action models across 170 tasks at difficulty levels L0-L2, covering Safety, Distractor, Extrapolation, and Long Horizon robot manipulation.
Explore the model leaderboard and task store, read our paper, or use the open-source benchmark framework.
Abstract
While Vision-Language-Action models (VLAs) are rapidly advancing toward generalist robot policies, quantitatively characterizing their capability boundaries and failure modes remains challenging. To address this, we introduce VLA-Arena, a comprehensive benchmark. It features a novel structured task design framework to quantify difficulty across three orthogonal axes: (1) Task Structure, (2) Language Command, and (3) Visual Observation. This allows us to systematically design tasks with fine-grained difficulty levels, enabling a precise measurement of model capability frontiers. For task structure, VLA-Arena comprises 11 task suites organized into four dimensions: Safety, Distractor, Extrapolation, and Long Horizon, totaling 170 tasks. Each suite spans three difficulty levels (L0-L2), with fine-tuning restricted to L0 to rigorously assess generalization. Orthogonal to this, language (W0-W4) and visual (V0-V4) perturbations can be applied to any task as diagnostic probes to distinguish robust grounding from superficial pattern matching. Our extensive evaluation of state-of-the-art VLAs reveals critical limitations: memorization over generalization, superficial visual perception, and a neglect of safety constraints. Additionally, model rank reversals across L0-L2 validate that each level provides non-redundant insights. To foster research addressing these model limitations and ensure reproducibility, we provide the complete VLA-Arena framework, including an end-to-end toolchain from task definition to automated evaluation and the VLA-Arena-S/M/L datasets for fine-tuning. Our benchmark, datasets, models, and leaderboard are publicly available at https://vla-arena.github.io.
BibTeX
@inproceedings{
zhang2026vlaarena,
title={{VLA}-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models},
author={Borong Zhang and Jiahao Li and Jiachen Shen and Yuhao Zhang and Yishuai Cai and Yuanpei Chen and Juntao Dai and Jiaming Ji and Yaodong Yang},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=qjeCH95rbF}
}
VLA-Arena Model Performance Leaderboard
Comprehensive evaluation of vision-language-action models across multiple difficulty levels and task categories. Interactive filtering and sorting enable detailed performance analysis and cross-model comparison.
🚀 Contribute Your ModelFiltering & Sorting Configuration
Legend
- • The VLA-Arena benchmark evaluates VLA models across four dimensions: Safety, Distractor, Extrapolation, and Long Horizon.
- • Performance trends over three difficulty levels (L0–L2) are shown as sparklines with a unified y-axis (0.0–1.0) for cross-model comparison.
- • Safety tasks report both Cumulative Cost (CC) (orange, above sparkline) and Success Rate (SR) (blue, below sparkline). Other tasks report only SR.
- • Bold numbers mark the highest CC or SR per difficulty level across all models.
- • ● and ● denote each model's maximum and minimum SR values, respectively.
Tip: Click on column headers to sort by that task's average performance.
Task Store
Browse and select from our collection of 12 benchmark tasks across Safety, Distractor, Extrapolation, Long Horizon, and LIBERO categories. Click on any task to view detailed visualizations across difficulty levels (L0-L2), or select multiple tasks to download.
✨ Contribute Your Tasks