NEO-VLA drops the vision encoder so action policies can read visual features at any depth
A paper from MMLab@NTU argues that vision-language-action models lose action-critical detail in their visual encoders and proposes NEO-VLA, an encoder-free design.
The problem with a fixed visual endpoint
Most vision-language-action models follow the same order of operations, according to the authors of “From Pixels to Actions - Towards Native Vision-Language-Action Policies” (arXiv 2611.06918, published 2026-11-04 and listed on Hugging Face daily papers the same day). A visual encoder first compresses the image, and action reasoning starts only after that compression.
The authors argue that this premise works against manipulation. Manipulation often depends on local geometry, boundaries, depth and subtle state changes, and those are the visual signals most vulnerable to abstraction by an encoder. The paper frames the question as whether action policies should inherit a fixed visual endpoint or directly access evolving visual computation.
How NEO-VLA works
NEO-VLA is described as a structurally encoder-free vision-language-action framework. There is no separate visual encoder in front of the model. Instead, lightweight patch convolutional layers tokenize the images, and those image tokens evolve jointly with language throughout the backbone.
The action side is a flow-matching action expert. Because the visual tokens pass through the whole backbone together with the instruction, the action expert can draw on instruction-conditioned visual representations at different depths rather than on a single encoder output.
What the authors report
All findings below are the authors’ own, as stated in the abstract.
- Native architectures outperform encoder-based counterparts under controlled comparisons.
- Full fine-tuning reduces the gap but does not erase it.
- Additional scale alone cannot recover the gap.
- Layer-wise probing shows that action-critical signals often peak well before the final visual representation.
- Across diverse manipulation settings and distribution shifts, NEO-VLA achieves stronger and more robust control.
The probing result ties back to the design choice: if the signals a policy needs often peak before the final visual representation, a policy that sees only that final output cannot use them where they are strongest. Taken together, the authors write, the results suggest that embodied control benefits from access to visual representations across the computational hierarchy.
What the abstract leaves open
The abstract gives no numbers. It names no benchmarks, task suites or success rates, and it does not say whether the manipulation settings are simulated, physical or both. It does not state the size of the backbone, which encoder-based models serve as baselines, or what “additional scale” means in the comparison. Inference cost and latency of the encoder-free design are not discussed, and no code repository is listed on the paper page. Those details would have to come from the full paper.
The paper has 17 authors from MMLab@NTU.
Source: From Pixels to Actions - Towards Native Vision-Language-Action Policies (arXiv 2611.06918)