SG-VLA: Learning Spatially-Grounded Vision-Language-Action Models for Mobile Manipulation

Pick

Place

Open Fridge

Close Fridge

SG-VLA learns spatially grounded representations for mobile manipulation through multi-view RGB-depth perception and progressive auxiliary-task co-training. It improves average success from 60% to 73% on ManiSkill-HAB household tasks.

1.3Bparameters

44K / 1.4Mepisodes / transitions

73%average success, up from 60%

Abstract

Vision-language-action models have shown strong semantic understanding in tabletop settings, but household mobile manipulation is harder: a robot must reason jointly about scene layout, object geometry, its own configuration, and coordinated base, arm, torso, and gripper motion. Direct imitation learning supplies only a sparse action-prediction signal for this demanding 13-dimensional control problem.

SG-VLA addresses this gap by enriching both what the model observes and what it learns to reconstruct. The model combines head- and hand-camera RGB observations with depth, then co-trains auxiliary decoders for global robot position, grasp state, joint configuration, target-object pose, and target-object segmentation. These dense objectives encourage the shared visual-language representation to encode spatial and manipulation-relevant structure. A progressive training schedule first adapts randomly initialized decoders without letting their auxiliary gradients disrupt the backbone, then jointly refines the full model.

SG-VLA Architecture

Spatially rich perception

Head and hand RGB views provide complementary global and local perspectives, while paired depth maps expose geometry explicitly. We also evaluate temporal context from the four previous actions; the strongest configuration uses multi-view RGB and depth without history.

Auxiliary spatial supervision

Five decoders reconstruct global position [x, y, z], grasp state, 12-dimensional joint configuration, 7-dimensional target-object pose, and target-object segmentation from shared VLM features.

Flexible action generation

The Qwen2.5-0.5B-based model can predict discrete action tokens or use an optional flow-matching expert for continuous actions. Each 13D action controls a 3D base-pose delta, torso height, seven arm joints, and two gripper dimensions.

Progressive Training

Three-stage SG-VLA training scheme: decoder adaptation with blocked auxiliary gradients, joint refinement with full gradient flow, and isolated flow-matching action-head training with a frozen VLM.

Stage 1 Decoder adaptation

Auxiliary gradients are blocked from entering the VLM backbone while the discrete action-token path continues to update it. The decoders learn to interpret established VLM features without disrupting them.
Stage 2 Joint refinement

Full auxiliary gradient flow is enabled. Action prediction and auxiliary objectives jointly refine the backbone toward spatially grounded, manipulation-aware representations.
Stage 3 Action-head training

The VLM is frozen and the flow-matching action head is trained in isolation to generate continuous 13-dimensional action chunks from stable latent features.

Experiments

Richer input modalities raise OpenVLA from 0.04 to 0.32 average success, and the compact base VLM reaches 0.60. Naive auxiliary co-training falls to 0.51, whereas progressive training reaches 0.73. On cross-task Pick and Place evaluation, segmentation and object-pose reconstruction improve the average from 0.27 to 0.47. All values below are success rates. Bold denotes the best result in a comparison; underlining denotes the second best.

Input modalities

Performance comparison between models using different input modalities.
Method	Pick	Place	Open		Close		Avg. ↑
Method	All Obj. ↑	All Obj. ↑	Fridge ↑	Drawer ↑	Fridge ↑	Drawer ↑	Avg. ↑
OpenVLA	0.00	0.19	0.02	0.00	0.00	0.04	0.04
+ Multiview	0.06	0.35	0.14	0.38	0.00	0.53	0.24
+ Multiview + Depth	0.12	0.41	0.43	0.30	0.00	0.67	0.32
Base VLM + Multiview	0.06	0.53	0.60	0.30	0.63	0.93	0.52
+ Multiview + Depth	0.16	0.56	0.67	0.36	0.83	1.00	0.60
+ Multiview + Depth + History	0.00	0.47	0.57	0.40	0.47	1.00	0.49

Auxiliary tasks and progressive training

Ablation of auxiliary objectives and progressive training on SetTable.
Progressive	Method	Pick	Place	Open		Close		Avg. S.R. ↑
Progressive	Method	All Obj. ↑	All Obj. ↑	Fridge ↑	Drawer ↑	Fridge ↑	Drawer ↑	Avg. S.R. ↑
No	SG-VLA	0.16	0.56	0.67	0.36	0.83	1.00	0.60
No	+ all	0.03	0.50	0.60	0.23	0.67	1.00	0.51
Yes	+ is_grasped	0.30	0.53	0.83	0.57	0.80	0.93	0.66
	+ qpos	0.23	0.67	0.87	0.70	0.90	0.90	0.71
	+ global pos	0.07	0.27	0.90	0.70	0.97	1.00	0.65
	+ all	0.13	0.70	0.87	0.77	0.90	1.00	0.73

“+ all” includes is_grasped, qpos, and global position.

Object-centric auxiliary supervision

Pick and Place performance with target-object segmentation and object-pose reconstruction.
Method	SetTable		PrepareGroceries		TidyHouse		Avg. S.R. ↑
Method	Pick ↑	Place ↑	Pick ↑	Place ↑	Pick ↑	Place ↑	Avg. S.R. ↑
SG-VLA	0.16	0.56	0.10	0.33	0.07	0.40	0.27
+ seg + obj_pos	0.26	0.78	0.13	0.60	0.33	0.73	0.47

Flow-matching action head

Performance with and without the optional flow-matching action head.
Method	Pick	Place	Open		Close		Avg. ↑
Method	All Obj. ↑	All Obj. ↑	Fridge ↑	Drawer ↑	Fridge ↑	Drawer ↑	Avg. ↑
SG-VLA	0.13	0.70	0.87	0.77	0.90	1.00	0.73
SG-VLA + action head	0.27	0.80	0.76	0.60	0.76	0.97	0.69

The action head improves Pick and Place, but lowers overall average success from 0.73 to 0.69, indicating a task-dependent tradeoff.

Simulation Rollouts

Pick

Place

Open Fridge

Close Fridge

Open Kitchen Counter

Close Kitchen Counter

Citation

If you find SG-VLA useful, please cite:

@misc{tu2026sgvlalearningspatiallygroundedvisionlanguageaction,
      title={SG-VLA: Learning Spatially-Grounded Vision-Language-Action Models for Mobile Manipulation}, 
      author={Ruisen Tu and Arth Shukla and Sohyun Yoo and Xuanlin Li and Junxi Li and Jianwen Xie and Hao Su and Zhuowen Tu},
      year={2026},
      eprint={2603.22760},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2603.22760}, 
}

SG-VLA in action