R³: Training Robots to Reason in Natural Language via Reinforcement Learning

Lehong Wu¹, Yuxiao Qu¹, Zheyuan Hu¹, Ivan Zhang¹, Limin Wei¹, Zackory Erickson¹, Aviral Kumar¹

¹Carnegie Mellon University

TL;DR

R³ trains VLM reasoners with single-step rubric-based RL to reason in natural language to guide low-level policies on long-horizon manipulation tasks. Instantiated on Language Table and simulated bimanual grocery packing tasks, R³ improves generalization on unseen tasks and significantly outperforms instruction-only imitation learning.

Reasoning in language allows foundation models to spend more test-time compute on hard problems, such as those requiring decomposition, constraint tracking, and prediction of future consequences. Whether this mechanism can improve robotic manipulation remains unclear, where long-horizon tasks require tracking partial progress, reasoning about object relations, recovering from mistakes, and steering noisy low-level policies. In this paper, we study whether VLMs can be trained to reason directly in natural language to guide low-level manipulation policies. We introduce R³, a simple post-training recipe that turns off-the-shelf VLMs into robotic reasoners: it first mid-trains a VLM on expert-generated reasoning traces to initialize the desired reasoning style, then improves the reasoner with single-step rubric-based RL from offline action data. Unlike prior robotic reasoning methods that mostly use structured traces as auxiliary supervision, R³ trains free-form language reasoning to produce test-time guidance for action.

We instantiate R³ on Language Table and simulated bimanual grocery packing, two controlled testbeds for studying robotic reasoning and long-horizon manipulation. R³ improves exploration and generalization across unseen tasks and significantly outperforms instruction-only imitation learning baselines on both benchmarks. Our analyses suggest that free-form language reasoning can function as a test-time compute mechanism for steering low-level policies.

Language Table

Grocery Packing

R³: Robotic Reasoners via Reinforcement Learning

Hierarchical Policy for Long-Horizon Manipulation

A high-level VLM takes a scene, goal, and previous response as input, then reasons in language and issues a short-horizon instruction. A fixed language-conditioned low-level policy then executes the instruction.

Policy architecture of R³ — **Policy architecture.**

Two-stage Training of R³

R³ mid-trains on limited reasoning traces, then improves the reasoner with single-step RL on instruction-only offline data. The recipe is shared; the expert, history, and reward are domain-specific.

Stage I: Mid-Training Reasoning Behaviors into the VLM. Supervised fine-tuning (SFT) on expert reasoning traces teaches the VLM to reason over scene and history before acting. We skip this stage on grocery packing, where the base VLM already produces useful reasoning.
Stage II: Rubric-Based Single-Step RL with Offline Data. Single-step RL (Dr.GRPO) on expert instructions, without reasoning supervision or multi-turn rollouts. Language Table uses a rubric-based VLM judge (semantic match). Grocery packing uses exact string match over a finite pack / remove / transfer set.

Main Results

Language Table

We evaluate R³ on 14 long-horizon block-arrangement tasks in Language Table that test relational transfer, compositional generalization, and geometric difficulty. Gemini 3 Flash provides expert reasoning that steers a pretrained language-conditioned policy.

RL post-training alone improves performance. Rubric-based RL can reinforce useful reasoning behaviors and improve task performance even without mid-training.
Mid-training improves RL post-training. Mid-training is a strong warm start; a modest amount of reasoning data is enough for RL, especially for OOD generalization.
R³ enables better OOD generalization than instruction-only imitation. R³ matches or beats instruction-only imitation on seen tasks, and significantly outperforms it on every held-out OOD task.

Language Table main results: average success rates across methods — **Main results on Language Table.** Average success rate across mid-training, RL, and OOD held-out task groups for base models, imitation baselines, and R³ variants. For detailed per-task numbers, see the table below.

Language Table main results: per-task success rates — **Main results.** We compare base models, imitation baselines, and R³ variants. Values are percentages with 95% confidence intervals. The green / red cells in the Δ column mark significant gains / losses (|Δ| > CI). Gemini’s performance during data collection is shown for reference. Bold / underlined values mark the best / second-best non-expert model.

Grocery Packing

We evaluate R³ on simulated grocery packing tasks, where two 7-DoF xArm-7s pack YCB objects into 3 trays. Humans provide instruction-only demonstrations via teleoperation, and we fine-tune π_0.5 as the low-level policy.

On 12 held-out bimanual packing tasks, R³ (RL only) reaches higher success rate and progress than instruction-only imitation (47.9% vs. 38.0% mean success). Mid-training can be skipped because the base VLM already produces useful reasoning on this domain.

Grocery packing main results: average success rate and progress — **Main results on grocery packing.** Average success rate and normalized task progress on 12 held-out tasks. For detailed per-task numbers, see the table below.

Grocery packing main results: per-task success rate and progress — **Main results on grocery packing.** Success rate and normalized progress on 12 held-out tasks. Values are percentages with 95% confidence intervals. Bold values mark the best model.

Analysis

Inference-Time Reasoning Matters Beyond Representation Learning

Takeaway: Explicit inference-time reasoning improves generalization beyond what is achieved by using reasoning only as training-time supervision.

Evidence A: R³ improves both static perception and action understanding, but these improvements alone do not explain its manipulation gains.
Evidence B: R³ generalizes better than non-reasoning policies that use reasoning as additional training-time supervision.
Evidence C: Increasing the inference-time reasoning budget improves success.

VQA results on perception and instruction comprehension — **Accuracy by VQA question class.** Gemini is reported as an external reference and is not included in the ranking. Bold values mark the best non-reference model. Even our best model remains far below Gemini on several VQA categories, yet matches or approaches Gemini on many manipulation tasks. These diagnostics suggest that R³ improves both static perception and action understanding, but they also show that VQA performance alone, i.e., improving static perception, does not fully explain the gains in manipulation performance.

IL pre-train/co-train comparison and R³ reasoning truncation — **IL with pre-training or co-training on reasoning, and truncation of R³ reasoning at test time.** Values are percentages. Parenthetical values for IL (Pre-train) and IL (Co-train) indicate absolute changes relative to IL; for truncated R³ variants, they indicate absolute changes relative to full R³.

Understanding Reasoning Behaviors Learned by R³

R³ learns reasoning strategies that are useful for long-horizon manipulation: comparing alternatives, self-correction, and resolving visual and historical uncertainty.

**Example of reasoning strategy learned via R³.** The R³ model considers several alternate plans before committing to one particular choice.

Mid-training largely aligns the model’s instruction distribution with the expert’s, providing a strong behavioral prior before RL. With mid-training as a warm start, RL refines an already reasonable behavior distribution rather than rediscovering behaviors from scratch.

Effect of RL on instruction distributions — **RL affects instruction distributions.** RL from the base model broadly rewrites the instruction distribution, while RL after mid-training makes localized edits.

Comparison with Approaches that Use Structured CoT Templates

We adapt ECoT to our hierarchical setting by augmenting the response with explicit simulator-state annotations. Our results show that adding ECoT-style structured state annotations generally does not provide additional benefit over free-form language reasoning in our setting. Long-horizon progress tracking, recovery, and closed-loop replanning matter more here than making low-level visual grounding explicit.

ECoT variants comparison — **Evaluation results of ECoT variants.** Trained by SFT on only our mid-training data. Values are percentages. Bold values mark the best model.

Examples of Evaluation Rollouts

Here we show evaluation rollouts of our model on 14 Language Table tasks and 12 unseen grocery packing tasks.

Language Table

Group blocks (group)

Example trajectory 1

Example trajectory 2

Make a line (line)

Example trajectory 1

Example trajectory 2

Task 1

Task 2

Task 3

Task 4

Task 5

Task 6

Task 7

Task 8

Task 9

Task 10

Task 11

Task 12

BibTeX

@misc{wu2026r3trainingrobotsreason,
  title={$\mathcal{R}^3$: Training Robots to Reason in Natural Language via Reinforcement Learning},
  author={Lehong Wu and Yuxiao Qu and Zheyuan Hu and Ivan Zhang and Limin Wei and Zackory Erickson and Aviral Kumar},
  year={2026},
  eprint={2608.26053},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2608.26053},
}