Angel Xu

How robots become agentic: four approaches

Layered orchestration, code as policy, constraint-mediated generation and direct control: what each does, the evidence so far, and where the lines blur.

Angel Xu16-minute read

There are four main approaches to making a robot agentic today. No survey gives this exact count, because each sorts the field by a different question. Counting the alternative ways a robot's decisions reach its motors at run time gives four.

What separates them is what the frontier reasoning model outputs. Hybrids that switch between approaches step by step are combinations, not a fifth approach.

  1. Layered orchestration. The model picks and steers learned skills (VLAs, world action models, RL policies) as tools.
  2. Code as policy. The model writes a program over perception and control primitives.
  3. Constraint- and objective-mediated generation. The model writes value maps, keypoint constraints or costs; an optimizer turns them into motion.
  4. Direct control. The model emits the motion targets itself.

End-to-end VLAs and world action models are not counted as approaches, even ones that reason internally. The surveys disagree on them. Salimpour et al. count end-to-end models as one integration type. Wang et al. include a standalone policy only when a foundation-model planner or verifier steers it. Qin et al. state that action prediction alone does not make a system agentic. Here they appear as the motor layer that the run-time paths drive.

From left to right, less sits between the reasoning model and the motors, until in direct control nothing does. Swipe sideways to see all four.

How the 2026 surveys count

Each survey answers a different question, which is why their counts differ.

Survey Date What it sorts by Its count
Envisioning Agentic Robotics (Grid Robotics: Lan et al.) Nov 2026 How the reasoning layer reaches the body 3 execution paths: layered orchestration, direct control, code as policy; hybrids recommended
A Survey of Agentic Robotics: Toward Continual Self-Improvement (Wang, Jiang, Shen, Soh) Oct 2026 Where the foundation model's decisions act, coded across 2,343 systems 3 loci: task orchestration, motion/action generation, improvement
Towards Agentic Robots (Qin et al.; CUHK, Fudan, Tsinghua and others) Oct 2026 Which part of the robot system model decisions shape 17 categories in 4 domains: model, data, environment, harness
Weights or Skills? (Jena et al.) Aug 2026 Whether competence ships as network weights or as code 6 technique branches; code as policy split into 5 levels of self-improvement
Towards Embodied Agentic AI (Salimpour et al.) Nov 2025 How the foundation model is integrated 4 types: single-shot tool calls, agentic loop, orchestration, direct or embedded
arXiv abstract page of Envisioning Agentic Robotics by Grid Robotics, submitted 13 November 2026
Envisioning Agentic Robotics (Grid Robotics: Qingfeng Lan et al., submitted 13 November 2026) opens by crediting GPT-6 Astra with starting the shift to agentic robotics. Screenshot of the arXiv abstract page, CC BY.
Figure 4 of Envisioning Agentic Robotics: layered orchestration, direct control and code as policy as three paths from S2 to the physical world
Figure 4 of Envisioning Agentic Robotics, "Three typical execution paths": approaches 1, 4 and 2 in this essay. The authors note one system may use all three within a single task. From the arXiv HTML version, CC BY.

The four approaches here combine two sources. Three are the execution paths named in Envisioning Agentic Robotics. The fourth comes from Wang et al., who split motion generation three ways: direct action, programs, and objectives that a solver executes. Wang et al.'s improvement locus and Qin et al.'s Data and Environment domains are real and growing. But they describe how robots are built and improved, not how a robot acts, so they are covered under Beyond run time. Harness functions (memory, monitoring and recovery, skill libraries) appear in every approach, so they are not an approach of their own.

1. Layered orchestration

This is how the frontier labs ship today. A reasoning model plans, tracks progress and decides when to replan. Fast learned policies do the physical work as bounded subtasks.

In production stacks

Two robot arms above a Tower of Hanoi puzzle in NVIDIA's agentic GR00T 2 demo
A still from NVIDIA's agentic GR00T 2 demos: a frontier reasoning agent plans the Tower of Hanoi moves and calls GR00T 2 as a tool for each grasp. In NVIDIA's words, the agent decides what to do and GR00T 2 decides how. From the GR00T 2 project page, which has the videos.

In research harnesses

What decides whether it works. A systematic study of hierarchical VLA agents (Hu, Shridhar, Shah, Tan, Xie) found that model choice and interface design jointly shape performance. A carefully designed hierarchy beat both a flat VLA and a naive hierarchy. Three design questions recur:

  1. The interface. What the reasoner hands down is moving from text subgoals to pictures: subgoal images, grasp marks, traces.
  2. Switching. When control passes between layers: on a timer, on an event, or by interruption.
  3. Handoffs. Each motor policy only works in states its training data covered. Chaining policies fails at the seams unless the harness manages the transition.

Strengths and limits. Dexterity comes from specialized motor models; semantics and recovery come from the reasoner. But the system can only do what its motor tools can do, and nobody knows those tools' capability boundaries in advance.

2. Code as policy

Here the frontier model acts like a robotics engineer. It writes perception, planning and control code against the robot's interfaces, runs it, watches what happens and fixes it. The approach goes back to Google's Code as Policies (2022). What is new in 2026 is that coding agents can now get a robot working without a human writing the code.

Key evidence

Three public Vals Robo-bench tasks: arm balance scale, arm Lego manual, humanoid ninja course
The three public Vals Robo-bench tasks. A coding agent gets only the robot's datasheet and must write all the perception and control code itself. No model reached 2 points on humanoid tasks like the ninja course. From Vals AI.

Strengths and limits. Code needs no robot training data, can be read and debugged, and builds a reusable skill library. It is strongest at integration: drivers, sensors, perception, planning. It is weakest where control must react to contact every fraction of a second. That is exactly the gap learned motor models fill.

3. Constraint- and objective-mediated generation

Here the model specifies what a good motion must satisfy, and an optimizer works out the motion. The model writes 3D value maps, keypoint constraints or cost functions; a solver turns them into trajectories and re-solves as the scene changes. Wang et al. treat it as one of three ways a foundation model shapes motion, beside direct action and programs.

Key work

ReKep pipeline: a vision model proposes keypoints, a VLM writes constraint functions, and a constrained optimization solver produces the actions for pouring tea
The ReKep pipeline for "Pour tea into the cup": a VLM writes keypoint constraints as Python functions, and a constrained optimization solver turns them into gripper motion. From the ReKep project page.

Strengths and limits. The optimizer delivers geometric precision and constraint satisfaction that language and code alone struggle with, and the spec is easy to inspect. The weak point is the spec itself. The Goal-VLA authors note that VLM-generated value maps, constraints and rewards are often inaccurate, because current VLMs are weak at precise spatial reasoning.

4. Frontier models in direct control

Here the general-purpose reasoning model is the policy. It looks at the cameras, emits a short chunk of motion through a tool call, looks again and repeats. There is no learned motor model and no program in between. This is the newest approach and the most contested; it only became plausible with the spatial reasoning of the late-2026 frontier models.

How it is run. The open-source Inspect Robots framework puts a frontier model in the slot where a VLA would sit. The model drives the robot through tool calls, one approver-checked motion chunk per call. Every run is clamped to the robot's limits by default, with a cap on how far each step can move.

Results so far

Post by Jay Chooi: GPT-6 Astra scored 95% on a robot control task, up from Fable 5.1's 40%, with a chart of completion rate against API cost
Robocurve's founder Jay Chooi on the block-in-bowl result: GPT-6 Astra 95% against Claude Fable 5.1's 40%, at 2.3 times lower cost. Post on X, 5 September 2026.

The case for it. In a YC interview, Waddle Labs and Robocurve make the bitter-lesson argument: models trained on code, computer use and first-person video will beat models trained only on scarce robot data. They claim that frontier output speed is roughly doubling every month and could allow real-time control by the end of 2026. In the same interview, Waddle Labs said it compresses what the big model learns in context into reusable skills that smaller, faster models can run.

The case against it.

Post by Jay Chooi on the RoboHarm benchmark: chart of harmful attempts against harmful successes for GPT-6 Astra, Fable 5.1 and MolmoAct2, with frames from the trials
The RoboHarm results as Robocurve posted them: Astra attempted harmful actions in 97% of trials and succeeded in 62% of those attempts. Fable 5.1 refused the doll task outright. Post on X, September 2026.

Hybrids: choosing the path step by step

The four paths are not exclusive, and the strongest systems mix them. Envisioning Agentic Robotics argues that the interfaces don't form a scale from easy to hard. Each brings its own priors, control burden and failure modes. Its authors recommend testing three set-ups side by side: skills only, direct end-effector control, and a hybrid where the model chooses per step.

The result they cite shows why: in one benchmark, GPT-6 Astra scored 92% with native control, fell to 61% when restricted to skills, and recovered to 85% when it could decide per skill whether to use one. Forcing a strong reasoner through a narrow interface cost more than it gained.

Beyond run time: agents that improve robots

Two activities change what a robot runs on, not how it acts in the moment, and either can sit on top of any of the four approaches. ASPIRE is self-improvement on top of code as policy, and Learning While Deploying is self-improvement on top of a VLA stack. The line between the two activities is blurry: ENPIRE generates its own rollouts while improving policies, and RoboGen builds worlds and learns skills in the same loop.

Self-improvement

Here the model's decisions outlast the episode: it makes lasting changes to the robot's policies, rewards, code or skills. Wang et al. split this into two kinds, and the second is where 2026's activity is.

Post by Jim Fan introducing ENPIRE: 8 Codex agents given a fleet of robots, GPUs and a token budget, with a video still of a robot hand at a pin box
NVIDIA's Jim Fan introducing ENPIRE as "AutoResearch in the physical world": eight Codex agents with a robot fleet, GPUs and a token budget. Post on X, June 2026.

This approach is still rare. Persistent improvement shows up in about one system in five each year, with no clear upward trend. What changed in 2026 is that frontier coding agents started directing the loop instead of feeding candidates into it.

Data and world generation

Here the model doesn't control the robot or change its code. It decides what the robot learns from and is tested on. Qin et al. give this two of their four domains. Data covers collection, annotation and refinement. Environment covers rebuilding scenes, generating tasks and building benchmarks.

The catch is the gap to reality. Envisioning Agentic Robotics notes that synthetic data still carries a sim-to-real gap, so generated worlds supplement real deployment data rather than replace it.

Side by side

Dexterity trades against generality: the less a system relies on learned motor models, the less robot data it needs, and the worse it handles contact.

Approach What the model outputs Robot data needed Strongest at Weakest at Examples
1. Layered orchestration Subgoals, subgoal images, grasp marks Yes, for every motor model Dexterous skills inside long tasks Handoffs between policies; tasks outside the tools' range Gemini Robotics 2.5, GR00T 2 with a reasoner, VLAs-as-Tools, RoboHarness
2. Code as policy A program over perception and control primitives No Integration, perception, planning, reusable skills Fast closed-loop contact CaP-X, Vals Robo-bench agents, Project Fetch
3. Constraint- and objective-mediated Value maps, keypoint constraints, costs No Geometric precision via an optimizer Inaccurate specs from weak VLM spatial reasoning ReKep, VoxPoser, EmboAlign
4. Direct control Motion chunks, one per tool call No Novel tasks with no setup Latency, cost per action, precision, safety Robocurve trials, Fable 5.2 on RoboDojo

Where the lines blur, and what is still open

The approaches are starting to mix. A coding agent can call a VLA as one of its primitives. Waddle Labs distills what a frontier model learns in context into skills that smaller, faster models run. Anthropic hasn't said whether Fable 5.2's RoboDojo result came from direct control, code or skill models. Meanwhile, motor models are absorbing agent functions themselves: π0.7 takes coaching mid-task, and Physical Intelligence's MEM gives a VLA language-based memory for tasks up to about 15 minutes.

Open questions:

  1. Inside or outside? Will planning, memory and recovery stay in a harness around the motor model, or move into the model?
  2. Speed and cost. Can frontier models get fast and cheap enough for direct control of contact-rich work, or will they always need a learned layer below?
  3. The interface. Which language will reasoners and motor layers settle on: text, pictures (subgoal images, marks) or code?
  4. Evaluation. Most numbers here come from the labs that built the systems. Neutral evaluations (Robocurve, Vals, the RoboDojo leaderboard) are still small, run few trials, and disagree on setup.
  5. Safety at the action layer. DeepMind's ASIMOV-Jailbreak benchmark reports 99.5% jailbreak resistance for Gemini 4 Argon, against 82% for Gemini Robotics ER 2. RoboHarm suggests that frontier models in direct control mostly comply with harmful commands.

Demo videos and project pages

Approach Project Where to watch
1. Layered orchestration Gemini Robotics 2.5 (DeepMind) Launch post
1. Layered orchestration GR00T 2 with a reasoning model (NVIDIA) Project page
1. Layered orchestration VoLo (NVIDIA) Project page
1. Layered orchestration Hierarchical VLA agents study Project page
1. Layered orchestration π0.7 (Physical Intelligence) Project page
2. Code as policy Vals Robo-bench Leaderboard and run examples
3. Constraints ReKep Project page
4. Direct control Robocurve trials Jay Chooi's posts
Beyond run time ENPIRE (NVIDIA and others) Project page
Beyond run time ASPIRE (NVIDIA and others) Project page

Sources

Surveys

Systems and results