Teach and Grow.

Teach and Grow (TGL) is a training-free architecture for general robot learning. A pretrained AI Agent turns a few successful demonstrations into explicit, reusable skills, so the robot acquires new manipulation tasks while its pretrained model weights stay fixed.

It means acquiring the incoming task invokes no gradient update, no fine-tuning, and no reinforcement-learning stage. The AI Agent and its specialist models may already be pretrained; what changes during task acquisition is the explicit skill and memory state, not the weights.

It is the recurring cost of repairing robot behavior through a policy update: new data collection, another optimization run, and regression checks on everything the policy already supported. It is called a tax because the cost returns with every new task, sensor, or gripper, and grows with the amount of behavior that already works.

VLA and world-action models absorb a new behavior by collecting more robot data and optimizing policy parameters. TGL repairs and extends behavior through explicit Skill Blocks instead. A learned policy can still take part: it may act as the executor inside a block, or later serve as a student of verified trajectories.

A Skill Block is the unit of reusable behavior: a goal, a reusable strategy, supported conditions, compatible executors, and an outcome test. The semantic effect is what gets retained, while the physical realization (object bindings, grasp geometry, collision-free motion) is recomputed from the current scene. An acquisition block, for instance, is never satisfied by a closed gripper alone.

The implementation uses OpenAI GPT-6 Astra for multimodal reasoning and Codex to connect the AI Agent to robot tools. Detection, segmentation, RGB-D geometry, Contact-GraspNet, MPLib and controllers supply the physical grounding and execution, evaluated in the LIBERO simulation suite.

Because finishing a task once and building a system that keeps getting better at it are different things. We started from exactly that experiment: an AI Agent driving the robot through perception, grasping, planning and control tools completed manipulation tasks with no teaching at all. What did not happen is accumulation. Every run reasoned from scratch, the successful route and the grasp that worked disappeared when the episode ended, and a mature behavior was never cheaper the second time. TGL keeps the same AI Agent loop and changes where the result is stored. A task that succeeds is split into explicit Skill Blocks with a stated scope and an outcome test, and the conditions and repairs go into Experience Memory. From then on the AI Agent retrieves an existing skill instead of re-deriving it, converges faster because the subgoal structure is already settled, reuses verified behavior across tasks, and carries the accumulated experience into later work. An AI Agent can solve a task on its own; the architecture is what turns that one-off success into a system that grows, rather than a run that is thrown away after use.

Autonomous exploration works, and we ran zero-shot AI Agent control, so this is a design choice rather than a limitation. It is a poor default for two reasons. Convergence is slow, because learning one physical behavior by trial and error consumes many robot interactions and many sequential model calls, as every attempt needs a fresh decision. And on real hardware it is unsafe to leave unbounded, since an exploratory action is a physical action and a wrong one can damage the object, the gripper or the scene. Few-shot teaching changes the starting point rather than the goal: a few successful demonstrations supply the subgoal order and the conditions worth checking, so exploration can concentrate on the variation, correction and recovery the demonstrations did not cover. Teaching is an accelerator, not a precondition. The source of a demonstration is open as well. Robot trajectories, simulation, human video and a written procedure all count, because anything that shows how the task is completed and what it accomplishes can provide the initial structure. A manual may reveal the order of operations while leaving the grasp unresolved, and the architecture keeps that distinction: semantic knowledge guides acquisition, while robot-specific grounding and validation decide what can actually run.

Because the AI Agent's own visual judgment is not precise enough to manipulate with. In our experiments it handled simple tasks from its own image understanding alone: recognizing which object is meant, choosing a rough order of operations, judging whether a scene looks like the goal. Measurement is where it becomes unreliable. A 6-DoF pose, a centimetre-scale clearance, the boundary of an occluded object, whether the gripper is actually holding something, whether a path would collide: these need metric answers that hold from one frame to the next, and a general-purpose model reading a camera image gives approximate ones. Dedicated tools close that gap. Detection and segmentation establish object identity and boundary, RGB-D geometry supplies depth and pose, grasp and motion planners produce collision-free reachable actions, and controllers hold the loop at the required rate. The AI Agent is then free to do what it is good at: organizing the task, choosing the next subgoal, reading the outcome, deciding what comes next. The split also keeps the system maintainable, because a detector is a replaceable component inside a Skill Block that can be swapped and retested without touching the reasoning, the library or the rest of the pipeline.

TGL is a new AI Agent-driven general-purpose robot operating system, and it is designed to work with VLA and world-action models rather than replace them. The two sit at different levels. A VLA or WAM remains the strongest available way to turn perception into fast continuous control; TGL supplies the operating layer above it, which decides what subgoal comes next, which capability applies in this scene, what actually happened, and what should be kept. In that arrangement a learned policy is a component the system calls. A policy trained for a bounded task, such as picking one category of object or opening one kind of drawer, can be registered as the executor of a Skill Block: the AI Agent selects it, grounds it in the current scene, checks its outcome and composes it with other blocks, and a mature block can run entirely inside the fast policy once the behavior is stable. The relationship is therefore composition, not competition. Where a VLA is strong, TGL uses it directly and benefits from every improvement to it; where a task is new, rare or outside the policy's distribution, the AI Agent acquires the missing structure explicitly instead of waiting for the next training round.

On unfamiliar tasks it is slow, and that trade is deliberate: deliberation is placed at subgoal boundaries while the executors below run continuous control at full rate. Three things follow. First, the cost is falling on its own, because each generation of reasoning models is faster and cheaper at the same capability, and prompt caching, tool-call batching and stronger multimodal perception keep reducing the number of sequential calls a task needs, so an architecture that puts reasoning at semantic boundaries benefits from that trend directly. Second, slow acquisition is what makes TGL a strong data source. A general method that can acquire an unfamiliar manipulation task produces, along the way, verified trajectories: real observations, real actions and real outcomes, already checked against an explicit success criterion. That is exactly the supervision a small fast policy, a VLA or a WAM needs, and it covers precisely the long-tail conditions that are hardest to collect by hand, so the system serves as a data generator for the fast models it later calls. Third, the two layers occupy different positions rather than competing: the agentic layer is strong and slow and handles new tasks, rare conditions and data collection, while the fast layer is narrow and quick and runs mature behavior at policy speed. Data flows from the slow side to the fast side, and when a fast policy meets something outside its competence, control returns to the AI Agent, which diagnoses the gap and grows the library.

Several recent systems work on neighbouring pieces, and the paper cites them: LRLL, ASPIRE, SkillMemo, SCE and PACTS study lifelong skill acquisition, agentic discovery, memory and compositional reuse, while PhyAgentOS, AEROS and RoboBridge build robot operating layers. Individual ingredients such as skill libraries, agentic tool use, episodic memory and demonstration decomposition are not new, and TGL does not claim them. TGL is the first system to propose the method as a whole: an AI Agent-driven general-purpose robot operating system that connects sparse teaching, explicit closed-loop Skill Blocks carrying a scope and an outcome test, weight-frozen execution, physical feedback and recomposition, structured failure memory, and persistent growth into one single learning cycle. Each neighbouring method covers part of that cycle, so the contribution here is the cycle itself and the interfaces between its parts. That whole-system view is what makes the practical properties available, namely converging faster on a new task, reusing a verified behavior instead of re-deriving it, keeping experience across tasks and embodiments, and serving as a data source for fast policies.