Cross-Controller Transfer via In-Context Behavior Adaptation

Jonathan Yang, Chelsea Finn, Dorsa Sadigh · Stanford University

We train one robot policy that adapts to unseen controllers by conditioning on interaction history. We then show that robots can learn from data collected under many different controllers and adapt their strategy to new tasks.

Abstract

Generalist robot policies increasingly learn from data collected across diverse tasks and embodiments, yet typically assume a single controller for each robot. This assumption limits transfer because different controllers can assign different physical meanings to the same action and may require different strategies to complete a task. We study how to transfer robot data across both embodiments and controllers without allowing conflicting action interpretations to interfere with learning.

Our setting spans six controller types as well as variations in controller gains and compliance. We propose a framework that encodes behavioral interaction under a controller through examples of commanded actions and the resulting robot motions. A transformer context encoder summarizes these interactions, allowing a single policy to adapt in context. Across real-world experiments, our method improves average success by 33 percentage points over naive data pooling, achieves 50% success on unseen controller modes and 52% on unseen controller gains, and transfers to tasks with limited controller-specific data. On contact-rich tasks such as USB insertion and surface wiping, behavioral interaction context enables the policy to adapt not only its actions but also its strategy.

The problem

The same action vector means different things under different controllers

VR teleoperation might record compliant Cartesian commands; puppeteering might record stiff joint targets. Pool the two and the same action a carries two physical meanings. Large multi-robot datasets sidestep this because each controller lives in its own lab, so the scene gives it away. Once one robot runs several controllers, it doesn't.

Controllers differ along two axes. The controller type sets what an action means: a joint-position target, a joint velocity, an end-effector pose solved through IK, or an impedance setpoint the arm is pulled toward like a spring. The gains set how hard it pulls. A high proportional gain Kp makes the arm stiff: it tracks commands precisely and pushes hard against anything in its way. A low Kp makes it compliant: it lags its commands and yields on contact. The derivative gain Kd damps the motion. We use six controller types with four gain settings each, and evaluate unseen joint-impedance gains from (0.28, 0.12) to (2.44, 1.32) kN/m. Two controllers can share an action space and still need different behavior, because the gains change what happens the moment the robot touches something.

The obvious fix is system identification: estimate the controller, then relabel each demonstrated command so the target controller produces the same torque. For the PD law shared by all our controllers, that remap is

q′d = q + (K′)−1[ K(qd − q) + (D − D′)(q̇d − q̇) ]

This reproduces how an existing motion is realized under new dynamics. It cannot invent the trajectory an expert would have chosen. Knowing the dynamics pc(ot+1 | ot, at) doesn't determine the behavior p(at | ot, ℓ, c).

SysID remapping. Identify how controller A tracks a command, then relabel the demonstration so controller B produces the same torque.

Remapping the action does not recover the expert behavior

dynamics remapping Torque-matched relabeling freezes on the board. It preserves a path that no longer works under the new controller.
stiff controller What an expert does instead. A different strategy, not the same path with different commands.
+33%average success over naive data pooling
50%success on unseen controller modes
52%success on unseen controller gains
78–92%of in-domain performance kept when context is reused across tasks
Why remapping falls short

The best strategy depends on the controller

Two wiping strategies evaluated across a grid of Kp and Kd gains. The compliant strategy succeeds 80 to 90 percent at low stiffness; the stiff strategy succeeds 75 to 85 percent at high stiffness. Below, side-view photos show the compliant strategy sweeping the eraser through contact and the stiff strategy tracing a straight line.

We evaluated two learned wiping strategies across the same (Kp, Kd) plane. At low stiffness, compliant sweeping reaches 80–90% while stiff tracing manages only 15–25%. At high stiffness the order flips: stiff tracing reaches 75–85% and compliant sweeping overloads under excessive contact force.

Neither strategy wins everywhere. So identifying the gains isn't enough; the policy has to choose behavior that suits them.

Method

We condition the policy on how the robot behaves under its controller

The policy receives two kinds of context. Nominal metadata ηc is a short text string naming the controller type and gains. Behavioral interaction context is a few trajectories recorded under the deployment controller, often on a different task, pairing camera frames and commanded actions with the motion the robot actually achieved.

A Perceiver-style encoder compresses that history into a fixed set of tokens. Controller-revealing events such as contact, braking, saturation and recovery occur at different times in each task, so the encoder attends over whole trajectories instead of a fixed crop.

Relabeling returns in a narrower role, as offline augmentation. It fills empty task–controller cells but leaves behind conflicting strategies under a single label. The behavioral context tells the policy which of those strategies to follow.

Backbone
π0.5 VLA, flow matching on 20-step action chunks
Vision
Frozen DINOv2 ViT-B/14, patches projected 768 → 512
Transitions
(q, acmd, qach) through a shallow MLP to 512-d tokens
Bottleneck
256 learned latent queries, cross-attention then 6 self-attention blocks
Context
N = 3 trajectories, T = 128 transitions
Inference
Context encoded once at t = 0 and KV-cached, so per-step latency is constant
Controllers
  • joint position
  • joint velocity
  • IK pose
  • Cartesian pose
  • joint impedance
  • Cartesian impedance
Data
4 tasks × 6 controllers × 4 gains × 30 demos = 2,880 trajectories
Rollouts

The policy adapts its strategy to the controller

Every clip is a real-world rollout of the same trained policy. Each row fixes the controller the robot runs at deployment. Each column fixes the conditioning: we encode an interaction trajectory recorded under a stiff or compliant controller and pass the resulting tokens to the policy, which then carries out the strategy that context implies. On the diagonal the context matches the controller; off the diagonal it comes from the other one.

Rows: deployment controller. Columns: the controller the encoded context trajectory was recorded under, which sets the strategy. Inset: wrist camera.

Results

Q1 · Is knowing the controller dynamics enough?
Bar charts of zero-shot and few-shot success across five tasks for controller conditioning, SysID, ours, and a same-controller upper bound.

No. On held-out task–controller cells, conditioning on ground-truth controller metadata reaches 12% zero-shot and SysID reaches 27%. Cross-task behavioral context reaches 50%, against a same-controller upper bound of 75%. With five target demonstrations the gap holds: 69% vs. 47% for SysID. N = 20 trials per cell.

Q2 · Does it generalize to unseen controllers and gains?
Bar charts of success on six held-out controller modes and four unseen joint-impedance gain pairs for joint policy, relabelling, and ours.

Yes. With the controller mode held out entirely, success rises from 17.5% for naive pooling to 50.3% with context. Relabeling alone drops to 15.0% because its actions conflict under identical labels. On four unseen joint-impedance gain pairs, success is 52.0%, against 28.5% for pooling and 23.5% for relabeling. Error bars show ±SE.

Q3 · What information in the context matters?
Left: success for combinations of text, context, and relabelling; all three reach 0.50. Right: success versus context timesteps for contact-rich, full, free-space, and proprio-only context.

Relabeling provides coverage and behavioral context resolves the ambiguity. Relabeling alone lowers success from 0.24 to 0.15; adding context raises it to 0.47, or 0.50 with text. Contact-rich transitions carry the most signal. Uniform full-trajectory context reaches 0.49 once the horizon is long enough to include contact, while free-space motion and proprioception alone stay at 0.30.

Q4 · Does context transfer across tasks?

We encoded interaction context from tool-bit insertion and used it to condition USB insertion under the same compliant controller. The reused context reached 35% success, 78% of the 45% achieved with in-domain USB context. Context from whiteboard wiping transfers to pancake flipping the same way: 55% success, 92% of the 60% with in-domain context. On USB, the policy picks up the compliant search it saw in tool-bit insertion: when it meets resistance at the port, it yields and keeps gentle surface contact while it jiggles.

Dumbbell plot of target-task success. Tool-bit to USB: in-domain context 45 percent, reused 35 percent, 78 percent retained. Wiping to flipping: in-domain 60 percent, reused 55 percent, 92 percent retained.
Citation
@article{yang2026crosscontroller,
  title   = {Cross-Controller Transfer via In-Context Behavior Adaptation},
  author  = {Yang, Jonathan and Finn, Chelsea and Sadigh, Dorsa},
  year    = {2026}
}