Research talk · GRAFT (Ch0) · 22 Oct 2026

GRAFT: learning from its own failures
Failure-informed steering of pre-trained flow-based VLAs

A released VLA collects its own successes and failures. We fit both outcomes as conditions inside the same action generator, then steer every denoising step away from the failure prediction.

Own experience, no new demosZero-init FiLM retrofit Velocity-field steeringΨ₀ · GR00T-N1.6

Haonan Zhou1, Gaoyang Pang1, Karl H. Johansson2, Daniel E. Quevedo1, Branka Vucetic1, Yonghui Li1

1 The University of Sydney, Australia  ·  2 KTH Royal Institute of Technology, Sweden

Submitted to ICRA 2027 (15 Sep 2026, under review) · ← → / Space to navigate · F for fullscreen

ASPIRE × SSS Global Forum 2026
Institute of Science Tokyo · 26–29 Oct 2026 Academy of Super Smart Society, Science Tokyo Institute of Science Tokyo
Background · motivation

A deployed VLA fails often, and those failures go unused

Expert demosteleoperation / plannerscostly to collect Pre-trained flow VLAimitation: copy the demose.g. Ψ₀ · GR00T-N1.6 Deploy: its own rollouts from one start same scene, same instruction, different outcomes ✓ ✓ ✗ ✓ ✗ ✓ ✗ ✓ Can these, failures included, improve the policy with no new demos?

Imitate everything

Mixture-SFT: failed actions become targets to copy.

Re-weight or filter

AWR / filtering: failures fade out, and the outcome label is gone by test time.

Heavier routes

RECAP (π*0.6) adds conditioning during generalist pre-training. AFIL trains and serves a second, failure-only generator.

Motor-learning analogy: outcome feedback changes later behaviour without a corrected movement to copy. We want the same from a VLA's own failures.

Our idea

Keep failures as a contrast, not as targets

WHAT HAPPENS TO A FAILED ROLLOUTAT TEST TIME Mixture-SFTlabels ignored ✗ copied like a success one policy, failure modes partly imitated AWRw = exp(Â/β) ✗ down-weighted one policy; the outcome signal is discarded GRAFTsigned condition ✗ fitted under condition “−”the model learns f₋ as well as f₊ steer: C = 2f₊ − f₋ = f₊ + (f₊ − f₋)push the success prediction away from the failure one

No new demos, no critic

Labels come from repeated outcomes of the policy's own rollouts.

One model

Both branches share all weights: two batched passes per step, with no second generator.

Retrofit

Starts exactly at the released checkpoint (zero-init FiLM) and keeps its native flow-matching loss.

Test-time knob

Steering strength (α, λ) is set at inference, with no retraining.

Method · three stages

Label rollouts → fit both outcomes → compose

① SIGNED LABELS② FIT (NATIVE LOSS)③ COMPOSE start 1start 2start 3 ✓✓ ✗✓ ✗✓ ✓✓ ✓✓ ✓✓ ✗✗ ✓✗ ✗✗ V̂ = .67✓ +.33 → z = +✗ −.67 → z = − V̂ = 1all  = 0 → z = 0no contrast V̂ = .17✓ +.83 → z = +✗ −.17 → z = − Âij = Gij − V̂i, z = sgn  V̂ᵢ = success rate at start i · all chunks get z · no critic One action generator fω(Aτ, τ | o, ℓ, c) success rows → c = + failure rows → c = − (still targets) z = 0, demos, dropout → no condition loss = the model's own flow-matching loss,unchanged · condition dropout keeps theunconditioned path trained Every denoising step f₊ = f(A, τ | o, ℓ, +) f₋ = f(A, τ | o, ℓ, −) C = 2f₊ − f₋ same weights, one batch of two the current attempt's outcome isnever needed at test time details: slide 6
Ψ₀ corpus · tabletop grasp, 10 starts × 72 own rollouts = 720, plus 100 public demos (fitted without a condition).
GR00T corpus · apple-to-plate, 32 starts × 16 rollouts = 512 episodes: 246 +, 202 −, 64 with z = 0.
Retrofit · where the condition enters Ψ₀

FiLM sits between the VLM and the action expert

Camera viewsimages Task textbare at serving Qwen3-VL-2Bvision–language backboneupdated · lr 1e-5 VL tokens hN × 1536 FiLM h′ = (1 + γ(c)) ⊙ h + δ(c) per-channel γ, δ, shared byevery VL token new · lr 3e-5 + state tokenproprioceptionnot modulated Action expertMM-DiT, ~500M params joint attention over[ h′ ; state ; Aτ ] updated · lr 3e-5input: noisy chunk Aτ, flow time τ velocity f(Aτ, τ | o, ℓ, c)flow velocity, not joint velocity 10 Euler steps → chunk of 30execute 24 (real-time chunking) condition code c ∈ ℝ²⁰⁴⁸ + − fixed codes: ones on disjoint halves γ(c) = Wγc + bγ, δ(c) = Wδc + bδ 2 × Linear(2048 → 1536), ≈ 6.3 M params all W and b start at exactly 0
Ψ₀ primary recipe · 3,000 steps, batch 128. 70% of ± rows carry the FiLM code plus a continuous text suffix; the rest, all z = 0 rows and all demos take the bypass path. Serving uses bare task text and the FiLM code alone.
GR00T-N1.6 is different · its action head cross-attends to the VLM. Primary carrier: a text suffix <|pace_adv|>+0.900000<|/pace_adv|>, with the VLM frozen and only the action head updated. The FiLM variants modulate the state + action query stream.
Test time · how steering works

We steer the flow's velocity field, not the joint commands

noise A⁰ ~ N(0, I)the whole action chunk (Ψ₀: 30 actions) repeat K times (Ψ₀: K = 10 · GR00T: K = 4) f₊ = f(A, τ | +)same weights f₋ = f(A, τ | −)one batch of two v = 2f₊ − f₋change the direction of motion A ← A + Δτ · v action chunk A¹execute the first 24 (Ψ₀) / 20 (GR00T) robot executes as-is steer here ✗ not here success actions failure actions background: field at τ = 0.4 · f₊ alone 2f₊ − f₋ (steered) noise A⁰ P = f₊lands between R = frN = f₋C = 2f₊ − f₋ dots = the 10 Euler steps (Ψ₀ uses 10) One denoising step, zoomedsame A, same τ, three readouts A f₊ fr f₋ +(f₊−f₋) C = 2f₊ − f₋ the difference f₊ − f₋points away from failure
Two different “velocities”. Joint velocity q̇ lives in robot time t. Flow velocity v = dA/dτ lives in denoising time τ and moves the whole 30-action chunk at once. GRAFT edits only v.
Why in the field? After each nudge the model re-evaluates at the new A, so the next step can correct it, and the chunk is still produced by the generator and stays coherent over time. General form: v = fr + α(f₊ − fr) − λ(f₋ − fr). C is (α, λ) = (2, 1), where fr cancels.

Right panel: an illustrative 2-D toy (two Gaussian action modes, exact flow-matching field, 10 Euler steps; fr = the unconditioned field of all ± data), not data from either robot.

Evidence · closed-loop simulation, Unitree G1 · same start, base vs. GRAFT

Composition beats the base and every direct readout

Ψ₀ · tabletop grasp

Head camera, one frame per action chunk (24 control steps) · start 0

✗ Base
Hovers near the box; no lift in 800 steps
✓ GRAFT C
Grasps and lifts the box by step 188

Base's weakest start: 54% → 91% with C. 4 of 6 starts improved, 2 reduced.

6 held-out starts · R, L, C: 384 episodes · P, N: 96 · base: 768 Base79.6% R reference51.0% P f₊ only70.8% N f₋ only35.4% L 2fr − f₋63.8% C 2f₊ − f₋94.0% C − base = +14.5 pp [+3.9, +25.1]C − P = +23.2 pp · C − R = +43.0 pp · C − L = +30.2 pp

GR00T-N1.6 · apple to plate

Ego camera, recorded at 50 fps, shown at 2× · start 600

✗ Base
Pushes the apple around, never lifts it
✓ GRAFT C
Grasps, turns, places it on the plate

Over all 32 starts: 17 improved, 7 unchanged, 8 reduced.

32 starts · 384 episodes per bar Base51.8% R reference53.9% P f₊ only55.7% N f₋ only47.7% L 2fr − f₋59.9% C 2f₊ − f₋62.0% C − base = +10.2 pp [+1.6, +19.5]C − P = +6.3 pp · C − R = +8.1 pp · C − L: CI spans 0

Labels matter. Ψ₀: shuffling labels across episodes (same trajectories, same recipe) costs composition 22.4 pp [12.8, 30.2], at every start.

Same corpus, other uses. Ψ₀: Mixture-SFT −17.8, AWR −20.2, GRAFT +14.5 pp vs base.
GR00T: +2.6 / +7.8 / +10.2 pp.

More experience helps. Refitting on new poses: GR00T C gains +9.1 pp [3.1, 15.4] over zero-shot. Ψ₀ C reaches +15.8 pp over its pose base.

Paired 95% CIs (10,000 bootstrap draws over starts) · one seed per fit · carriers chosen after the sweeps · dashed = base · per-start counts: paper Fig. 3d · videos: hand-picked episodes from the paper's runs, behaviour only (media/SOURCES.md).

Summary

Takeaways, limits and what comes next

Failures as a contrast

Fitting f₋ and steering away from it beats using f₊ alone, on both models.

A safe retrofit

Zero-init plus an exact bypass: it starts at the released policy, and the condition can always be switched off.

Steer in the field

Composition acts on dA/dτ at every denoising step, with no edits to the robot's commands and no second model.

ScopeLIMITTwo simulated G1 tasks; held-out poses only within the sampled ranges
LabelsLIMITEpisode-level, need repeated resets, and cannot say which action caused a failure
CarrierLIMITModel-dependent: FiLM for Ψ₀, text for GR00T; not a causal claim about attention types
PaperUNDER REVIEWICRA 2027, submitted 15 Sep 2026; decision expected 31 Jan 2027
NextIN PROGRESSA learned stage-wise critic (PACE graph model) to replace repeated-reset labels, then the real robot
All numbers come from the submitted paper (Figs. 3–5) and its result files in the paper/ submodule; the videos are episodes from its evaluation runs. The toy field on slide 6 is illustrative.
Team · authors of the ICRA 2027 submission

The GRAFT team

Haonan Zhou

Haonan Zhou ↗

PhD Student
The University of Sydney

Verified, graph-anchored progress for humanoid vision-language-action (VLA) manipulation

Gaoyang Pang

Gaoyang Pang ↗

Research Associate
The University of Sydney

Wireless communications and control for networked human–machine collaboration

PhD, The University of Sydney (2025)

Karl H. Johansson

Karl H. Johansson ↗

Swedish Research Council Distinguished Professor
KTH Royal Institute of Technology, Sweden

Networked control and cyber-physical systems

Founding Director, Digital Futures · IEEE Fellow

Daniel E. Quevedo

Daniel E. Quevedo ↗

Professor · Deputy Head of School
The University of Sydney

Networked control systems, cyber-physical security and control of power converters

IEEE Fellow · Axelby Outstanding Paper Award (2018)

Branka Vucetic

Branka Vucetic ↗

Australian Laureate Professor
The University of Sydney

Wireless communications and IoT: coding, low-latency communications, 6G

FAA · FTSE · FIEEE · Eureka Prize (2019)

Yonghui Li

Yonghui Li ↗

Professor · Director, Wireless Engineering Laboratory
The University of Sydney

Wireless communications, IoT and wireless networked control

IEEE Fellow · ARC Industry Laureate Fellow (2025)

Resources · hardware

Our robot platforms

Unitree H2 humanoid robot

Unitree H2 ×1

Full-size humanoid, 180 cm
Unitree Dex5 dexterous hands
NVIDIA Jetson Thor

Unitree G1 humanoid robot

Unitree G1 ×1

Humanoid, 132 cm
Unitree Dex3 dexterous hands
NVIDIA Jetson Orin

Unitree R1 humanoid robot

Unitree R1 ×1

Compact humanoid, ≤123 cm, about 29 kg
NVIDIA Jetson Orin

Unitree Go2 quadruped robot

Unitree Go2 ×2

Quadruped, 70 × 31 × 40 cm

Mountable accessories: a Hesai XT16 3D LiDAR · a robotic arm with a gripper for the Go2 · a set of dexterous hands for our humanoid robots.
Backup · figures from the paper

New poses, other uses of the data, and update scope

Paper Fig. 4: held-out pose performance, zero-shot and refitted, relative to pose-specific bases

Fig. 4 · held-out poses. Refitting on new own experience beats zero-shot serving: GR00T C gains +9.1 pp [3.1, 15.4] over zero-shot, and the Ψ₀ continuous refit reaches +15.8 pp over its pose base.

Paper Fig. 5: Mixture-SFT, AWR and GRAFT relative to base, label-shuffle control, and VLM update scope

Fig. 5 · controls. (a) On the same corpus, Mixture-SFT and AWR hurt Ψ₀ while GRAFT helps both models; true labels beat shuffled ones by +22.4 pp. (b) On GR00T, freezing the VLM wins in all five carrier pairs.

Figures exported unchanged from the submitted PDF (paper/figs/). Whiskers are paired 95% CIs.