Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors

Abstract

Bridging the sim-to-real gap is a core challenge in deploying learned manipulation policies. Sim-to-real learning is attractive because it can replace expensive real robot demonstrations with scalable synthetic data, yet world-action models have not previously been shown to transfer from simulation to real robotic manipulation. We study whether a world-action model can be trained from synthetic priors and deployed zero-shot in the real world. To this end, we build upon Cosmos Policy, a video diffusion model adapted for visuomotor control. We construct simulation environments with extensive domain randomization and generate demonstrations using the AnyTask motion planning pipeline. We evaluate our approach across object lifting, drawer opening, and pick-and-place tasks using ~800 synthetic demonstrations per task and no real demonstrations. When deployed zero-shot on a Franka Robot, our policy attains a 35% average success rate. To our knowledge, this represents the first successful sim-to-real transfer of a world-action model for robotic manipulation.

Key Contributions

• First reported successful sim-to-real transfer of a world-action model (as distinct from a standard behavior-cloned policy) for robotic manipulation — prior world-action-model work had not been shown to survive the simulation-to-reality jump at all.
• Builds on Cosmos Policy (a video-diffusion model repurposed for visuomotor control), showing that generative-video world models can be adapted into deployable action policies.
• Demonstrates that zero real-world demonstrations are needed: ~800 synthetic demonstrations per task, generated via the AnyTask motion-planning pipeline inside heavily domain-randomized simulation, are sufficient for a non-trivial (35%) real-world zero-shot success rate.

Methodology

The authors adapt Cosmos Policy — a video diffusion model — for visuomotor control. Simulation environments are built with extensive domain randomization, and demonstrations are synthetically generated via the AnyTask motion-planning pipeline (no human teleoperation or real-robot data involved). The resulting world-action model is evaluated across three task families — object lifting, drawer opening, and pick-and-place — using approximately 800 synthetic demonstrations per task, then deployed zero-shot (no real-world fine-tuning) onto a physical Franka robot arm.

Results

• 35% average success rate across the three task families when deployed zero-shot on a real Franka robot, using only synthetic training data.
• Authors position this as the first successful sim-to-real transfer specifically of a world-action model (video-diffusion-based) for manipulation, distinguishing it from the broader (and more mature) sim-to-real literature for conventional RL/behavior-cloning policies.

Limitations

35% average success is well below production-grade reliability, and the paper does not claim otherwise — it is presented as a proof-of-concept establishing feasibility, not a deployment-ready result. Evaluation is limited to three relatively simple task families (lifting, drawer-opening, pick-and-place) on a single robot platform (Franka arm); generalization to contact-rich dexterous manipulation or humanoid whole-body control is untested.

Full Content

This paper is the direct sim-to-real complement to the KB's existing "World Models for Robot Learning" frontier: where 1X's NEO world model and NVIDIA's Isaac Sim address environment understanding and simulation fidelity, this work asks whether a generative video world model can itself become the action-producing policy, and whether that policy survives sim-to-real transfer using only synthetic priors. The 35% zero-shot number is modest in absolute terms but is the more important data point structurally — it is a "does this work at all" result for an approach (video-diffusion-as-controller) that could, if it scales, sidestep the teleoperation-data bottleneck other labs (Xiaomi, Figure, 1X) are solving by scaling UMI/mocap capture instead. It sits at the intersection of two frontier areas already tracked here: "Sim-to-Real at Production Scale" and "World Models for Robot Learning."

Source: Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors by Zixing Wang et al.