World Models

World models are internal representations that allow robots to predict how their environment will change in response to actions. Unlike reactive control — where the robot responds to current sensor readings — world models enable planning by mentally simulating the consequences of potential actions before executing them. This is analogous to how humans can imagine the outcome of reaching for a cup before moving their arm.

1X Technologies released a visual perception world model for their NEO humanoid robot. The model enables NEO to observe its environment and predict future states, forming the basis for what 1X describes as enabling robots to "teach themselves new tasks" through observation rather than explicit programming. This represents a shift from scripted or imitation-based behaviors toward autonomous skill acquisition.

NVIDIA's Isaac Sim 5.1 takes a complementary approach: rather than building an internal world model, it creates a high-fidelity external simulation that serves a similar function. The deliberate injection of sensor imperfections (noise, latency, miscalibration) forces training policies to develop internal robustness — effectively requiring the learned policy to maintain its own implicit world model that accounts for uncertainty.

The distinction between internal world models (learned by the robot) and external simulation (built by engineers) is blurring. As learned models improve, they may eventually replace hand-crafted simulations for certain tasks, while simulation remains essential for domains where prediction errors are costly.

Key Claims

• Visual perception world model enables autonomous learning — 1X's NEO world model allows the robot to observe and predict environment states, supporting self-directed skill acquisition. Evidence: strong (1X NEO World Model)
• Deliberate imperfection forces implicit world modeling — Isaac Sim 5.1 injects sensor noise and latency during training, requiring policies to develop internal robustness to real-world conditions. Evidence: strong (ABB/NVIDIA RobotStudio HyperReality)
• Internal and external world models are converging — Learned predictive models and high-fidelity simulations serve overlapping functions, with the boundary between them increasingly fluid. Evidence: moderate

Open Questions

• Can world models scale to unstructured home environments where object types, arrangements, and interactions are highly variable?
• What are the real-time inference constraints — can world models predict fast enough for reactive manipulation tasks?
• How do world models handle novel objects and materials not seen during training?
• What is the right architecture for world models — pixel-level prediction, abstract state spaces, or hybrid approaches?

Related Concepts

• Sim-to-Real Transfer — External simulation as a form of world modeling
• Foundation Models for Robotics — Language models encode world knowledge that complements perceptual world models

Related Entities

• 1X Technologies — Visual perception world model for NEO
• NVIDIA — Isaac Sim as high-fidelity external world model

Backlinks

Pages that reference this concept:

• Sim-to-Real Transfer
• Foundation Models for Robotics
• 1X Technologies
• NVIDIA