Foundation Models for Robotics

Foundation models — large-scale vision-language models like GPT-4V — are enabling a paradigm shift in robot control: from task-specific training to zero-shot behavior generation from natural language instructions. Rather than collecting thousands of demonstrations for each new task, robots can leverage the broad knowledge already encoded in these models to interpret scenes, decompose tasks, and generate executable actions.

Wen et al.'s Humanoid-COA is the first framework that integrates foundation model reasoning specifically for zero-shot humanoid loco-manipulation. The architecture follows a perception-reasoning-action paradigm: GPT-4V handles scene understanding and spatial reasoning, the Chain of Action (CoA) mechanism decomposes high-level instructions into a sequence of actionable subtasks, and pre-trained locomotion/manipulation controllers execute the resulting behaviors.

The key insight is separation of concerns — the foundation model handles semantic understanding and task planning (what to do and in what order), while specialized controllers handle the physics of execution (how to move). This avoids the need to train end-to-end policies that must simultaneously understand language and control actuators.

Gu et al.'s survey contextualizes this within the broader landscape, noting that learning-based approaches are increasingly displacing model-based methods that required explicit task-specific engineering.

Cao (2024) adds a paradigm-level framing: GenAI and LLM integration is precisely the mechanism by which humanoids transition from the "human-looking" to the "human-like" paradigm. The key enabling technology on the horizon is vision-language-action (VLA) modeling — systems that jointly process visual scenes, natural language instructions, and action histories to generate coherent behavioral outputs in real time. VLA represents the next step beyond current two-stage approaches (VLM reasoning + pre-trained controllers) toward genuinely end-to-end multimodal control.

A distinct 2026 role has emerged alongside "foundation model as the policy": foundation model as training-pipeline infrastructure. DexSim2Real (Zeng et al., May 2026) uses a VLM as a visual-realism critic to optimize simulation parameters (foundation-model-guided domain randomization) and an LLM to decompose tasks into a difficulty curriculum — the model never controls the robot directly, but it shapes how the controller is trained, and the combination cuts the dexterous sim-to-real gap to 8.3%. This generalizes the Text2Touch pattern (LLM-designed rewards) from rewards to the whole training loop. Separately, a KDD 2026 position paper (Liu et al.) argues the broader foundation-model-agent field should treat its own robustness failures as a sim-to-real problem over the four MDP elements and import robotics' domain-randomization toolkit — a sign the conceptual traffic between robotics and LLM-agent research now runs both ways.

The "humanoid foundation model" recipe race (2024–2026)

By 2026 the VLA frontier had moved from two-stage VLM-reasoning + scripted-controller pipelines toward end-to-end vision-language-action policies pre-trained at scale, and several competing recipes appeared within roughly twelve months. They differ less in goal than in where the data comes from:

• Heterogeneous co-training — Physical Intelligence's π₀ → π₀.₅ lineage mixes multi-robot demonstrations, web data, high-level semantic prediction, object detections, and low-level actions in one training run, betting that the diversity itself buys open-world generalization to tasks outside the training distribution. π₀ was pre-trained on 10,000+ hours of robot data; π₀.₅ is the generalization-focused successor.
• Data-scale + ecosystem — NVIDIA's GR00T (N1 → N1.6 → N1.7) pre-trains on ~20K hours of "EgoScale" egocentric human video and wires the result into the Isaac Sim simulation/compute stack, positioning GR00T as the foundation model any OEM lacking in-house AI talent can license — NVIDIA's bid to be the "Android of humanoids."
• Extreme data efficiency — Ψ₀ uses a two-stage recipe (autoregressive VLM pre-training on ~800h of egocentric human video, then a flow-based action expert post-trained on ~30h of real robot data) and reports outperforming baselines trained on more than 10× as much data — the humanoid analog of the passive-video-plus-small-adapter bet (see World Models and V-JEPA 2-AC in the ai KB).

All three are released as open models intended for community extension. The unresolved commercial question is whether recipe innovation (Ψ₀'s data efficiency) beats distribution muscle (NVIDIA's ecosystem) or breadth-of-data (π₀.₅'s co-training) — each reports strong results in different regimes and there is no clear winner yet. The broader frontier framing of this race lives in frontier.md.

July 2026: three more entrants, three more bets

A single-week wave (Jul 1-16, 2026) added three further distinct recipes rather than converging the race onto a winner:

• Data-scale at production scope — Xiaomi-Robotics-1 pushes the "data-scale" bet an order of magnitude further than GR00T's 20K hours: 100k+ hours of real-world UMI-collected trajectories plus a scalable auto-labeling pipeline that turns raw capture into richly language-conditioned training data without manual annotation. The paper's central claim is that this scaling is unsaturated — data and model-size increases keep improving downstream performance, with pre-training gains transferring cleanly into post-training — setting new SOTA on RoboCasa365 (57.4% vs. 46.6% prior) and RoboDojo (20.07 vs. 13.07 prior). It is a first-party corporate research release (Xiaomi, 33 authors) rather than an academic lab, signaling a consumer-electronics company entering the VLA race directly. See Xiaomi.
• Modular function-unification — ACE-Brain-0.5 takes a different architectural bet: rather than one end-to-end policy, it factors "robot intelligence" into five coupled functions (spatial perception, decision-making, embodied interaction, self-monitoring, self-improvement) behind a single 8B backbone, unified via a new SSR+ (Scaffold-Specialize-Reconcile-Reactivate) training recipe that guards against cross-task interference. It improves on 14 of 18 spatial-perception benchmarks over its predecessor ACE-Brain-0, but its own framing calls navigation/manipulation performance merely "competitive" — trading some peak task performance for architectural coherence and an explicit self-improvement-via-external-memory mechanism.
• Architecture over scale — S²-VLA argues close to the opposite of both data-scale bets: a compact 2B-parameter model with a State-Space Guided Adaptive Attention (SSGAA) mechanism — a belief-state tracker over task progression that dynamically gates visual/language/action fusion — beats 7B-scale baselines on long-horizon manipulation (LIBERO, SimplerEnv). This directly answers an open problem already logged in this KB's frontier tracker ("long-horizon combined tasks still 56-63% success") with an architectural fix rather than more data or parameters.

Xiaomi-Robotics-1 and S²-VLA stake out opposite ends of the same open question — does VLA capability require ever-larger data/model scale, or can adaptive-fusion architecture substitute for it? — while ACE-Brain-0.5 stakes a third position (modular unification of discrete functions rather than either scale or fusion architecture). None of the three resolves the recipe race opened by π₀.₅/GR00T/Ψ₀ above; the July 2026 wave added two more distinct bets rather than converging on one. All three models are also now what ROSA's shared-GPU-pool serving argument is about serving at fleet scale — the recipe race (training-time compute) and the serving-economics question (inference-time compute) are two separate, compounding demand curves.

Key Claims

• First framework for zero-shot humanoid loco-manipulation via foundation models — Humanoid-COA integrates GPT-4V reasoning with pre-trained controllers, achieving 96.6% grasping and 90% mobile pick without task-specific training. Evidence: strong (Humanoid-COA: Chain of Action)
• CoA decomposes high-level instructions into executable whole-body behaviors — Chain of Action bridges the gap between language understanding and physical execution through structured task decomposition. Evidence: strong (Humanoid-COA: Chain of Action)
• Learning-based methods are displacing model-based approaches — Three-decade survey confirms the trend toward data-driven rather than physics-engineered robot behaviors. Evidence: strong (Humanoid Locomotion & Manipulation Survey)
• GenAI integration enables the human-looking → human-like transition — Cao (2024) identifies GenAI/LLM integration as the mechanism that unlocks real-time interactive multimodal capabilities previously unattainable. Evidence: strong (Humanoid Robots & Humanoid AI Review)
• VLA modeling is the emerging frontier beyond current two-stage approaches — Vision-language-action models that jointly process perception, language, and action history represent the next step toward end-to-end humanoid control. Evidence: moderate (Humanoid Robots & Humanoid AI Review)
• Only limited humanoids leverage GenAI or LLMs as of 2024 — Despite the capability advantages, most deployed systems still rely on narrow AI or pre-programmed behaviors. Evidence: strong (Humanoid Robots & Humanoid AI Review)
• Foundation models now shape the training pipeline, not just the policy — VLM-as-realism-critic domain randomization plus LLM-decomposed curriculum (DexSim2Real) cut the dexterous sim-to-real gap to 8.3%, extending the Text2Touch "LLM-designed rewards" pattern to the full training loop. Evidence: strong (DexSim2Real)
• Agent robustness is a sim-to-real problem — Foundation-model-agent failures map onto the four MDP gap categories and should adopt robotics' domain-randomization tooling; conceptual traffic between robotics and LLM-agent research now runs both ways. Evidence: moderate (position paper) (Sim-to-Real Gap of Foundation Model Agents)
• Heterogeneous co-training yields open-world VLA generalization — π₀.₅ mixes multi-robot data, web data, semantic prediction, object detections, and low-level actions in one training run, performing broadly on real-world manipulation outside its training distribution; successor to the π₀ flow-matching VLA pre-trained on 10,000+ hours of robot data. Evidence: moderate (π₀.₅)
• Large-scale egocentric human video pre-training powers an OEM-licensable humanoid VLA — NVIDIA GR00T N1.7 pre-trains on ~20K hours of EgoScale human video and integrates with Isaac Sim, positioned as the foundation model OEMs without in-house AI talent can adopt. Evidence: moderate (technical report) (GR00T N1.7)
• A data-efficient two-stage recipe can beat 10× more data — Ψ₀ pre-trains an autoregressive VLM on ~800h of egocentric human video, then post-trains a flow-based action expert on ~30h of real robot data, reportedly outperforming baselines pre-trained on more than 10× the data; an open foundation model for universal humanoid loco-manipulation. Evidence: moderate (Ψ₀)
• VLA scaling has not saturated at 100k+ hours — Xiaomi-Robotics-1 pre-trains on 100k+ hours of UMI-collected real-world trajectories with auto-labeled language conditioning; scaling behavior is consistent and unsaturated across data volume and model size, transferring cleanly from pre-training into post-training; new SOTA on RoboCasa365 (57.4% vs. 46.6%) and RoboDojo (20.07 vs. 13.07). Evidence: moderate (preprint, author-benchmarked; code/checkpoints not yet public) (Xiaomi-Robotics-1)
• A five-function modular architecture unifies robot intelligence behind one backbone — ACE-Brain-0.5 factors spatial perception, decision-making, embodied interaction, self-monitoring, and self-improvement into a single 8B backbone via a new SSR+ training recipe that prevents cross-task interference; improves 14/18 spatial-perception benchmarks over ACE-Brain-0, with self-improvement realized via external memory updates rather than backbone retraining. Evidence: moderate (preprint; navigation/manipulation explicitly framed as "competitive," not best-in-class) (ACE-Brain-0.5)
• Adaptive-fusion architecture can substitute for parameter scale — S²-VLA's State-Space Guided Adaptive Attention maintains a belief state over task progression to dynamically gate visual/language/action fusion; a compact 2B-parameter model beats 7B-scale baselines on long-horizon manipulation (LIBERO, SimplerEnv), diagnosing static feature fusion (not capacity or data) as the specific mechanism behind long-horizon VLA degradation. Evidence: moderate (preprint, simulation-benchmark only — no real-robot deployment reported) (S²-VLA)

Open Questions

• How does dependence on external APIs (GPT-4V) affect reliability and latency for real-time robot control?
• Can foundation models generalize to truly novel environments they were not exposed to during pre-training?
• What happens when the semantic understanding of the scene is incorrect — how do error recovery mechanisms work?
• Is the perception-reasoning-action separation optimal, or will end-to-end VLA models eventually outperform modular approaches?
• How should foundation models encode ethical and social reasoning for the human-level paradigm?
• Which humanoid-VLA recipe wins — heterogeneous co-training (π₀.₅), data-scale + ecosystem (GR00T), data-efficiency (Ψ₀), data-scale-at-production-scope (Xiaomi-Robotics-1), modular unification (ACE-Brain-0.5), or adaptive-fusion architecture (S²-VLA) — or do they specialize by deployment regime?
• Will OEM humanoid makers license a foundation model (GR00T, π₀.₅) or build in-house?
• Does architecture (S²-VLA) or scale (Xiaomi-Robotics-1) win the long-horizon-manipulation failure mode identified in this KB's frontier tracker, or do the two approaches stack?
• Does ACE-Brain-0.5's modular five-function unification trade too much peak task performance for coherence to compete commercially with monolithic end-to-end VLA policies?
• Now that a fourth well-resourced entrant (Xiaomi) has entered on data-scale alone, does the recipe race concentrate around 2-3 dominant approaches, or keep fragmenting?

Related Concepts

• Humanoid Loco-Manipulation — The primary application domain for foundation model-driven robot control
• World Models — Internal predictive models that complement foundation model reasoning
• Humanoid Capability Paradigms — GenAI integration as the enabler for the human-like paradigm
• Sim-to-Real Transfer — Foundation models now act inside the sim-to-real training loop (DexSim2Real); agent robustness reframed as sim-to-real (Liu et al.)
• Embodied-AI Inference Serving & Compute Economics — the models this concept tracks (training-time scaling) are exactly what ROSA's shared-GPU-pool argument is about serving at fleet scale (inference-time scaling) — two separate, compounding compute-demand curves

Related Entities

• Unitree — H1-2 and G1 used as test platforms for Humanoid-COA
• Physical Intelligence — π₀ / π₀.₅ heterogeneous co-training VLA recipe
• NVIDIA — GR00T humanoid foundation model + Isaac Sim ecosystem
• Xiaomi — data-scale-at-production-scope entrant (Xiaomi-Robotics-1); consumer-electronics company entering the VLA race directly

Backlinks

Pages that reference this concept:

• Humanoid Loco-Manipulation
• Humanoid Capability Paradigms
• Unitree
• Xiaomi
• Embodied-AI Inference Serving & Compute Economics

Changelog

• 2026-07-23 — Compiled (xiaomi-robotics-1-vla-scaling, ace-brain-0-5-embodied-foundation-model, s2-vla-state-space-long-horizon-manipulation). Added the "July 2026: three more entrants, three more bets" section, three Key Claims, five Open Questions, the Xiaomi entity, and a cross-link to the new embodied-ai-inference-serving concept.
• 2026-06-24 — Compiled (psi-zero-humanoid-loco-manipulation)
• 2026-06-24 — Compiled (pi-zero-point-five-physical-intelligence)
• 2026-06-24 — Compiled (groot-n1-7-nvidia). Added the "humanoid foundation model recipe race" section (co-training vs data-scale vs data-efficiency), three Key Claims, recipe open questions, and links to the new Physical Intelligence entity + NVIDIA.
• 2026-06-24 — Added the "foundation model as training-pipeline infrastructure" role (DexSim2Real FM-DR + LLM curriculum) and the KDD 2026 agent-MDP bridge paper (agent robustness as sim-to-real). Linked sim-to-real-transfer.
• 2026-04-14 — Added VLA modeling as emerging frontier from Cao (2024). Added GenAI paradigm-transition claims. Linked humanoid-capability-paradigms.
• 2026-04-05 — Initial compilation.