DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation

Abstract

"Sim-to-real transfer remains a critical bottleneck for deploying dexterous manipulation policies learned in simulation to real-world robots." DexSim2Real addresses the reality gap for contact-rich dexterous manipulation by recruiting vision-language foundation models into the sim-to-real pipeline — both as a perceptual critic that tunes simulation realism and as a task planner that structures the learning curriculum.

Key Contributions

The system combines three components:

1. Foundation Model-Guided Domain Randomization (FM-DR) — uses a vision-language model as a visual realism critic to optimize simulation parameters via closed-loop CMA-ES. This augments traditional text-based / hand-specified randomization with a learned signal for "how real does this rendering look."
2. Tactile-Visual Cross-Attention Policy (TVCAP) — adapts cross-attention visuo-tactile fusion to zero-shot sim-to-real reinforcement learning, so the policy reasons jointly over vision and touch.
3. Progressive Skill Curriculum (PSC) — builds on LLM-based task decomposition with a difficulty scheduler tailored to contact-rich dexterous tasks.

Results

• Real-world success rate: 78.2% average across six challenging manipulation tasks (blinded evaluation).
• Sim-to-real performance gap reduced to 8.3% — the drop between simulation and real-world success.
• Outperforms DrEureka and DeXtreme — two strong prior sim-to-real / dexterous-RL baselines.

Limitations

• The paper does not explicitly enumerate limitations in the indexed content. Open from context: dependence on the quality of the VLM critic's realism judgments; the six-task evaluation set bounds the generalization claim; contact-rich tasks beyond the tested primitives remain unproven.

Significance

DexSim2Real is a concrete instance of a 2026 pattern: foundation models are no longer just the policy in robot learning — they are increasingly the infrastructure around the policy (realism critics, curriculum designers, reward authors). It directly extends the tactile-sensing line (Text2Touch's LLM-designed rewards) and the sim-to-real line (ABB/NVIDIA's 99% positioning correlation) by attacking the harder dexterous-manipulation case, where the ABB result was explicitly an open question.

Source: arXiv 2605.05241, submitted May 3, 2026.