HumanoidVLN — Physics-Grounded VLN Benchmark for Humanoid Robots

Overview

Why humanoid VLN needs its own benchmark

Existing VLN benchmarks move robots by kinematic teleportation, place them in scenes bipeds can't traverse, and pair videos with hallucination-prone automated instructions. HumanoidVLN closes all three gaps at once.

Vision-language navigation (VLN) has largely been studied on abstracted mobile agents, leaving a wide sim-to-real gap for humanoid robots, whose gait stability, joint limits, and center-of-mass dynamics fundamentally change what a feasible trajectory is. HumanoidVLN is a physics-grounded simulator and benchmark built on NVIDIA Isaac Sim that evaluates VLN under authentic bipedal embodiment.

The platform supports four humanoid configurations spanning 10–12 lower-body DoF and 1.20–1.80 m in height, each driven by a hierarchical control stack: an RL policy generates stable low-level gaits while PD or MPC trackers follow global plans. Scenes are curated from artist-designed environments, 3D Gaussian Splatting reconstructions, and robot-captured image streams — all with more than 100 m² of navigable area. Navigation instructions are produced by a Dual Generator-Reviewer + Paraphraser multi-agent system with human-in-the-loop verification, yielding annotations that are scalable, linguistically diverse, and physically consistent with the executed trajectories.

Gap 01 · Simulation

Teleportation isn't locomotion

Habitat-style simulators slide robots through space kinematically, bypassing every physical constraint that makes bipedal walking hard. No existing VLN simulator models humanoid embodiment diversity.

Our answer: an Isaac Sim platform where an RL policy commands joint torques and PD/MPC trackers follow the plan — every trajectory is physically executed by the specific morphology under test.

Gap 02 · Scenes

Most scenes can't fit a biped

Benchmarks scale scene count, not navigability. Narrow doorways and dense furniture create bottlenecks that a wide-stepping, upright humanoid simply cannot pass.

Our answer: scenes curated for >100 m² of traversable floor from GRScenes and SAGE-3D, plus a 3DGS pipeline that reconstructs scenes directly from a walking robot's own camera.

Gap 03 · Instructions

Single-pass VLMs hallucinate space

Automated instruction pipelines confuse left with right and misalign time; manual annotation is reliable but doesn't scale. And idealized renders miss the camera shake of a real walking robot.

Our answer: egocentric video recorded from the walking simulator, described by a Generator–Reviewer–Paraphraser agent loop, then verified by human annotators.

The Platform

One simulator, four humanoid bodies

Every embodiment loads interchangeably into identical environments, so navigation performance can be compared across morphologies under controlled conditions.

Overview diagram of the humanoid VLN simulation platform: a multi-humanoid library, hierarchical control framework, Isaac Sim environment, dataset generation, and VLN evaluation interface. — **Platform overview.** Overview of the HumanoidVLN physics-grounded benchmark pipeline, from embodiment-diverse humanoid platforms and hierarchical control to scene curation, episode construction, multimodal dataset generation, and plug-and-play VLN evaluation. *Click to enlarge.*

1.32 m

Unitree G1

12 lower-body DoF · 1.32 m

A compact 12-DoF platform. Shorter limbs shift the center of mass lower, changing step reachability and the gait the RL policy converges to — exactly the kind of embodiment variation the benchmark isolates.

DoF 12 Height 1.32 m Control RL + PD/MPC

High level · Global

PD / MPC path tracker

Converts the global navigation plan into velocity and heading references. Both tracker variants are interchangeable, so tracker fidelity itself becomes an experimental variable.

Low level · Local

RL locomotion policy

Commands joint torques directly, producing stable bipedal gaits that respect per-morphology joint limits and center-of-mass dynamics.

Output · Physical

Stable bipedal walking

Trajectory following subject to real contact dynamics — with the camera shake and lighting variation of an actual walking robot recorded into the dataset.

Scene Suite

Environments curated for bipedal traversal

Two complementary scene sources, one admission criterion: a humanoid must actually be able to walk there.

RGB render

Normal map

A · Artist-designed

GRScenes selections

Artist-created scenes offer perfect geometry but non-photorealistic renders at the cost of labor-intensive material and 3D model authoring.

3D Gaussian Splatting reconstruction of the same scene

RGB capture

GS reconstruction

GS · Gaussian Splatting

GS2Sim: our custom 3DGS pipeline built on gsplat

Our custom 3DGS pipeline reconstructs simulation-ready scenes directly from multi-view captures, producing photorealistic renders with comparable geometry.

Navigability criterion. Every scene admitted to the benchmark exceeds 100 m² of traversable floor area, verified against the dilated occupancy footprint of the tallest supported humanoid.

Floor area per scene

Artist-designed GS ≥100 m²

Scenes, sorted by floor area (m²) · hover a bar for details

n = 87 · all ≥100 m² · min / med / mean 100 / 266 / 387 m²

Scenes by indoor class

Residential Food & retail Culture & leisure Workplace & education Healthcare Fitness

17 indoor classes · 6 application domains

Instruction Generation

Agents draft, agents critique, humans verify

The Dual Generator-Reviewer + Paraphraser multi-agent system replaces single-pass VLM generation with an iterative loop of drafting, spatial verification against the scene graph, and stylistic synthesis — then a human takes the final pass.

Pipeline diagram: egocentric navigation video and 2D occupancy map feed a Dual Generator-Reviewer plus Paraphraser multi-agent system, producing structured JSON with fine-grained and coarse-grained instructions. — **Generation pipeline.** Egocentric video and a 2D occupancy map enter the MAA; structured JSON with fine- and coarse-grained instructions comes out. The Paraphraser is constrained to preserve action order and landmark order. *Click to enlarge.*

Generator (Describer)

Visual-linguistic bridge · Chain-of-Thought

Takes the chronological egocentric frames and spatial prior, performs route understanding (turn and transition detection) and landmark mining, and drafts a step-by-step instruction in second-person imperative voice.

Generator · Draft 1"Go straight down the hallway, turn left at the painting, then pass the sofa and stop at the bookshelf."

Reviewer (Verifier)

Logic-driven critic · scene-graph grounded

Cross-checks the draft against a rigid spatial-semantic scene graph from simulator metadata and the observed trajectory data. Physically implausible relations trigger a correction, and the loop repeats until the text is spatially and temporally consistent.

Reviewer · CritiqueTrajectory heading changes by −87° at the painting: that left turn is actually a right turn. Revise.

Generator · Draft 2 ✓ verified"Go straight down the hallway, turn right at the painting, then pass the sofa and stop at the bookshelf."

Paraphraser (Synthesizer)

Stylistic adaptation · order-preserving

Rewrites the verified instruction into diverse linguistic styles — formal, natural, casual — without altering the grounding facts. Action order and landmark order are hard constraints.

Paraphraser · Casual"Head down the hall, hang a right at the painting — past the couch, bookshelf's right there."

Human annotator

Final verification pass

Human annotators correct residual errors and flag edge cases — the scalability of automation with the reliability of manual annotation, without its full cost.

Annotator · ApprovedEpisode 0413 ✓ — instruction grounded, landmarks consistent, endpoint condition explicit. Packaged to JSON.

Detailed step-by-step guidance: explicit turns, passed landmarks, and side-of-path locations.

fine · verified "Go straight down the hallway, turn right at the painting on your left, pass the sofa on your right, and stop in front of the bookshelf."

Each episode ships as structured JSON: landmarks[] route_summary fine coarse

The Benchmark

Where HumanoidVLN sits in the landscape

No existing benchmark jointly delivers humanoid embodiment diversity, navigability-curated scenes, and verified instruction generation.

Dataset	Simulator	Humanoid	DoF range	Scene source	Instruction	Action space
R2R (2018)	Matterport3D	—	—	A	Human	Graph
RxR (2020)	Matterport3D	—	—	A	Human	Graph
REVERIE (2020)	Matterport3D	—	—	A	Human	Graph
VLN-CE (2020)	Habitat	—	—	A	Human	Discrete
ALFRED (2020)	AI2-THOR	—	—	A	Human	Discrete
AerialVLN (2023)	AirSim	—	—	A	Human	Discrete
LH-VLN (2025)	Habitat	—	—	A	VLM	Graph
GSA-R2R (2025)	Habitat	—	—	GS	VLM	Graph
VLN-PE (2025)	Isaac	—	—	A	VLM	Hybrid
VLNVerse (2025)	Isaac	—	—	A	VLM	Hybrid
HumanoidVLN (Ours, 2026)	Isaac (Humanoid)	✓	10–12	A + GS	MAA + Human	Hybrid

Scene source — A: artist-designed · GS: 3D Gaussian Splatting reconstruction Instruction — Human: manual annotation · VLM: automated generation · MAA + Human: Dual Generator-Reviewer + Paraphraser multi-agent annotation with human-in-the-loop verification.

Evaluation protocol

Success Rate

Share of episodes where the agent stops within a threshold distance of the target.

Navigation Error

Average distance between the agent's stopping position and the goal.

SPL

Success weighted by Path Length

Trajectory efficiency — success discounted by how far the agent wandered.

nDTW

Normalized Dynamic Time Warping

Path fidelity — how closely the agent adhered to the instructed route.

Oracle Success

Measures a robot's maximum capability to reach a destination, assuming it makes the perfect decision about when to stop.

Fall Rate

The fraction of steps in which the robot is detected to have fallen, measuring locomotion stability under physical execution.

Plug-and-play VLN models: NAVILA DualVLN StreamVLN JanusVLN