HumanoidVLN — Physics-Grounded VLN Benchmark for Humanoid Robots

Overview

Why humanoid VLN needs its own benchmark

Existing VLN benchmarks move robots by kinematic teleportation, place them in scenes bipeds can't traverse, and pair videos with hallucination-prone automated instructions. HumanoidVLN closes all three gaps at once.

Vision-Language Navigation (VLN) for humanoid robots presents unique challenges that existing benchmarks fail to address: bipedal locomotion imposes physical constraints absent from wheeled or idealized agents, humanoid morphologies vary significantly across platforms, and egocentric observations are distorted by locomotion-induced camera dynamics. We present HumanoidVLN, a physics-grounded simulator and benchmark for VLN across diverse humanoid embodiments. Built on NVIDIA Isaac Sim, our platform supports an extensible set of humanoid configurations — demonstrated on four robots (Unitree G1, Unitree H1, Motion2, MiniMotion) spanning 10–12 lower-body DoF and heights from 1.17 m to 1.80 m — via a hierarchical control stack combining a Reinforcement Learning locomotion policy with interchangeable PD or MPC path trackers. New robots and VLN models can be integrated with minimal effort, and we demonstrate compatibility with NaVILA, DualVLN, StreamVLN, and JanusVLN. Environments are drawn from artist-designed scenes and 3D Gaussian Splatting reconstructions, filtered for navigability areas exceeding 100 m². Navigation instructions are generated by a Dual Generator-Reviewer + Paraphraser Multi-Agent Annotation with human-in-the-loop verification, yielding 933 collision-aware reference episodes, each paired with one spatially grounded fine-grained instruction and three coarse-grained stylistic variants (Formal, Natural, Casual). Across four models and four humanoid embodiments, JanusVLN achieves the highest mean SR of 43.55% and nDTW of 48.38. In a 20-episode sim–real pilot with DualVLN and Unitree G1, navigation errors are strongly correlated (r = 0.935), with a mean absolute difference of 0.68m and mean trajectory similarity of 0.782 ± 0.188 nDTW. These results highlight the interaction between VLN models, controllers, and humanoid embodiments under physical execution. Code, benchmark, and data will be released upon acceptance.

Gap 01 · Simulation

Teleportation isn't locomotion

Habitat-style simulators slide robots through space kinematically, bypassing every physical constraint that makes bipedal walking hard. No existing VLN simulator models humanoid embodiment diversity.

Our answer: an Isaac Sim platform where an RL policy commands joint torques and PD/MPC trackers follow the plan — every trajectory is physically executed by the specific morphology under test.

Gap 02 · Scenes

Most scenes can't fit a biped

Benchmarks scale scene count, not navigability. Narrow doorways and dense furniture create bottlenecks that a wide-stepping, upright humanoid simply cannot pass.

Our answer: scenes curated for >100 m² of traversable floor from GRScenes and SAGE-3D, plus a 3DGS pipeline that reconstructs scenes directly from a walking robot's own camera.

Gap 03 · Instructions

Single-pass VLMs hallucinate space

Automated instruction pipelines confuse left with right and misalign time; manual annotation is reliable but doesn't scale. And idealized renders miss the camera shake of a real walking robot.

Our answer: egocentric video recorded from the walking simulator, described by a Generator–Reviewer–Paraphraser agent loop, then verified by human annotators.

The Platform

One simulator, four humanoid bodies

Every embodiment loads interchangeably into identical environments, so navigation performance can be compared across morphologies under controlled conditions.

Overview diagram of the humanoid VLN simulation platform: a multi-humanoid library, hierarchical control framework, Isaac Sim environment, dataset generation, and VLN evaluation interface. — **Platform overview.** Overview of the HumanoidVLN physics-grounded benchmark pipeline, from embodiment-diverse humanoid platforms and hierarchical control to scene curation, episode construction, multimodal dataset generation, and plug-and-play VLN evaluation. *Click to enlarge.*

1.32 m

Unitree G1

12 lower-body DoF · 1.32 m

A compact 12-DoF platform. Shorter limbs shift the center of mass lower, changing step reachability and the gait the RL policy converges to — exactly the kind of embodiment variation the benchmark isolates.

DoF 12 Height 1.32 m Control RL + PD/MPC

High level · Global

PD / MPC path tracker

Converts the global navigation plan into velocity and heading references. MPC is used for continuous-action DualVLN; PD is used for discrete-action NaVILA, StreamVLN, and JanusVLN.

Low level · Local

RL locomotion policy

Commands joint torques directly, producing stable bipedal gaits that respect per-morphology joint limits and center-of-mass dynamics.

Output · Physical

Stable bipedal walking

Trajectory following subject to real contact dynamics — with the camera shake and lighting variation of an actual walking robot recorded into the dataset.

Scene Suite

Environments curated for bipedal traversal

Two complementary scene sources, one admission criterion: a humanoid must actually be able to walk there.

RGB render

Normal map

A · Artist-designed

GRScenes selections

Artist-created scenes offer perfect geometry but non-photorealistic renders at the cost of labor-intensive material and 3D model authoring.

3D Gaussian Splatting reconstruction of the same scene

RGB capture

GS reconstruction

GS · Gaussian Splatting

GS2Sim: our custom 3DGS pipeline built on gsplat

Our custom 3DGS pipeline reconstructs simulation-ready scenes directly from multi-view captures, producing photorealistic renders with comparable geometry.

Navigability criterion. Every scene admitted to the benchmark exceeds 100 m² of traversable floor area, verified against the dilated occupancy footprint of the tallest supported humanoid.

Floor area per scene

Artist-designed GS ≥100 m²

Scenes, sorted by floor area (m²) · hover a bar for details

n = 87 · all ≥100 m² · min / med / mean 100 / 266 / 387 m²

Scenes by indoor class

Residential Food & retail Culture & leisure Workplace & education Healthcare Fitness

17 indoor classes · 6 application domains

Instruction Generation

Agents draft, agents critique, humans verify

The Dual Generator-Reviewer + Paraphraser multi-agent system replaces single-pass VLM generation with an iterative loop of drafting, spatial verification against the scene graph, and stylistic synthesis — then a human takes the final pass.

Pipeline diagram: egocentric navigation video and 2D occupancy map feed a Dual Generator-Reviewer plus Paraphraser multi-agent system, producing structured JSON with fine-grained and coarse-grained instructions. — **Generation pipeline.** Egocentric video and a 2D occupancy map enter the MAA; structured JSON with fine- and coarse-grained instructions comes out. The Paraphraser is constrained to preserve action order and landmark order. *Click to enlarge.*

The interactive conversation and instruction variants below are illustrative examples, not quoted dataset annotations.

Generator (Describer)

Visual-linguistic bridge · Chain-of-Thought

Gemma-4-31B-it and InternVL3.5-38B independently infer structured route graphs from egocentric keyframes alone, sharing a goal condition grounded by Qwen3-VL-30B-A3B. The graphs encode actions, landmarks, side-of-path relations, ordinals, and turn magnitudes.

Generator · Draft 1"Go straight down the hallway, turn left at the painting, then pass the sofa and stop at the bookshelf."

Reviewer (Verifier)

Logic-driven critic · scene-graph grounded

Reconciles the two route graphs and verifies contradictions using Qwen3-VL-30B-A3B, trajectory metadata, the A* route, and a spatial–semantic scene graph restricted to visible objects. Deterministic geometry checks validate turns, order, landmark identity, and spatial relations; the VLM checks completeness and ambiguity.

Reviewer · CritiqueTrajectory heading changes by −87° at the painting: that left turn is actually a right turn. Revise.

Generator · Draft 2 ✓ verified"Go straight down the hallway, turn right at the painting, then pass the sofa and stop at the bookshelf."

Paraphraser (Synthesizer)

Stylistic adaptation · order-preserving

GPT-5.5 receives only the verified text and produces Formal, Natural, and Casual variants at T = 1.0. Each variant is re-parsed and accepted only when its navigation-critical attributes match the verified route.

Paraphraser · Casual"Head down the hall, hang a right at the painting — past the couch, bookshelf's right there."

Human annotator

Final verification pass

Three trained annotators review every generated instruction, correct residual errors, and flag ambiguous or infeasible cases for revision.

Annotator · ApprovedIllustrative episode ✓ — instruction grounded, landmarks consistent, endpoint condition explicit. Packaged to JSON.

Detailed step-by-step guidance: explicit turns, passed landmarks, and side-of-path locations.

fine · verified "Go straight down the hallway, turn right at the painting on your left, pass the sofa on your right, and stop in front of the bookshelf."

Each episode ships as structured JSON: landmarks[] route_summary fine coarse

The Benchmark

Where HumanoidVLN sits in the landscape

No existing benchmark jointly delivers humanoid embodiment diversity, navigability-curated scenes, and verified instruction generation.

Dataset	Simulator	Humanoid	DoF range	Scene source	Instruction	Action space
R2R (2018)	Matterport3D	—	—	A	Human	Graph
RxR (2020)	Matterport3D	—	—	A	Human	Graph
REVERIE (2020)	Matterport3D	—	—	A	Human	Graph
VLN-CE (2020)	Habitat	—	—	A	Human	Discrete
ALFRED (2020)	AI2-THOR	—	—	A	Human	Discrete
AerialVLN (2023)	AirSim	—	—	A	Human	Discrete
LH-VLN (2025)	Habitat	—	—	A	VLM	Graph
GSA-R2R (2025)	Habitat	—	—	GS	VLM	Graph
VLN-PE (2025)	Isaac	—	—	A	VLM	Hybrid
VLNVerse (2025)	Isaac	—	—	A	VLM	Hybrid
HumanoidVLN (Ours, 2026)	Isaac (Humanoid)	✓	10–12	A + GS	MAA + Human	Hybrid

Scene source — A: artist-designed · GS: 3D Gaussian Splatting reconstruction Instruction — Human: manual annotation · VLM: automated generation · MAA + Human: Dual Generator-Reviewer + Paraphraser multi-agent annotation with human-in-the-loop verification.

Zero-shot cross-embodiment results

All four public checkpoints are evaluated without training or fine-tuning on the same 933 episodes. One instruction per episode is assigned in a fixed, approximately balanced distribution across Fine, Formal, Natural, and Casual styles, shared by every model–robot configuration.

Results on 933 episodes per configuration. SR, OS, SPL, nDTW and FR are reported on a 0–100 scale; NE is in meters.
Model	Robot	SR ↑	OS ↑	NE ↓	SPL ↑	nDTW ↑	FR ↓
NaVILA	Unitree G1	28.19	41.80	5.86	22.24	43.65	7.93
NaVILA	Unitree H1	21.97	37.41	6.14	17.12	41.97	70.95
NaVILA	Motion2	31.73	56.27	5.54	15.81	31.99	2.68
NaVILA	MiniMotion	26.80	42.77	5.94	21.00	37.93	6.97
StreamVLN	Unitree G1	26.80	34.30	6.35	19.50	41.38	9.54
StreamVLN	Unitree H1	14.36	20.58	7.32	11.17	35.45	64.52
StreamVLN	Motion2	31.83	40.41	6.07	19.51	39.72	2.68
StreamVLN	MiniMotion	21.54	29.80	7.13	11.63	30.15	7.93
DualVLN	Unitree G1	35.05	42.55	5.73	25.83	46.67	5.47
DualVLN	Unitree H1	17.68	22.19	6.80	14.70	39.17	27.12
DualVLN	Motion2	32.69	41.37	5.70	21.86	43.27	3.64
DualVLN	MiniMotion	30.23	40.51	5.93	23.29	44.44	7.07
JanusVLN	Unitree G1	44.59	55.31	4.87	33.18	50.08	7.40
JanusVLN	Unitree H1	29.37	38.48	5.80	23.24	44.93	30.76
JanusVLN	Motion2	50.38	63.34	4.26	33.56	50.51	2.79
JanusVLN	MiniMotion	49.84	64.31	4.54	36.18	48.01	9.97

JanusVLN achieves the highest mean SR (43.55%) and nDTW (48.38). Motion2 has the highest average SR (36.66%); G1 has the highest average SPL (25.19). H1 fall rates range from 27.12% to 70.95%, highlighting the interaction between models, controllers, and embodiments.

Sim-to-real pilot

DualVLN on Unitree G1 was evaluated on 20 paired episodes in a pantry and a studio room (10 per scene). Simulation and real-world navigation errors correlate strongly: Pearson r = 0.935 and Spearman ρ = 0.911. The mean absolute NE difference is 0.68 m; paired trajectory similarity is 0.782 ± 0.188 nDTW on a 0–1 scale. This two-scene, one-checkpoint pilot provides initial episode-level evidence, without establishing generalization across scenes.

Evaluation protocol

Success Rate

Share of episodes where the agent stops within 3.0 m of the goal.

Navigation Error

Average distance between the agent's stopping position and the goal.

SPL

Success weighted by Path Length

Trajectory efficiency — success discounted by how far the agent wandered.

nDTW

Normalized Dynamic Time Warping

Path fidelity — how closely the agent adhered to the instructed route.

Oracle Success

Measures a robot's maximum capability to reach a destination, assuming it makes the perfect decision about when to stop.

Fall Rate

Percentage of episodes with at least one detected fall. Detection uses embodiment-relative base-height drops and downward speed or sustained collapse; a fall ends the episode without automatic recovery.

Plug-and-play VLN models: NaVILA DualVLN StreamVLN JanusVLN

HumanoidVLN: a physics-grounded simulator & benchmark for vision-language navigation across diverse humanoid embodiments