X2Real's 0.84 correlation covers eight rigid tasks and one robot policy

X Square reports a 0.84 simulation to real correlation, but the test covers eight rigid tasks and one policy. A policy that works 70% on hardware scored zero in simulation.

X Square Robot, a Beijing startup, says its new X2Real benchmark predicts real robot results with a 0.84 correlation. The figure covers a narrow test. X2Real cannot yet decide which policy goes onto a robot for precision work.

A simulator that predicts hardware behaviour is worth a great deal to anyone who pays for robot time. Hardware is slow, breaks, and needs a technician in the room. X2Real can rank policies on familiar rigid tasks, where objects do not bend. A policy is the software that controls the robot.

The 0.84 is a correlation of progress scores, the scores for how far each task gets. It covers eight tasks and one policy. It shows how well that policy's scores match between two worlds, and says little about other policies.

The test has two more limits. The success rate correlation is lower, at 0.74. The out of distribution half of the benchmark, meaning tasks that differ from the training ones, is not public.

X2Real is described in an arXiv paper first submitted on 23 September and revised on 13 October. It is built on NVIDIA's Isaac Lab-Arena, a simulation toolkit. It spans 44 long horizon tasks across 10 capability dimensions, with nearly 300 hours of annotated simulation trajectories.

The code followed the paper. The project's GitHub page dates the open source release of version 0.9.0 to 10 October.

The 0.84 comes from eight tasks in one scene. In Section 4.1 of the paper, the correlation is computed over a green booth taken from the ManipArena benchmark. Five tasks are general and three demand precision. Every object is rigid.

Only one policy was tested on both hardware and simulation. It is Wall-OSS-0.5, and it was fine tuned on real demonstrations only. No simulation data went into its training or fine tuning.

The company's wording is broader than that test. Its post on X on Monday says the 0.84 holds "on policies trained only with real-world data". The post adds that policies are tested across three out of distribution axes: environment appearance, task settings and robot embodiment.

The public evidence covers one model, not a set of policies. The benchmark page says "a model", and the paper's hardware comparison covers the one.

The task split matters. Across all eight, the progress score correlation is 0.84 and the success rate correlation is 0.74. For the five general tasks alone, the figures rise to 0.99 and 0.96. The shortfall therefore comes entirely from the precision tasks.

The glasses task exposes the gap. A policy placing glasses on a wooden shelf succeeded 70 percent of the time on the real robot and 0 percent in simulation. That is a 70 point miss.

The paper explains why. A failed placement can fold the glasses' legs into a shape the policy never saw in training. Contact rich placement, where the robot must touch and press things, is where simulators are least forgiving. The error runs one way because simulation punishes the policy more than the real world does.

A simulator need not match hardware task by task. It only needs to put policies in the right order. On general tasks the 0.96 and 0.99 figures support that.

But the correlation was measured on one policy. It does not show that the order of different policies survives the move to hardware. The paper's Table 1 ranks four VLA policies in simulation only, with no hardware results for them. A VLA is a vision language action model, which turns camera images and instructions into robot moves.

The ranking separates the policies sharply. On success rate in distribution, wall-oss-preview leads at 53.90 percent. Then come π0.5 at 26.12 percent, DreamZero at 4.85 percent and Wall-OSS-0.5 at 4.08 percent.

Out of distribution, the same order holds at 34.26, 16.13, 3.45 and 3.20 percent. The strongest policy keeps 63.6 percent of its score when the tasks change, and π0.5 keeps 61.8 percent. DreamZero keeps 71.1 percent.

The table separates a 53.90 percent policy from a 4.08 percent one on familiar tasks. It keeps the same order when the tasks change. It has not shown that the order would appear on a real bench. No hardware run exists for any of the four.

Wall-OSS-0.5 sits at the bottom of the table, although the paper chose it as the base for sim to real work. The reason is that it performed best on ManipArena. All four policies were post trained, meaning trained further, on the same ArtiXon Arm-6A demonstrations for about one epoch.

The low scorers also retain more of their score out of distribution, 78.4 percent for Wall-OSS-0.5. That is because there is little left to lose.

The open release limits what outsiders can check. Version 0.9.0 contains the in distribution evaluation runtime, with five boards and 73 runnable configurations. They span four robots: the ArtiXon Arm-6A, Quanta X1, ARX R5 and Franka Panda. Those 73 configurations are cases the evaluator can run, a different unit from the 44 tasks in the paper.

Reproducing the benchmark requires more than a download. Simulation assets ship as a manifest and are downloaded separately from Hugging Face, the model hosting site. The dataset is separate too. The code needs a Linux machine with an NVIDIA GPU that supports Isaac Sim 6.0.1.0.

The bundled template policy returns zero actions and only validates the interface. The repository's public scope note lists the out of distribution definitions, training and automatic planner data collection as excluded.

The 73 configuration runtime can screen general manipulation policies cheaply, trading GPU hours for robot hours. The 0.96 correlation on general tasks supports that use. Precision and contact tasks belong on hardware. A simulator that scores a working policy at zero will reject the candidate most worth testing.

The withheld out of distribution result would make X2Real a rigorous test of generalisation. Outside labs cannot rebuild it from the repository because the definitions are excluded.

The benchmark page does publish out of distribution scores for the four tabletop policies. Two of them come from other organisations, Physical Intelligence and NVIDIA. The developer platform also has a model submission route behind a sign in. So the ranking can be consulted but not audited.

A second policy's hardware run would show whether the correlation belongs to the simulator or to Wall-OSS-0.5. The paper has already been revised once in three weeks, on 23 September and 13 October. If no version published by 31 December 2026 carries a second policy's real world results, the 0.84 stays a one policy number.