1X World Model

Redwood AI: Mobility

Redwood AI

Artificial Intelligence

Now Hiring

Capable of navigation to objects of interest and coordinate precise placement of the body in order to maximize chance of successful grasps and execute chores that require movement space.

Mobile Maniuplation

Redwood is training on both successes and failures, allowing NEO to learn from every interaction — and gain valuable context from failure data

Learns from Mistakes

Redwood is among the first VLAs to control locomotion jointly with manipulation, enabling bracing and leaning behaviors during manipulation

Whole Body Control

Redwood can handle variation in tasks, such as picking up never-before seen objects that appear in unique locations

Generalization

Integrated with an off-board language that predicts user intent from voice and conversation context in real time—allowing users to simply talk to NEO and ask it to take action

Voice Control

Default

Having complete and unfettered access to all parts of the world is important for NEO to accomplish tasks in home environments, and to enable our Redwood AI to learn from the broadest set of physical interactions possible. Our latest RL controller provides Redwood with a complete mobility toolkit to access the world, including natural walking in any direction, sitting, standing, kneeling, getting up from the floor, going up and down stairs with stereo vision. Bringing all of these capabilities together for the first time in a unified controller represents an important milestone in unlocking the full potential of humanoid robots.

SIT & STAND

Full stairs

Bridging Natural Movement with Omnidirectional steering

The past decade has seen rapid progress in legged robot locomotion, primarily driven by torque-transparent actuators, deep Reinforcement Learning (RL) algorithms, and GPU-accelerated simulation. Using off-the-shelf software packages and a desktop GPU, it is now possible to train a bipedal robot to stand upright in simulation and follow walking direction commands in under an hour. Even though the policy is trained completely in a simulation environment whose physics are merely an approximation of the real world, it is trained on so many randomized physics parameters – e.g. friction, mass, sensor noise – that the model ends up being robust to the real world’s physics parameters. Once trained, this walking controller can receive walking direction commands from either the teleoperator or the AI model (e.g. Redwood), and then translate those high level directions into the dynamic, contact-aware interactions with the world.



Beyond walking and turning, bipeds can also perform side-stepping. This is useful for navigating the close quarters of a kitchen or the space between the sofa and the coffee table, where the footprint would be too small for a wheeled base robot.



However, these basic walking RL controllers often require additional “shaping rewards” to achieve a natural human-like gait in all directions. These tend to be highly specialized to walking, which means that the same process of hand-tuning rewards must be repeated for every new behavior. Gait patterns can vary based on the direction of walking, so this often requires unique shaping terms based on the direction of motion.

Is there a more scalable way to increase controller capabilities without hand-written shaping rewards for each of them? One method is to collect motion capture references from humans moving naturally, retarget them to NEO’s joints and body, and then train the RL controller to match those kinematic reference trajectories.

Because the reference only specifies where the body should be, the RL controller still needs to figure out how to keep the robot stable, while “keeping tempo” and tracking the reference trajectory as closely as possible.

Using these techniques, it is possible to over-fit a policy to track a single human motion capture trajectory and achieve very dynamic and fluid motions such as dancing or walking. Examples of “single trajectory replay controller” are shown below for natural running and pivoting:

Pivot



These behaviors, while elegant, are not readily useful for general purpose tasks as they only can replay a single trajectory. They do not expose a steerable interface with which a high level policy like the AI model can execute the right actions. It is also not obvious how to transition smoothly from one reference to another, as motion capture datasets rarely include the transition behaviors between arbitrary task-centric motions, e.g. switching from shuffling quickly side-to-side to a skipping motion.

To handle multiple trajectories, we could train the controller to follow multiple mocap references, taking in the encoded kinematic trajectory as an input. However, this approach encounters a teleoperation UX problem: it is not obvious how one provides the high-dimensional kinematic trajectories at test-time using a more limited input device like a gamepad joystick or a VR controller. The model is trained on high-dimensional kinematic trajectories with nuanced rhythm and periodicity, but the commands provided by teleoperation are coarse-grained, which results in the RL controller interpreting it into an unnatural gait.

How does one achieve steerability and robustness, while still achieving the fluidity of motion capture-based RL training? We developed a two-stage controller, consisting of a high level “kinematic planner” to synthesize kinematic targets that resemble human motion capture data, and a low level controller that tries to achieve those plans.



The lower level RL controller takes as input a kinematic reference trajectory of body poses that it must attempt to track while maintaining balance. This is paired with a high level motion generator model that is trained with supervised learning to convert input commands like joystick direction to the richer kinematic trajectories. The generative model also plays the role of smoothing transitions during behavior changes.

An important reason for having robots with legs in the home is to traverse stairs. We develop a “stair mode” in our controller which engages the use of stereo RGB vision to infer the height of the floor around NEO.

To climb up and down stairs in a graceful way, NEO’s RL controller must anticipate the necessary height of each step well in advance of making contact with that step. Unlike most humanoid robots, which employ a time-of-flight depth sensor or a lidar to estimate the floor plane, NEO’s RL controller is purely vision based. Depth is predicted directly from the RGB stereo pair, and this is fused with the proprioceptive history for NEO to figure out how and where to step.



Stairs are not always idealized; through domain randomization in simulation, the controller is also robust enough to support side-stepping and handling stairs of mixed heights.

There are numerous home chores that require NEO to work at floor height for extended periods of time: removing a stain from the carpet, reorganizing the bottom drawer of a cabinet, packing a suitcase, and sorting socks. We extend our RL controller to be able to safely sit, kneel and lie down on the floor, as well as get back up from each of these poses.

Kneel

Sit & stand

The controller provides an “action interface” for which teleoperation or Redwood AI is able to interact in a safe, contact-rich manner with the physical world. To demonstrate the controllability of the natural walking behavior, we fine-tuned the Redwood model to do a soccer ball dribbling task.

Here is Redwood interacting with this new RL controller, where it predicts whole body joint targets and walking pelvis velocities from vision. The controller then translates those intents into the specific forces applied by the leg to walk in the direction of the ball.

We have developed the first general-purpose, fully AI and teleoperation compatible controller that unlocks the full kinematic workspace that is available to a bipedal humanoid robot. This will enable us to train Redwood AI to fully explore the entire state space of the home: every high and low shelf, every nook and cranny, every floor.

We will then use that data to make an AI like no one has ever seen.

Redwood AI’s latest Reinforcement Learning (RL) controller gives NEO a complete mobility toolkit—enabling natural walking in any direction, sitting, standing, kneeling and stair climbing using stereo vision.

Walk

Stair

Shuffle

Home robots need common sense behavior and a deep understanding of the physical world.

Many robot foundation models today are vision-language-action models (VLAs), which take a pretrained VLM and add an output head to predict robot actions (

). VLMs benefit from internet-scale knowledge, but are trained on objectives that emphasize visual and semantic understanding over prediction of physical dynamics. Tens of thousands of hours of costly robot data are needed to teach a model how to solve tasks considered simple for a human. Additionally, auxiliary objectives are often used to further coax spatial reasoning of physical interactions (

In this blog, we introduce our video-pretrained world model, 1XWM, integrated into NEO as a robot policy. While VLAs directly predict action trajectories from static image-language input, our world-model based policy derives robot actions from text-conditioned video generation. By leveraging the world dynamics inherent in internet-scale video, our world model generalizes to novel objects, motions, and tasks without pre-training on large-scale robot data or any related teleoperated demonstrations. This represents a shift towards a new regime: allowing robot intelligence to benefit from the scaling of video pretraining, enabled by a hardware stack designed for high fidelity transfer from human embodiment.

Internet video implicitly encodes the structural priors of reality: how people and objects move, where forces are applied, and the implicit constraints of interaction. Accurate translation of video into action comes from more than just a model—it benefits from an embodiment that is kinematically and dynamically congruent with the human form.

The hardware as a first-class citizen in the AI stack closes the human-robot translation gap. By combining embodiment with human-like compliance, interaction dynamics—friction, inertia, and contact behavior—often match human motion closely enough for the model’s learned priors to remain in distribution. What the model can visualize, NEO usually can do.

Today, frontier text-to-video models like 

 generate strikingly realistic videos. However, zero-shot generations from these models are not grounded in robot embodiments and often fall short along key axes required for control:

: Is the rollout consistent with the robot’s camera intrinsics and egocentric viewpoint? Does it preserve depth and the precise spatial relationships required for manipulation?

: Does the robot in the rollout move in ways feasible for the embodiment, respecting morphology, joint limits, velocities, and actuation constraints?

: Does the rollout avoid physically impossible outcomes (e.g. object teleportation) which would not translate to real-world success?

. To bring video knowledge to world model form, we utilize a two-stage grounding process enabled by our integrated stack, following existing works like 

 A text-conditioned diffusion model trained on web-scale video, mid-trained on egocentric human data, and fine-tuned on NEO-specific sensorimotor logs. It predicts the high-fidelity evolution of a scene with strong visual, spatial, and physical accuracy.

 We bridge pixels to actuators by training an IDM to predict the exact action sequence required to transition between the model's generated frames. We use the IDM’s metrics and rejection sampling to enforce kinematic correctness of generations.

At inference time, the system receives a text prompt and a starting frame. The WM rolls out the intended future, the IDM extracts the necessary trajectory, and the robot executes the sequence in the real world.

At test-time, we send a prompt from our frontend to the model inference server, which will then execute the action on-robot.

The 1XWM backbone is built upon a 14B generative video model. To adapt this model to NEO’s embodiment, we use a multi-stage training strategy:

 We train on 900 hours of egocentric human video to align the model with first-person manipulation tasks. At this stage, the model captures general manipulation behaviors but struggles to generate videos of NEO performing tasks.

 We then fine-tune on 70 hours of robot data, which adapts the model to NEO’s visual appearance and kinematics.

 show that the prompt adherence of visual foundation models can be significantly improved by training on descriptive visual captions. However, many egocentric datasets contain only brief task descriptions. To address this, we prompt a VLM to generate more detailed captions to use for training through a 

To select the best 1XWM backbone checkpoint, we use an evaluation metric that compares the 

 distance between the ground truth future action sequence and the actions from our trained IDM run on the generated video. This helps filter for generations that not only look good visually but also yield accurate actions.

Similarly to DreamGen, we use a two-image IDM predictor with a sliding window of W=8 frames for efficient training and inference. However, we use a 

 backbone with a separate flow matching head instead of a single diffusion transformer. Frames at times t and t+W are passed through the depth backbone and the embeddings are used as conditioning for the flow matching head. The IDM is trained on 400 hours of unfiltered robot data including random play data and motions that don’t correspond to any meaningful task. This allows us to faithfully track NEO’s motions anywhere.

At test-time, given the starting frame and a text prompt instructing NEO on what to do, 1XWM rolls out future video frames. We then extract the corresponding robot action trajectory with the IDM, and execute it directly on the robot. To ensure smooth trajectories, the IDM’s outputs are timewise averaged across a batch of initial noise values and sliding windows.

Caption: Our NEO post-training dataset contains primarily high quality pick-and-place data (98.5%), filtered for table top manipulation with hands in view. By leveraging the web-scale pretraining of the base video model, 1XWM can generalize to a wide range of unseen objects, environments, and tasks.

We first seek to understand: what are the limits on task generalization beyond what NEO has already seen and done? How closely does generated video align with real-world execution?

manipulate unseen objects with interesting affordances

perform entirely novel tasks requiring new motions

We observe that 1XWM generated videos generally align well with real-world execution. Generated and post-factual video can look very similar viewed side-by-side, showing 1XWM has strong spatial, kinematic, and physical understanding.

Grasping

New Behaviors


Next, we try tasks requiring two-handed coordination and human interaction—abilities not included in our training dataset. This suggests that such knowledge comes from video pre-training and ego human mid-training. Because NEO’s embodiment is so similar to humans, the affordances learned in human video data translate directly.

Two-Handed

Human Interaction


Sometimes, generations tend to be overly optimistic about task completion and depth understanding. Generated rollouts can look visually plausible while subtly violating real-world constraints (e.g. object consistency, depth, geometry, contact). Our post-training recipe significantly reduces these errors, but monocular pretraining can still lead to weak 3D grounding, where the real robot undershoots or overshoots even when the generated video “succeeds.” This opens future work to integrate depth or stereo for better spatial grounding. 

Beyond anecdotal examples, we show real-world results measuring the performance of 1XWM on in-distribution (ID) and out of distribution (OOD) tasks, running 30 times each. 1XWM succeeds with stable success rates across diverse action primitives, although some dextrous tasks remain challenging (e.g. pouring, drawing).

Can we connect video quality and task success?

If so, we can measure and improve video quality using visual metrics and estimate the likelihood of real-world success. 

Sometimes it is visually obvious whether a generated rollout is likely to succeed. For example, prompting 1XWM with “pull tissue” can occasionally generate videos of NEO picking up the tissue box instead. We’ve generally found close to 0% success rate when executing bad generations. 

 can improve task success. Inspired by this, we try generating multiple rollouts in parallel and executing the best one. We study the “pull tissue” task between one and eight parallel generations, finding that the ability to select the highest quality generation from eight choices does lead to improved task success.

This selection process can be done manually, but is amenable to automation with a VLM evaluator. We leave exploration of best-of-N sampling strategies to future work. For simplicity and disambiguation with our quality of video ranking, all other results in this blog post use a single generation per attempt.

The importance of egocentric data and high-quality captions

Given our hypothesis of correlation between video quality and task success, we visually ablate a few training choices, specifically the impact of upsampling captions and training on egocentric human data.

To evaluate, we generate videos of NEO performing tasks and then ask for human feedback. We use three evaluation datasets, each containing 500 starting image-prompt pairs:

 contains hard tasks and backgrounds in-distribution with our robot training data: mostly pick and place in scenes with clutter and challenging object positions.

 consist of a split across novel tasks: whisk the bowl, pull tissue, relative size object identification (pick bigger object), bimanual manipulation, etc. collected with simple backgrounds in the real world.

 is composed entirely of pick tasks, using seed frames generated by a text-to-image (T2I) model, which randomly samples out-of-distribution household objects and backgrounds.

In-distribution

New Tasks

We ask human annotators to review each generated video and accept or reject it based on physical plausibility, task completion, and consistency with NEO’s embodiment and capabilities.


We find that upsampling captions improves video generation quality on every evaluation split. Upsampled captions better match the detailed text conditioning used during video-model pretraining and also provide clearer conditioning for task-specific motion.

Adding egocentric human data improves generation quality on both 

 splits. This is consistent with our hypothesis that ego human data contributes a transferable prior for manipulation that maps well onto NEO’s humanoid embodiment. For in-distribution tasks for which we have good NEO data coverage, egocentric data may dilute the post-training mix and cause a neutral or negative change in visual score.


Finally, we run real-world ablations on egocentric mid-training and caption upsampling, techniques we found to improve visual generation quality. For each task, we run all models 30 times in the same setting, on the same robot, and without test-time filtering. We do not control further with blind evaluation or checkpoint selection.

, we find that all models perform similarly, suggesting that in-distribution tasks are least sensitive to changes in data or caption.

 task was the most challenging across models. Models without either the egocentric human midtraining phase or caption upsampling incorrectly interpret the instruction. We find that our model with both is the only model to achieve a nonzero success rate.

As a whole, we do see a connection where including changes that impact visual model quality improve task success in real-world experiments.

Ego human data

Only NEO data

NEO + ego human data

1XWM transfers the knowledge of web-scale video data to imagine and then execute diverse tasks on NEO. From here, we’re excited to improve 1XWM to solve more complex, longer-horizon tasks required for useful household autonomy.

Building on the rapid progress observed in generative video models and equipped with data from our NEO production ramp, we will continue scaling 1XWM to improve quality and task success. Today, the 1XWM backbone takes 11 seconds to run, using multi-gpu inference built in collaboration with our friends at Verda (Antonio and Aditya). Each inference generates 5 seconds of real-time video, and the IDM takes 1 second to extract actions from the generated rollout. To solve reactive tasks and contact-rich manipulation, faster inference will improve reaction time, or latency between prompt and action execution. As we consider tasks longer than 5 seconds, closed-loop replanning with memory context will help us handle drift, partial observability, and recovery.

We don’t need perfect performance across the board to make deep progress. Nonzero success across a broad set of tasks provides the lever to unlock self-improvement.

1XWM creates a flywheel where exploration, evaluation, and policy refinement are driven by NEO’s own experience, rather than being limited by expert demonstrations. When learning is driven by experience, NEO can only improve from here, and we are excited to build a future where NEO can teach itself to master any task in any home.

If this sounds motivating to you, come help us build the future: 

https://1x.recruitee.com/o/ai-research-engineer-world-models

1X World Model | From Video to Action: A New Way Robots Learn

The 1X World Model now serves as NEO’s cognitive core—enabling generalization to unseen tasks. Ask NEO anything and watch it attempt the task entirely autonomously.

We are excited to introduce Redwood, 1X’s breakthrough AI model that we will be deploying to homes. Redwood is a vision-language transformer tailored for the humanoid form factor and capable of performing end-to-end mobile manipulation tasks like retrieving objects for users, opening doors, and navigating around the home. Redwood empowers NEO Gamma to learn from real-world experience, on top of hardware designed for compliance, safety, and resilience.

: Handles variation in tasks—like picking up never-before-seen objects in unfamiliar locations. Trained on a large dataset of teleoperated and autonomous episodes from EVE and NEO, Redwood exhibits emergent behaviors such as choosing

 Redwood is among the first VLAs to control locomotion jointly with manipulation, enabling bracing and leaning behaviors during manipulation.

 Allows NEO to position itself precisely for tasks, perform actions that require movement across space, and manipulate objects while on the move.

Redwood is compute-efficient and runs fully on NEO’s onboard embedded GPU.

In order to power autonomy on both EVE and NEO platforms, Redwood fuses pre-trained language embeddings, vision tokens from a pre-trained vision transformer, and proprioception embeddings from a sequence of joint positions and joint applied forces. These are passed through several more transformer blocks, which extract a latent representation vector. We decode this representation into EVE or NEO actions using a 



Generalizing to manipulating new objects in locations not seen in the training data is crucial for the model to work in home environments, where the home is never in the exact same configuration twice. This is achieved by training on a diverse dataset gathered on NEOs in 1X offices as well as employee homes.

To further improve generalization to new scenarios despite its small size (160M parameters), Redwood is trained not only to predict actions, but a variety of “cognitive” prediction targets like estimating the current location of NEO’s hands and relevant objects in image space. These cognitive tasks help ground NEO’s visual representations and allow it to generalize better to unseen environments, despite having a small model size. Below, we show a continuous take of Redwood being able to grasp unseen bottles from a variety of locations.

Whole Body Control and Multi-contact manipulation

Manipulation and locomotion behaviors are typically decoupled in most robotic systems. However, manipulation in the home necessitates going beyond picking small objects on counter-tops and tables: humans use their legs, hips and spine to bend down to pick toys and clothes off the ground, and lean into heavy doors when pushing them open. These “whole body control” tasks make it impossible to cleanly separate locomotion and manipulation.

To enable similar capabilities on NEO, Redwood predicts not only the arm and hand commands, but also walking, manipulation, and pelvis pose commands simultaneously. This greatly expands the kinematic reach and payload capacity NEO can work with.



Coordinating all parts of the body to engage with the environment also enables multi-contact manipulation, such as bracing a hand against a wall when pulling open heavy doors.

Solving chores requires combining manipulation with navigating across the home. In real home tasks, the objects of interest are rarely all in front of the robot at the start. Furthermore, navigating to and getting close enough to an object to grasp it needs to take into account the way the model will choose to pick up the object. If one does not train navigation skills jointly with those for manipulation, then a separate navigation stack may fail to position the robot in an optimal position to grasp the target object. Vice versa, if manipulation behavior does not take into account navigation behavior that might follow it, this could lead to unwanted collisions or carrying the object across the room in an unsafe way.

To that end, Redwood is trained on a large diverse set of object navigation and pick-and-place demonstrations within the home and is trained to plan navigation and manipulation behaviors jointly. An emergent property of training from these demonstrations is that Redwood can automatically decide to use the left, right, or both hands to pick up an object.

Running Redwood onboard allows NEO to be deployed in more diverse environments: in basements, in the garden, in homes with spotty Internet infrastructure, in wilderness campsites.

To that end, Redwood is a 160M parameter transformer model that runs on NEO’s onboard GPU at around 5hz. To pack as much intelligence as possible into a relatively small number of parameters, we’ve found that the additional cognitive losses help with grounding the representations, especially in unseen environments.

Voice control is an intuitive interface to interact with general-purpose robots in the home. Using an offboard speech-to-speech LLM, we extract the goal the user intends to command NEO with from a conversational context, and then convert the command into a vector offboard using a sentence encoder. This vector is then passed as an input into the Redwood model, which is trained on thousands of such text embeddings.

Large-scale behavior cloning methods typically only imitate successful demonstrations. Redwood is trained to learn from both successful and failure rollouts, allowing NEO to improve from any interaction it has with the world regardless of success. The failure rollouts provide supervision signals on the cognitive prediction heads, which helps prevent overfitting to a relatively narrow distribution of states seen during successful demonstrations. The successful demonstrations supervise both the action diffusion heads and the cognitive predictions.

We think that general purpose autonomous humanoids, with their intelligence incubated in the home, will be a generation-defining technology that reshapes quality of life for the elderly, for busy parents, and for use cases that scarcely cannot be imagined today. We’re looking for driven, high-agency engineers to help us scale up Redwood to the next level, and to deploy a production-grade AI in as many homes as possible this year. If working on Redwood excites you, we have a large number of open roles in Palo Alto:

Research Engineer, Reinforcement Learning

Redwood is the AI model that enables NEO to help around the home—moving, perceiving, and interacting with the world autonomously.

Redwood is a vision-language transformer tailored for the humanoid form factor and capable of performing end-to-end chores around the house. As NEO spends more time doing new chores across more homes, Redwood becomes more robust and versatile.

INTRODUCING

At 1X, we build robots that help humans in the most diverse environment imaginable: people’s homes. To deploy our Redwood AI model safely and reliably, we need to anticipate its policy behavior across all that it can encounter. From retrieving a rarely-used kitchen gadget tucked away in a cluttered drawer, to navigating a living room unexpectedly rearranged overnight, physically evaluating each policy we've trained across these varied scenarios would take several lifetimes. How can we accelerate the evaluation of generalist robot models?

Today, we share progress on the 1X World Model (1XWM): a bridge between the world of atoms and the world of bits.

The 1X World Model enables NEO to anticipate the outcomes of robot actions and their consequences on the world.



This approach significantly accelerates experimentation, allowing us to evaluate the reliability and effectiveness of robotic policies in a fraction of the time, without requiring intensive, environment-specific engineering within traditional physics-based simulators or mock real-world sets.

Precise action controllability that enables us to compare policies that take different actions given the same observations.

Scaling results when leveraging a specific source of data: autonomous policy rollouts

High correlation between the world model and real world evaluations

Quickly iterate on architectural decisions.

Select the best checkpoint from training runs.

Curate datasets of long-tail scenarios in production and re-evaluate models on them.

Optimize robot policies at scale through efficient training-evaluation cycles.



We train our world model on sequences of video frames, robot observations, and input action trajectories. We encode the inputs to latent representations, and predict the latent encodings of future frames. We also predict the state value of the final frame to evaluate task success and completion.

Specializing Video Generation Models for Action Controllability

Most video generation models are text-to-video (T2V), using a language prompt to generate the video, and optionally, one or more reference frames for guidance. However, world models for simulating robots need to be action-controllable, steered by exact robot trajectories rather than loose directives like “Grab the mug” or “Wipe the countertop.” We demonstrate the action-controllability of 1XWM by providing it with a few initial frames of real footage along with multiple subsequent action trajectories. From this anchor point, the 1XWM simulates the consequences of taking those exact actions, including the physics of objects like doors being opened and cloths being wiped across a countertop.

LAUNDRY

OPEN

CLEAN

Does 1XWM improve as data is scaled up? What kinds of data best improve its understanding of physics?

To study this, we train 1XWM to not only predict future states and images, but also whether the task attempt succeeded or failed at the end of each generation. We see improvements in accuracy across the board as we scale up the number of tasks and the diversity of robot behaviors. We explore this in the following tasks: Airfryer, Arcade, and Shelf.

Air fryer

Arcade

Shelf



We observe a clear improvement in generation quality as we train on task-specific data. When confronted with an unfamiliar task and environment, the WM often struggles to model the object interactions exactly, without having knowledge of their specific properties. Training on task-specific data allows the WM to update based on the subtle dynamics of the task at hand.



For example, when trained on smaller amounts of data, 1XWM hallucinates the air fryer tray and body as a single unit, pulling the entire unit off the counter. After adding interaction data with the air fryer, 1XWM gains a better understanding of how the tray separates from the air fryer, and even learns to model subtle interactions such as the confinement of the tray movement within the base of the air fryer.



We also observe that training on both shelf and arcade data improves accuracy compared to training on shelf alone. This positive transfer of accuracy and task understanding from one task to another reinforces our belief in the capability of 1XWM on scaling.

The more task-oriented robotics data we accumulate, the more accurately we can predict task-level future outcomes. We then turn this capability into an evaluation engine as a novel application of world modeling for robotics development.

An aligned world model can solve the evaluation problem by forecasting the actions of candidate robot policies. Given 1X World Model generations from each policy on datasets of initial states, we can compare their respective performances. Importantly, we can curate datasets of production-setting states and generate counterfactual results from states that an autonomous policy has previously failed in.

For every set of model checkpoint weights, we predict future states and success likelihood that we show are distributionally aligned to actual real-world futures. This gives us insight into model performance at scale and allows us to make model architecture and checkpoint selection decisions with an instant feedback loop.



In the plot below, we ablate the decision to include proprioception (robot joint states) as input to our robot policy, and plot the 1X World Model predicted success rate for each checkpoint. We then run real world evaluation on the most and least promising checkpoints according to 1XWM. We find that there is indeed a correlation between the predicted success rates and the true task scores. This allows us to make a likely forecast that proprioception improves policy performance.

Given a true real-world success rate gap of 15% between two policies, a World Model with 70% accuracy can accurately predict the better policy with 90% success. Given that we see a consistent predicted performance gap across checkpoints, and that we can evaluate policies on identical starting states, we can be even more confident in such a verdict.



As another experiment, we compare using two different image encoders for a policy. We compare the most promising predicted checkpoints from both policies, and see that the predicted better ViT-L model does indeed perform better in the real world. For more experiments, see our technical report.

As we deploy robots in home, we will need to move away from task-specific evaluation and towards production-level evaluation, capable of handling a wider, more ambiguous array of full-body manipulation tasks and objects. Improving the generalization capability and accuracy of 1XWM will be the
first step towards this goal.

LIMITATION #1

LIMITATION #2

We've shown promising results in scaling up the 1X World Model (1XWM) to predict the future. As we increase the amount of training compute and real world NEO data, 1XWM's accuracy of predicting whether NEO accomplishes a task also increases. 

The implications of accurate hallucinated rollouts extend far beyond rigorous evaluation for humanoid robots. Consider the implications of what happens when the data generated by 1XWM – the joint distribution over all the sensor readings and actions observed by the robot – become indistinguishable from the real data. This moment has already happened for LLMs, and we think that it will also soon be true for synthetic robotics data. Data and evals are the cornerstone of solving autonomy, and 1XWM provides a unified path for tackling both challenges.

The 1X World Model is a video model grounded in physics. It allows us to predict—or "hallucinate"—the outcomes of NEO's actions before they're taken in the real world.

Artificial Intelligence

Learning New Abilities

Evaluation

Mobility

Introducing Redwood AI

Now Hiring

Body Language Engineer, AI Companion

Palo Alto, CA

Multimodal Research Engineer, AI Companion

Palo Alto, CA

AI Research Engineer, Scaling

Palo Alto, CA

Software Engineer, Simulation

Palo Alto, CA

AI Research Engineer, VLA Models

Palo Alto, CA

AI Research Engineer, Data Infrastructure

Palo Alto, CA

ExperienceHumanoid