Opt in to receive updates. Unsubscribe anytime.

Experience NEO

Home robots need common sense behavior and a deep understanding of the physical world.

Many robot foundation models today are vision-language-action models (VLAs), which take a pretrained VLM and add an output head to predict robot actions (

). VLMs benefit from internet-scale knowledge, but are trained on objectives that emphasize visual and semantic understanding over prediction of physical dynamics. Tens of thousands of hours of costly robot data are needed to teach a model how to solve tasks considered simple for a human. Additionally, auxiliary objectives are often used to further coax spatial reasoning of physical interactions (

In this blog, we introduce our video-pretrained world model, 1XWM, integrated into NEO as a robot policy. While VLAs directly predict action trajectories from static image-language input, our world-model based policy derives robot actions from text-conditioned video generation. By leveraging the world dynamics inherent in internet-scale video, our world model generalizes to novel objects, motions, and tasks without pre-training on large-scale robot data or any related teleoperated demonstrations. This represents a shift towards a new regime: allowing robot intelligence to benefit from the scaling of video pretraining, enabled by a hardware stack designed for high fidelity transfer from human embodiment.

Internet video implicitly encodes the structural priors of reality: how people and objects move, where forces are applied, and the implicit constraints of interaction. Accurate translation of video into action comes from more than just a model—it benefits from an embodiment that is kinematically and dynamically congruent with the human form.

The hardware as a first-class citizen in the AI stack closes the human-robot translation gap. By combining embodiment with human-like compliance, interaction dynamics—friction, inertia, and contact behavior—often match human motion closely enough for the model’s learned priors to remain in distribution. What the model can visualize, NEO usually can do.

Today, frontier text-to-video models like 

 generate strikingly realistic videos. However, zero-shot generations from these models are not grounded in robot embodiments and often fall short along key axes required for control:

: Is the rollout consistent with the robot’s camera intrinsics and egocentric viewpoint? Does it preserve depth and the precise spatial relationships required for manipulation?

: Does the robot in the rollout move in ways feasible for the embodiment, respecting morphology, joint limits, velocities, and actuation constraints?

: Does the rollout avoid physically impossible outcomes (e.g. object teleportation) which would not translate to real-world success?

. To bring video knowledge to world model form, we utilize a two-stage grounding process enabled by our integrated stack, following existing works like 

 A text-conditioned diffusion model trained on web-scale video, mid-trained on egocentric human data, and fine-tuned on NEO-specific sensorimotor logs. It predicts the high-fidelity evolution of a scene with strong visual, spatial, and physical accuracy.

 We bridge pixels to actuators by training an IDM to predict the exact action sequence required to transition between the model's generated frames. We use the IDM’s metrics and rejection sampling to enforce kinematic correctness of generations.

At inference time, the system receives a text prompt and a starting frame. The WM rolls out the intended future, the IDM extracts the necessary trajectory, and the robot executes the sequence in the real world.

At test-time, we send a prompt from our frontend to the model inference server, which will then execute the action on-robot.

The 1XWM backbone is built upon a 14B generative video model. To adapt this model to NEO’s embodiment, we use a multi-stage training strategy:

 We train on 900 hours of egocentric human video to align the model with first-person manipulation tasks. At this stage, the model captures general manipulation behaviors but struggles to generate videos of NEO performing tasks.

 We then fine-tune on 70 hours of robot data, which adapts the model to NEO’s visual appearance and kinematics.

 show that the prompt adherence of visual foundation models can be significantly improved by training on descriptive visual captions. However, many egocentric datasets contain only brief task descriptions. To address this, we prompt a VLM to generate more detailed captions to use for training through a 

To select the best 1XWM backbone checkpoint, we use an evaluation metric that compares the 

 distance between the ground truth future action sequence and the actions from our trained IDM run on the generated video. This helps filter for generations that not only look good visually but also yield accurate actions.

Similarly to DreamGen, we use a two-image IDM predictor with a sliding window of W=8 frames for efficient training and inference. However, we use a 

 backbone with a separate flow matching head instead of a single diffusion transformer. Frames at times t and t+W are passed through the depth backbone and the embeddings are used as conditioning for the flow matching head. The IDM is trained on 400 hours of unfiltered robot data including random play data and motions that don’t correspond to any meaningful task. This allows us to faithfully track NEO’s motions anywhere.

At test-time, given the starting frame and a text prompt instructing NEO on what to do, 1XWM rolls out future video frames. We then extract the corresponding robot action trajectory with the IDM, and execute it directly on the robot. To ensure smooth trajectories, the IDM’s outputs are timewise averaged across a batch of initial noise values and sliding windows.

Caption: Our NEO post-training dataset contains primarily high quality pick-and-place data (98.5%), filtered for table top manipulation with hands in view. By leveraging the web-scale pretraining of the base video model, 1XWM can generalize to a wide range of unseen objects, environments, and tasks.

We first seek to understand: what are the limits on task generalization beyond what NEO has already seen and done? How closely does generated video align with real-world execution?

manipulate unseen objects with interesting affordances

perform entirely novel tasks requiring new motions

We observe that 1XWM generated videos generally align well with real-world execution. Generated and post-factual video can look very similar viewed side-by-side, showing 1XWM has strong spatial, kinematic, and physical understanding.

Grasping

New Behaviors


Next, we try tasks requiring two-handed coordination and human interaction—abilities not included in our training dataset. This suggests that such knowledge comes from video pre-training and ego human mid-training. Because NEO’s embodiment is so similar to humans, the affordances learned in human video data translate directly.

Two-Handed

Human Interaction


Sometimes, generations tend to be overly optimistic about task completion and depth understanding. Generated rollouts can look visually plausible while subtly violating real-world constraints (e.g. object consistency, depth, geometry, contact). Our post-training recipe significantly reduces these errors, but monocular pretraining can still lead to weak 3D grounding, where the real robot undershoots or overshoots even when the generated video “succeeds.” This opens future work to integrate depth or stereo for better spatial grounding. 

Beyond anecdotal examples, we show real-world results measuring the performance of 1XWM on in-distribution (ID) and out of distribution (OOD) tasks, running 30 times each. 1XWM succeeds with stable success rates across diverse action primitives, although some dextrous tasks remain challenging (e.g. pouring, drawing).

Can we connect video quality and task success?

If so, we can measure and improve video quality using visual metrics and estimate the likelihood of real-world success. 

Sometimes it is visually obvious whether a generated rollout is likely to succeed. For example, prompting 1XWM with “pull tissue” can occasionally generate videos of NEO picking up the tissue box instead. We’ve generally found close to 0% success rate when executing bad generations. 

 can improve task success. Inspired by this, we try generating multiple rollouts in parallel and executing the best one. We study the “pull tissue” task between one and eight parallel generations, finding that the ability to select the highest quality generation from eight choices does lead to improved task success.

This selection process can be done manually, but is amenable to automation with a VLM evaluator. We leave exploration of best-of-N sampling strategies to future work. For simplicity and disambiguation with our quality of video ranking, all other results in this blog post use a single generation per attempt.

The importance of egocentric data and high-quality captions

Given our hypothesis of correlation between video quality and task success, we visually ablate a few training choices, specifically the impact of upsampling captions and training on egocentric human data.

To evaluate, we generate videos of NEO performing tasks and then ask for human feedback. We use three evaluation datasets, each containing 500 starting image-prompt pairs:

 contains hard tasks and backgrounds in-distribution with our robot training data: mostly pick and place in scenes with clutter and challenging object positions.

 consist of a split across novel tasks: whisk the bowl, pull tissue, relative size object identification (pick bigger object), bimanual manipulation, etc. collected with simple backgrounds in the real world.

 is composed entirely of pick tasks, using seed frames generated by a text-to-image (T2I) model, which randomly samples out-of-distribution household objects and backgrounds.

In-distribution

New Tasks

We ask human annotators to review each generated video and accept or reject it based on physical plausibility, task completion, and consistency with NEO’s embodiment and capabilities.


We find that upsampling captions improves video generation quality on every evaluation split. Upsampled captions better match the detailed text conditioning used during video-model pretraining and also provide clearer conditioning for task-specific motion.

Adding egocentric human data improves generation quality on both 

 splits. This is consistent with our hypothesis that ego human data contributes a transferable prior for manipulation that maps well onto NEO’s humanoid embodiment. For in-distribution tasks for which we have good NEO data coverage, egocentric data may dilute the post-training mix and cause a neutral or negative change in visual score.


Finally, we run real-world ablations on egocentric mid-training and caption upsampling, techniques we found to improve visual generation quality. For each task, we run all models 30 times in the same setting, on the same robot, and without test-time filtering. We do not control further with blind evaluation or checkpoint selection.

, we find that all models perform similarly, suggesting that in-distribution tasks are least sensitive to changes in data or caption.

 task was the most challenging across models. Models without either the egocentric human midtraining phase or caption upsampling incorrectly interpret the instruction. We find that our model with both is the only model to achieve a nonzero success rate.

As a whole, we do see a connection where including changes that impact visual model quality improve task success in real-world experiments.

Ego human data

Only NEO data

NEO + ego human data

1XWM transfers the knowledge of web-scale video data to imagine and then execute diverse tasks on NEO. From here, we’re excited to improve 1XWM to solve more complex, longer-horizon tasks required for useful household autonomy.

Building on the rapid progress observed in generative video models and equipped with data from our NEO production ramp, we will continue scaling 1XWM to improve quality and task success. Today, the 1XWM backbone takes 11 seconds to run, using multi-gpu inference built in collaboration with our friends at Verda (Antonio and Aditya). Each inference generates 5 seconds of real-time video, and the IDM takes 1 second to extract actions from the generated rollout. To solve reactive tasks and contact-rich manipulation, faster inference will improve reaction time, or latency between prompt and action execution. As we consider tasks longer than 5 seconds, closed-loop replanning with memory context will help us handle drift, partial observability, and recovery.

We don’t need perfect performance across the board to make deep progress. Nonzero success across a broad set of tasks provides the lever to unlock self-improvement.

1XWM creates a flywheel where exploration, evaluation, and policy refinement are driven by NEO’s own experience, rather than being limited by expert demonstrations. When learning is driven by experience, NEO can only improve from here, and we are excited to build a future where NEO can teach itself to master any task in any home.

If this sounds motivating to you, come help us build the future: 

https://1x.recruitee.com/o/ai-research-engineer-world-models

1X World Model | From Video to Action: A New Way Robots Learn

1X is excited to announce that Mohi Khansari has been appointed Head of Robot Learning, expanding his role after more than a year as Distinguished AI Engineer at the company.

Mohi is a veteran roboticist with over two decades of experience in robot learning & autonomy and control. He has been the chief architect behind Redwood AI, 1X’s in-house vision-language transformer powering end-to-end mobile manipulation in NEO, and his work has been central to advancing NEO’s real-world perception, learning, and manipulation capabilities.

Prior to 1X, Mohi was a founding member and lead of the imitation learning effort at Everyday Robots, a Google X moonshot focused on bringing embodied AI into human environments. He later served as a key technical lead at Cruise, advancing the intersection of motion planning and deep robot learning for large-scale autonomous systems.

Mohi holds a PhD in Robotics from École Polytechnique Fédérale de Lausanne (EPFL) and completed a postdoctoral appointment at the Stanford AI Lab, where his research focused on robot learning and control.

“Joining 1X has been an incredible opportunity to move humanoid robotics beyond 'impressive research demos' and into technology that delivers real value for people. 1X’s humanoid robot, NEO, is the kind of platform this moment has been waiting for—engineered for safe, capable interaction around people, with the strength and dexterity to do useful work. I’m excited to lead Robot Learning at 1X, and build the intelligence that enables NEO to help people in their daily lives with my team.” -Mohi Khansari

As Head of Robot Learning, Mohi will lead efforts to advance learning systems that bridge research and deployment, accelerating 1X’s mission to deliver safe, general-purpose humanoid robots.

“Mohi is one of those rare builders who has contributed key academic innovations in the field and can still ship. He’s spent twenty years turning learning and control into working robots—not papers, not prototypes, but real capability. Putting him in this role strengthens the bridge between our research ambition and what NEO can do in the world.” - Bernt Børnich, Founder & CEO.

1X Appoints Mohi Khansari as Head of Robot Learning

Palo Alto, CA, October 28, 2025 – 1X is proud to announce the launch of NEO, the world’s first consumer-ready humanoid robot designed to transform life at home. NEO automates everyday chores and offers personalized assistance so people can spend more time on the things that matter.

With intuitive, user-friendly features and a revolutionary hardware platform, NEO is the safest, most affordable, and most capable humanoid on earth. NEO brings the science-fiction idea of robots from the movie screen into reality — creating space for us to enjoy our time at home, instead of coming home just to do more work.

"Humanoids were long a thing of sci-fi... then they were a thing of research, but today — with the launch of NEO — humanoid robots become a product. Something that you and me can reach out and touch. NEO closes the gap between our imaginations and the world we live in, to the point where we can actually ask a humanoid robot for help, and help is granted."

- Bernt Børnich, CEO and Founder, 1X

The Chores feature allows owners to give NEO a list of tasks to complete around the home, schedule a time for when they want them completed, and come back to a cleaner space every day. NEO can also handle chores in real time while you relax or work nearby. With the click of a button or a simple verbal command, NEO transforms into a personal housekeeper; tackling tasks like folding laundry, organizing shelves, and tidying spaces. For any chore that NEO doesn't yet know, owners can schedule a 1X Expert to guide it through unknown tasks — helping NEO learn while getting the job done.


NEO is built to engage naturally while providing help and utility through conversation. With a built-in large language model (LLM), NEO gives owners instant access to knowledge and personalized assistance without relying on devices with screens.

Its Audio Intelligence allows NEO to recognize when it is being addressed, ensuring it listens and responds only when appropriate. Visual Intelligence adds contextual awareness — letting NEO use what it sees to enhance interactions, like recognizing ingredients on a kitchen counter and suggesting dishes to make. Memory enables continuity across conversations, allowing NEO to recall past context and adapt its responses over time. From scheduling appointments and setting reminders for birthdays to keeping grocery lists and tracking progress in language lessons, not only does NEO free up time, but also valuable mental space.

NEO Operates Autonomously and Learns New Skills Over Time

The core of the NEO experience is driven by AI. Each NEO will be able to have conversations, and navigate around the home to provide help where needed. NEO will arrive to early owners with the ability to handle basic tasks, and with continued use, learn and develop new skills. From the first day, NEO performs functions like opening doors for guests, fetching items, and turning off the lights at night — and grows in abilities with every software update. Leveraging 1X’s AI model, which is trained on real-world data, NEO is able to adapt to the world’s most diverse environment: the home.

At the core of the NEO experience is a revolutionary hardware platform that succeeds 1X’s prior models (NEO Gamma and NEO Beta). NEO is built with 1X’s patented Tendon Drive that uses the highest-torque density motors on earth to drive its tendon-based transmissions. This one-of-a-kind actuation system creates gentle movements that make NEO uniquely safe and compliant around people.

NEO represents a generational leap in humanoid hardware with the addition of Human Level Dexterity (22 DoF Hands) and a head-to-toe soft body made of custom 3D lattice polymer structures. At just 66 pounds (29.94 kg), NEO is capable of lifting over 150 pounds (68 kg) and carrying 55 pounds (24.95kg), and has a noise level of 22dB — making it more silent than a modern day refrigerator. Additionally, NEO possesses built-in communication with WiFi, Bluetooth, and 5G as well as a three-stage speaker in the pelvis and chest area, giving owners a built-in mobile home entertainment system.

NEO is approachable and comfortable to be around. NEO’s essentialist design aims to complement living spaces rather than disrupt them – with organic neutral tones, and a soft knit suit and shoes.

NEO is now available for pre-order, with first orders shipping to consumer homes in 2026.

NEO is now available via 1X’s online store [

] and comes in three distinct colors (Tan, Gray, Dark Brown). Customers interested owning one of the first NEOs can purchase Early Access for $20,000, which includes priority delivery in 2026. There is also a subscription model offering of $499/month which will be shipped at a later date.

1X will start delivering NEOs primarily in the U.S in 2026 and expand to other markets starting in 2027.

To the community that supports us and the team here at 1X that works tirelessly to build a better future, we can’t thank you enough. Another special thanks to Eli Russell Linnetz for capturing NEO for this campaign.

NEO Home Robot | Order Today

We're excited to announce Vikram Kothari as our new VP of Operations.

He joins the 1X team after over eight years at SpaceX leading Dragon, Starship, Raptor, and Launch Avionics supply chain orgs. Preceding his time at SpaceX he was a Director at Microsoft, where he also led hardware supply chain for almost a decade.

“Joining 1X Technologies has been an inspiring step for me. The company’s commitment to building safe, privacy-first humanoids that are designed to coexist meaningfully with humans sets it apart in the robotics space. The team’s depth of talent and shared sense of purpose create an environment that’s both ambitious and grounded. I’m excited to contribute to a mission that places human values at the center of innovation." -Vikram Kothari.

Vikram's tenure at SpaceX streamlined operations and powered historic milestones, including the development of Starship and Starlink’s global expansion-- his expertise in scaling complex supply chains will serve as an even greater accelerant to 1X’s mission to deploy humanoids into homes.

“Vikram joining the team is a game-changer. Manufacturing has yet to find its heroes—but as we bring NEO into the world, Vik’s impact on the re-industrialization of America and the future of humanoids will be a defining part of the story.” -Bernt Bornich

Welcoming Vikram Kothari as VP of Operations

We're excited to announce our new HQ in Palo Alto, California.

The 80,000 sq. foot facility will seat 400 people, with employees from the company's previous offices migrating to the new space over the course of summer 2025.

Our leadership is confident that by positioning the team at the global center of AI and robotics, 1X will be able to attract top talent and foster even closer collaboration with our partners and investors.

As we grow, consolidating our Sunnyvale [CA] and Moss [Norway] team under one roof will accelerate our speed to ramping production and getting NEO into homes near you.

Join us on our mission to create an abundance of labor in the world.

Opening our new HQ in Palo Alto

At 1X, we build robots that help humans in the most diverse environment imaginable: people’s homes. To deploy our Redwood AI model safely and reliably, we need to anticipate its policy behavior across all that it can encounter. From retrieving a rarely-used kitchen gadget tucked away in a cluttered drawer, to navigating a living room unexpectedly rearranged overnight, physically evaluating each policy we've trained across these varied scenarios would take several lifetimes. How can we accelerate the evaluation of generalist robot models?

Today, we share progress on the 1X World Model (1XWM): a bridge between the world of atoms and the world of bits.

The 1X World Model enables NEO to anticipate the outcomes of robot actions and their consequences on the world.



This approach significantly accelerates experimentation, allowing us to evaluate the reliability and effectiveness of robotic policies in a fraction of the time, without requiring intensive, environment-specific engineering within traditional physics-based simulators or mock real-world sets.

Precise action controllability that enables us to compare policies that take different actions given the same observations.

Scaling results when leveraging a specific source of data: autonomous policy rollouts

High correlation between the world model and real world evaluations

Quickly iterate on architectural decisions.

Select the best checkpoint from training runs.

Curate datasets of long-tail scenarios in production and re-evaluate models on them.

Optimize robot policies at scale through efficient training-evaluation cycles.



We train our world model on sequences of video frames, robot observations, and input action trajectories. We encode the inputs to latent representations, and predict the latent encodings of future frames. We also predict the state value of the final frame to evaluate task success and completion.

Specializing Video Generation Models for Action Controllability

Most video generation models are text-to-video (T2V), using a language prompt to generate the video, and optionally, one or more reference frames for guidance. However, world models for simulating robots need to be action-controllable, steered by exact robot trajectories rather than loose directives like “Grab the mug” or “Wipe the countertop.” We demonstrate the action-controllability of 1XWM by providing it with a few initial frames of real footage along with multiple subsequent action trajectories. From this anchor point, the 1XWM simulates the consequences of taking those exact actions, including the physics of objects like doors being opened and cloths being wiped across a countertop.

LAUNDRY

OPEN

CLEAN

Does 1XWM improve as data is scaled up? What kinds of data best improve its understanding of physics?

To study this, we train 1XWM to not only predict future states and images, but also whether the task attempt succeeded or failed at the end of each generation. We see improvements in accuracy across the board as we scale up the number of tasks and the diversity of robot behaviors. We explore this in the following tasks: Airfryer, Arcade, and Shelf.

Air fryer

Arcade

Shelf



We observe a clear improvement in generation quality as we train on task-specific data. When confronted with an unfamiliar task and environment, the WM often struggles to model the object interactions exactly, without having knowledge of their specific properties. Training on task-specific data allows the WM to update based on the subtle dynamics of the task at hand.



For example, when trained on smaller amounts of data, 1XWM hallucinates the air fryer tray and body as a single unit, pulling the entire unit off the counter. After adding interaction data with the air fryer, 1XWM gains a better understanding of how the tray separates from the air fryer, and even learns to model subtle interactions such as the confinement of the tray movement within the base of the air fryer.



We also observe that training on both shelf and arcade data improves accuracy compared to training on shelf alone. This positive transfer of accuracy and task understanding from one task to another reinforces our belief in the capability of 1XWM on scaling.

The more task-oriented robotics data we accumulate, the more accurately we can predict task-level future outcomes. We then turn this capability into an evaluation engine as a novel application of world modeling for robotics development.

An aligned world model can solve the evaluation problem by forecasting the actions of candidate robot policies. Given 1X World Model generations from each policy on datasets of initial states, we can compare their respective performances. Importantly, we can curate datasets of production-setting states and generate counterfactual results from states that an autonomous policy has previously failed in.

For every set of model checkpoint weights, we predict future states and success likelihood that we show are distributionally aligned to actual real-world futures. This gives us insight into model performance at scale and allows us to make model architecture and checkpoint selection decisions with an instant feedback loop.



In the plot below, we ablate the decision to include proprioception (robot joint states) as input to our robot policy, and plot the 1X World Model predicted success rate for each checkpoint. We then run real world evaluation on the most and least promising checkpoints according to 1XWM. We find that there is indeed a correlation between the predicted success rates and the true task scores. This allows us to make a likely forecast that proprioception improves policy performance.

Given a true real-world success rate gap of 15% between two policies, a World Model with 70% accuracy can accurately predict the better policy with 90% success. Given that we see a consistent predicted performance gap across checkpoints, and that we can evaluate policies on identical starting states, we can be even more confident in such a verdict.



As another experiment, we compare using two different image encoders for a policy. We compare the most promising predicted checkpoints from both policies, and see that the predicted better ViT-L model does indeed perform better in the real world. For more experiments, see our technical report.

As we deploy robots in home, we will need to move away from task-specific evaluation and towards production-level evaluation, capable of handling a wider, more ambiguous array of full-body manipulation tasks and objects. Improving the generalization capability and accuracy of 1XWM will be the
first step towards this goal.

LIMITATION #1

LIMITATION #2

We've shown promising results in scaling up the 1X World Model (1XWM) to predict the future. As we increase the amount of training compute and real world NEO data, 1XWM's accuracy of predicting whether NEO accomplishes a task also increases. 

The implications of accurate hallucinated rollouts extend far beyond rigorous evaluation for humanoid robots. Consider the implications of what happens when the data generated by 1XWM – the joint distribution over all the sensor readings and actions observed by the robot – become indistinguishable from the real data. This moment has already happened for LLMs, and we think that it will also soon be true for synthetic robotics data. Data and evals are the cornerstone of solving autonomy, and 1XWM provides a unified path for tackling both challenges.

1X World Model

Having complete and unfettered access to all parts of the world is important for NEO to accomplish tasks in home environments, and to enable our Redwood AI to learn from the broadest set of physical interactions possible. Our latest RL controller provides Redwood with a complete mobility toolkit to access the world, including natural walking in any direction, sitting, standing, kneeling, getting up from the floor, going up and down stairs with stereo vision. Bringing all of these capabilities together for the first time in a unified controller represents an important milestone in unlocking the full potential of humanoid robots.

SIT & STAND

Full stairs

Bridging Natural Movement with Omnidirectional steering

The past decade has seen rapid progress in legged robot locomotion, primarily driven by torque-transparent actuators, deep Reinforcement Learning (RL) algorithms, and GPU-accelerated simulation. Using off-the-shelf software packages and a desktop GPU, it is now possible to train a bipedal robot to stand upright in simulation and follow walking direction commands in under an hour. Even though the policy is trained completely in a simulation environment whose physics are merely an approximation of the real world, it is trained on so many randomized physics parameters – e.g. friction, mass, sensor noise – that the model ends up being robust to the real world’s physics parameters. Once trained, this walking controller can receive walking direction commands from either the teleoperator or the AI model (e.g. Redwood), and then translate those high level directions into the dynamic, contact-aware interactions with the world.



Beyond walking and turning, bipeds can also perform side-stepping. This is useful for navigating the close quarters of a kitchen or the space between the sofa and the coffee table, where the footprint would be too small for a wheeled base robot.



However, these basic walking RL controllers often require additional “shaping rewards” to achieve a natural human-like gait in all directions. These tend to be highly specialized to walking, which means that the same process of hand-tuning rewards must be repeated for every new behavior. Gait patterns can vary based on the direction of walking, so this often requires unique shaping terms based on the direction of motion.

Is there a more scalable way to increase controller capabilities without hand-written shaping rewards for each of them? One method is to collect motion capture references from humans moving naturally, retarget them to NEO’s joints and body, and then train the RL controller to match those kinematic reference trajectories.

Because the reference only specifies where the body should be, the RL controller still needs to figure out how to keep the robot stable, while “keeping tempo” and tracking the reference trajectory as closely as possible.

Using these techniques, it is possible to over-fit a policy to track a single human motion capture trajectory and achieve very dynamic and fluid motions such as dancing or walking. Examples of “single trajectory replay controller” are shown below for natural running and pivoting:

Pivot



These behaviors, while elegant, are not readily useful for general purpose tasks as they only can replay a single trajectory. They do not expose a steerable interface with which a high level policy like the AI model can execute the right actions. It is also not obvious how to transition smoothly from one reference to another, as motion capture datasets rarely include the transition behaviors between arbitrary task-centric motions, e.g. switching from shuffling quickly side-to-side to a skipping motion.

To handle multiple trajectories, we could train the controller to follow multiple mocap references, taking in the encoded kinematic trajectory as an input. However, this approach encounters a teleoperation UX problem: it is not obvious how one provides the high-dimensional kinematic trajectories at test-time using a more limited input device like a gamepad joystick or a VR controller. The model is trained on high-dimensional kinematic trajectories with nuanced rhythm and periodicity, but the commands provided by teleoperation are coarse-grained, which results in the RL controller interpreting it into an unnatural gait.

How does one achieve steerability and robustness, while still achieving the fluidity of motion capture-based RL training? We developed a two-stage controller, consisting of a high level “kinematic planner” to synthesize kinematic targets that resemble human motion capture data, and a low level controller that tries to achieve those plans.



The lower level RL controller takes as input a kinematic reference trajectory of body poses that it must attempt to track while maintaining balance. This is paired with a high level motion generator model that is trained with supervised learning to convert input commands like joystick direction to the richer kinematic trajectories. The generative model also plays the role of smoothing transitions during behavior changes.

An important reason for having robots with legs in the home is to traverse stairs. We develop a “stair mode” in our controller which engages the use of stereo RGB vision to infer the height of the floor around NEO.

To climb up and down stairs in a graceful way, NEO’s RL controller must anticipate the necessary height of each step well in advance of making contact with that step. Unlike most humanoid robots, which employ a time-of-flight depth sensor or a lidar to estimate the floor plane, NEO’s RL controller is purely vision based. Depth is predicted directly from the RGB stereo pair, and this is fused with the proprioceptive history for NEO to figure out how and where to step.



Stairs are not always idealized; through domain randomization in simulation, the controller is also robust enough to support side-stepping and handling stairs of mixed heights.

There are numerous home chores that require NEO to work at floor height for extended periods of time: removing a stain from the carpet, reorganizing the bottom drawer of a cabinet, packing a suitcase, and sorting socks. We extend our RL controller to be able to safely sit, kneel and lie down on the floor, as well as get back up from each of these poses.

Kneel

Sit & stand

The controller provides an “action interface” for which teleoperation or Redwood AI is able to interact in a safe, contact-rich manner with the physical world. To demonstrate the controllability of the natural walking behavior, we fine-tuned the Redwood model to do a soccer ball dribbling task.

Here is Redwood interacting with this new RL controller, where it predicts whole body joint targets and walking pelvis velocities from vision. The controller then translates those intents into the specific forces applied by the leg to walk in the direction of the ball.

We have developed the first general-purpose, fully AI and teleoperation compatible controller that unlocks the full kinematic workspace that is available to a bipedal humanoid robot. This will enable us to train Redwood AI to fully explore the entire state space of the home: every high and low shelf, every nook and cranny, every floor.

Opening our new HQ in Palo Alto

Discover

1X World Model | From Video to Action: A New Way Robots Learn

1X Appoints Mohi Khansari as Head of Robot Learning

NEO Home Robot | Order Today

ExperienceNEO