LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition
Researchers from Carnegie Mellon University have introduced LUCID, a two-stage framework that enables robots to learn dexterous manipulation skills from unstructured human videos at internet scale. The system separates intent prediction (what should happen next in a scene) from embodiment-specific motor control, allowing the same intent model to work across different robot platforms — from dexterous hands to parallel-jaw grippers. Evaluated on five real-world tasks including stirring, wiping, binning, push-T, and cable routing, LUCID achieved zero-shot transfer to novel scenes and objects using only internet video as training data.

Why Robot Learning Needs a New Approach
For years, teaching robots new skills has followed a painfully expensive formula. Engineers collect thousands of hours of robot demonstrations, carefully structure every movement, and train models that are tightly coupled to a single physical embodiment. A robot arm trained to pick up a cup cannot transfer that knowledge to a different gripper. A system trained on structured demos cannot learn from the endless supply of unlabeled human videos on the internet.
Researchers from Carnegie Mellon University have now introduced LUCID (Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos), a framework that fundamentally rethinks this pipeline. Their work, available on arXiv (arXiv:2606.11628), proposes a clean separation between understanding what should happen in a scene and figuring out how to make it happen — a distinction that opens the door to internet-scale robot learning.
The Core Insight: Separate Intent from Control
The central innovation of LUCID is a two-stage architecture that decouples scene-level understanding from motor execution:
Stage 1: The Intent Model
The intent model is trained on unstructured human videos drawn from internet-scale datasets — the kind of video already available by the billions on platforms like YouTube. Rather than trying to extract robot-specific actions from these videos, the model learns to predict short-horizon intent: what should happen next in the scene based on the current observation. For example, given a frame of a human hand approaching a spoon in a pot, the model predicts that the spoon should make contact with the liquid — a scene-level change that is independent of which actuator will perform it.
This intent model is embodiment-agnostic by design. It describes scene outcomes, not motor commands. It learns from any manipulation video regardless of who or what is doing the manipulating. And crucially, it operates in a closed loop, continuously predicting the next short-horizon change as the scene evolves.
Stage 2: The Sensorimotor Policy
The second stage is an embodiment-specific sensorimotor policy that converts the intent model's predictions into concrete robot actions. Because the intent interface — the representation of what should happen next — is shared across all embodiments, the same intent model can be paired with different sensorimotor policies for different robot platforms.
The researchers validated this cross-embodiment capability by deploying LUCID on two very different robot types: a dexterous five-fingered robotic hand and a simple parallel-jaw gripper. Both platforms used the same trained intent model; only the sensorimotor policy was swapped. This is the kind of architectural reuse that has made large language models so effective — a single "brain" trained on diverse data, adapted to specific tasks through smaller, specialized components.
Real-World Validation: Five Tasks, Minimal Supervision
The LUCID team evaluated their framework on five real-world manipulation tasks, and the results demonstrate the power of their approach:
- Stirring, wiping, and binning — These three tasks were trained using only internet video data. The robot was shown no robot demonstrations and no structured human teleoperation data. Despite this minimal supervision, LUCID achieved zero-shot transfer to novel scenes and object instances it had never encountered during training.
- Push-T and cable routing — These two more complex tasks were supervised by just one hour each of self-collected smartphone video. A single hour of casual video was enough for the system to learn the manipulation skill and generalize it to new configurations.
The primary test platform was a dexterous robotic hand developed at CMU, capable of complex in-hand manipulation. The successful transfer to a parallel-jaw gripper — a mechanically much simpler device — confirmed that the embodiment-agnostic design works in practice, not just in theory.
Why This Matters for Robotics
LUCID addresses what is arguably the most fundamental bottleneck in robot learning today: the data problem. Internet-scale video is abundant, diverse, and free. If robots can learn from it effectively, the path to general-purpose manipulation becomes significantly more practical.
Current robot learning pipelines rely on one of two expensive sources: robot demonstrations (which require access to the physical robot and cost thousands per hour in researcher time) or structured human data collection (which requires controlled environments and specialized equipment). Both approaches produce data that is tied to specific embodiments — meaning the data collected for one robot cannot easily transfer to another.
LUCID breaks both constraints simultaneously. By learning intent from unstructured human videos, it taps into a virtually unlimited data source. By making the intent model embodiment-agnostic, it decouples the data collection from the target hardware. A single intent model trained on millions of human manipulation videos could, in principle, power everything from factory robot arms to medical surgical assistants to home service robots.
Technical Architecture: How LUCID Works Under the Hood
The intent model in LUCID is trained using a contrastive learning objective. Given a current observation (an image or video frame), the model learns to predict a representation of the next desired scene state. This representation is not a pixel-level prediction — it is a compact latent vector that captures the essential properties of the manipulation: which objects should be in contact, their relative positions, and the expected physical outcome.
The sensorimotor policy, meanwhile, is trained in massively-parallel simulation. This is a crucial design choice: training the motor control component in simulation allows the system to explore millions of variations of how to achieve the intended scene change, without the time and safety constraints of real-world trial-and-error. Once trained, the policy transfers to the real world with the intent model providing the closed-loop guidance.
This training strategy — internet-scale intent learning plus massively-parallel simulation for control — is computationally intensive during training but highly efficient at inference time. The intent model runs at near-real-time speeds, and the sensorimotor policy is a lightweight feedforward network that can execute at control frequencies suitable for real-time manipulation.
Limitations and the Road Ahead
The current LUCID framework is an important proof of concept, but it has clear limitations that the researchers acknowledge:
- Task scope: LUCID was evaluated on five manipulation tasks. Scaling to hundreds or thousands of diverse tasks will require larger intent models trained on even more diverse video data.
- Long-horizon planning: The system predicts short-horizon intent — what should happen next — rather than reasoning through a complete multi-step task plan. For complex assembly or multi-stage manipulation tasks, a higher-level planning layer would be needed on top.
- Generalization boundaries: While zero-shot transfer to novel object instances and scenes was demonstrated, the system's ability to handle truly out-of-distribution scenarios — objects it has never seen in any video — remains an open question.
The project page and code are publicly available, and the authors plan to continue scaling LUCID to more complex and diverse manipulation scenarios. The combination of internet-scale video learning with embodiment-agnostic intent modeling represents a promising direction for breaking the data bottleneck that has constrained robot learning for years.
As the robotics community grapples with the fundamental question of how to build general-purpose manipulation systems, LUCID offers a compelling answer: learn what to do from the vast library of human video already available, and figure out how to do it through simulation. Separating these two challenges may be the key that finally unlocks scalable robot learning.

