World Models

The taxonomy

World model is one of the most overloaded terms in AI, since computer vision, robotics, reinforcement learning, and generative AI each claim to be building one and each means something different by it. Fei-Fei Li's taxonomy resolves the confusion by returning to the standard loop from reinforcement learning: an agent takes an action, that action changes the state of the world, the agent never observes the state directly but only receives an observation such as pixels or a sensor reading, and the new observation informs the next action.¹ What gets called a world model is really one of three different projections of that same loop.

A renderer outputs observations, essentially pixels for a human viewer, and is judged on visual fidelity rather than structural truth; this is the most commercially mature category, widely deployed already, but its outputs can look correct while being physically impossible, which means a renderer alone cannot design a building or train a robot. A simulator outputs state, the actual geometry, physics, and dynamics underneath, judged on structural accuracy, and serves two audiences simultaneously: human professionals who need real accuracy rather than plausibility, and downstream programs like a robot controller that need to interact with the resulting world at scale. This is the bridge between the other two categories, and the least publicly discussed despite being arguably the most consequential, even though Nvidia alone estimates more than one trillion dollars in addressable market for simulation across factories, warehouses, supply chains, and digital twins.¹ A planner does the reverse of a renderer, taking an observation and a goal as input and producing an action as output; this is the least mature of the three, with most robot demonstrations still confined to constrained lab settings.

The convergence thesis is that all three share a common underlying knowledge base, geometry, physics, and dynamics, so a model that genuinely understands how an object behaves should in principle be able to render it from any angle, simulate what happens when it is disturbed, and plan an action to manipulate it, meaning the three categories are ultimately three projections of one underlying understanding rather than three separate technologies.

A lay version: the consequence model

Mark Cuban gives a non-technical articulation of the same gap in the simulator category, framed around a diagnostic he calls consequence prediction: today's models are, in his words, basically bimodal, working across text and pictures with some video, and simply cannot model the world. "AI right now doesn't understand the consequences of its recommendations. It has no idea what happens next. A two-year-old kid... knows if it pushes the sippy cup off the high chair, mom's coming running. Large language models don't understand."² In the language of the taxonomy above, a toddler is running a forward model: the action of pushing produces a state change, the cup falls, which produces an observation, a parent reacting; a model trained only on correlations across text and images has learned patterns over observations without learning the underlying state-transition dynamics that make what happens next computable in advance.

Cuban adds a genuinely physical constraint the pure taxonomy leaves out: learning a world model from video at scale runs into a real bandwidth bottleneck, since much of that learning happens on infrastructure such as satellite and 5G networks that has to move and serve enormous amounts of video data, and he doubts there is currently enough bandwidth to do this broadly.² This is also the basis for his skepticism about general AI risk: without a working world model, capability stays tied to specific applications rather than becoming general, and it underlies his preference for narrowly co-designed robots over general-purpose humanoid robots.

The renderer-side builder's case

Alex Mashrabov of Higgsfield argues the renderer category is actually the highest-bandwidth path into the other two, the opposite vantage point from the simulator-first framing above. His core quantification: a single minute of video can be described with something on the order of 60,000 words, compared to roughly 100 to 150 words a person reads in a minute, meaning video carries orders of magnitude more latent structure per unit of time than text does.³ His claim is that video models are effectively forced to learn implicit geometry, physics, and dynamics as a side effect of getting better at video generation at all, whether or not that structure is ever explicitly exposed. He is notably careful about the larger claim, crediting Demis Hassabis and Elon Musk with starting the narrative that this path leads to general intelligence while calling it "still unclear," though he considers it "definitely a path to advanced robotic systems," citing Tesla's camera-only self-driving as proof that perception and visual understanding generalize to embodied control, and separately offering a deliberately measured timeline: no unexpected research breakthrough before the end of the decade, with the near future looking largely similar to the present.⁴

Why it matters

For physical agents, planners are the missing piece that would turn robots into reliable economic actors, extending agent-driven automation from software into the physical world once they mature. For anyone evaluating claims in this space, the taxonomy gives a way to ask a sharper question than whether a company is building a world model: which of the three functions, rendering, simulating, or planning, is actually being claimed, since the three have very different maturity levels and very different implications for what the resulting system can and cannot do. It also bears on the reliability of AI systems generally: several observed LLM failure modes, including the inability to anticipate the consequences of an action, read as symptoms of the same missing simulator layer described here.

Open questions

Whether scaling video generation induces genuine physical understanding or only better-looking output that remains physically impossible underneath is unresolved, and it is worth noting that a video-generation company has a direct commercial interest in video being the right path. Three-dimensional data annotated with real geometry and physics is far scarcer than the ordinary internet video renderers train on, which is a large part of why renderers are mature today and simulators are not, and even a good simulation can fail once applied to the real world, since generated geometry can look correct while containing errors invisible to a human eye that nonetheless produce nonsensical physics.