What are world models in AI, and why are they important?

"World models" or "world simulators" in artificial intelligence are viewed by many experts as the next major breakthrough. One indicator of this is the startup World Labs, founded by AI pioneer Fei-Fei Li, which raised $230 million to develop “large world models,” while DeepMind has brought on Sora, one of the developers behind OpenAI's video generator, to work on “world simulators.”

So, what exactly are these world models, and why are they drawing so much interest?

World models are inspired by the concept of mental models, which people naturally develop. Our brain takes information from our senses and transforms it into abstract representations, allowing us to interact meaningfully with our environment. These "models" are used by the brain to make predictions, determining how we perceive and react to events around us.

As researchers David Ha and Jürgen Schmidhuber explain, in baseball, for example, a batter has only fractions of a second to respond to a pitch and decide how to hit it. This moment is too brief for the signal to reach the brain, yet thanks to their “internal world model,” the batter can instinctively predict the ball’s trajectory and timing.

Researchers believe that these intuitive, subconscious processes based on world models are key to bringing artificial intelligence closer to human-level understanding.

World Modeling and Its Significance

The concept of world models has been around for decades, but it’s now experiencing renewed interest due to advancements in generative video. One main challenge in modern AI models is that they often don’t understand the cause-and-effect relationships of actions, even if they can predict them. For example, a model may predict that a basketball will bounce off the floor but doesn’t understand why this happens.

Unlike this, a world model with a basic understanding of why the ball bounces can realistically simulate such processes. These models learn from diverse data — images, videos, texts, and sounds — to better model events, causes, and consequences.

Applications and Prospects

World models offer vast potential for application. For example, Meta’s Chief AI Scientist Yann LeCun predicts that, over time, these models will enable complex forecasting and planning. A model that can perceive a current situation (e.g., a messy room) and a goal (a clean room) could come up with a plan of action to achieve this goal — cleaning, taking out the trash, washing dishes — based not on a pre-set schema but on a deep understanding of the necessary sequence to move from the initial to the final state.

As LeCun pointed out, creating such world models will require at least another decade. However, even today, these models show promising results, particularly in simulating basic physical processes and generating more realistic, consistent video.

Limitations and Challenges

Building and training world models requires tremendous computational resources, far exceeding those of current generative AI models. World models are also prone to errors and biases inherent in the training data. For example, a model trained primarily on images of sunny cities may struggle with depicting snowy landscapes or Korean cities.

Overcoming these limitations will require that training data be as diverse as possible and cover many different scenarios. Models must also be able to create interconnected maps of their surroundings and interact with them, as noted by Runway’s CEO, Cristóbal Valenzuela.

Future Potential

If world models can overcome today’s challenges, they could serve as a foundation for advancements in robotics and automated decision-making. Such models would enable AI to interact more reliably with the real world, potentially improving interactions within virtual and physical environments.

Today’s robots are limited in their capabilities because they lack comprehensive awareness of their surroundings. World models could help them gain this understanding, allowing them to interpret what’s happening around them more accurately and make better decisions.

Thus, world models in AI have enormous potential for creating sophisticated virtual worlds, improving robotics, and making substantial progress in understanding and simulating the physical world, leading to more human-like and intuitive artificial intelligence.