Imagine a world where you could not just watch a video, but actively participate in it, influencing events and seeing the direct consequences of your choices unfold. For years, our cutting-edge text-to-video models have been incredible at generating stunning, short clips from a simple prompt. They’re like digital filmmakers, crafting a perfect scene and then hitting “cut.” But what if we wanted the scene to keep going, to evolve with every action we took, maintaining a consistent, persistent reality? This has been a significant hurdle in AI, limiting our ability to create truly interactive and dynamic simulations.
Enter PAN: a groundbreaking new model introduced by researchers at MBZUAI’s Institute of Foundation Models. PAN isn’t just another video generator; it’s designed to be a general world model, capable of predicting future world states as video, all conditioned on past events and, crucially, natural language actions. This isn’t just watching a film; it’s stepping onto the set and directing the ongoing narrative, in real-time. It’s a leap from passive observation to active engagement, and the implications are truly mind-bending.
Beyond Simple Video Generation: Enter the Interactive World Model
The distinction between generating a video clip and simulating an interactive world might seem subtle, but it’s profound. Most current models are “fire-and-forget” – they create a sequence of frames and that’s it. They don’t retain an internal “world state” that adapts and persists over time as new actions are introduced. This is where PAN shines, redefining what a video model can be.
PAN is explicitly defined as a general, interactable, and long-horizon world model. Think of it this way: it holds a continuous, internal understanding of the “world” it’s simulating, represented as a latent state. When you, as an external agent, provide a natural language action—something as simple as “turn left and speed up” or “move the robot arm to the red block”—PAN doesn’t just generate a new, isolated clip. Instead, it updates its internal world state to reflect that action. Then, it decodes this updated state into a short video segment that visually depicts the consequence of your command.
This cycle is key. The same world state evolves across many steps, meaning the environment, objects, and agents within the simulation maintain coherence and continuity. This design allows PAN to support open-domain, action-conditioned simulation, enabling it to roll out countless “counterfactual futures” based on different action sequences. Imagine asking it, “What if I had turned right instead of left?” and seeing the alternative reality unfold. An external agent can query PAN like a sophisticated simulator, comparing predicted futures and making informed decisions based on those predictions. It’s a foundational step towards AI agents that can truly reason about and plan within dynamic environments.
The Engineering Marvel Behind PAN: GLP Architecture and Causal Swin DPM
So, how does PAN achieve this remarkable feat? The secret sauce lies in its robust architecture, particularly the Generative Latent Prediction (GLP) framework. GLP is designed to smartly separate the abstract “what happens” (world dynamics) from the concrete “how it looks” (visual rendering). This modularity is a stroke of genius, allowing specialized components to handle different aspects of the simulation with greater efficiency and accuracy.
At its base, a vision encoder takes raw images or video frames and distills them into a compact, latent world state. This is where the world’s current reality is captured in a form the AI can understand and manipulate. For PAN, this component is built on Qwen2.5-VL-7B-Instruct, a powerful vision-language model that tokenizes frames into structured embeddings, grounding the dynamics in both text and vision.
Next, an autoregressive latent dynamics backbone, also powered by a large language model (Qwen2.5-VL-7B-Instruct), predicts the *next* latent state. This prediction isn’t random; it’s meticulously conditioned on the history of previous world states and actions, along with learned query tokens. Essentially, this backbone is the “brain” that understands cause and effect, determining how the world will change given your command and its current situation.
Finally, a video diffusion decoder takes this predicted latent state and reconstructs the corresponding video segment. This is the “artist” of the system, bringing the abstract latent state back to life as a visually coherent and realistic video. PAN adapts Wan2.1-T2V-14B, a diffusion transformer renowned for high-fidelity video generation, for this task. It’s trained with a sophisticated flow matching objective, employing a thousand denoising steps to ensure pristine output. What’s clever is that this decoder conditions not only on the predicted latent world state but also on the natural language action itself, using dedicated cross-attention streams to ensure both elements are perfectly integrated into the visual outcome.
Stabilizing the Future: The Causal Swin DPM and Sliding Window Magic
One of the thorniest challenges in long-horizon video generation is maintaining consistency. Simply chaining together single-shot video models, conditioning each new frame only on the very last one, quickly leads to visual discontinuities and a rapid degradation in quality. It’s like trying to tell a continuous story by only remembering the very last word you said; things fall apart fast.
PAN tackles this head-on with an ingenious mechanism called Causal Swin DPM (Shift Window Denoising Process Model augmented with chunk-wise causal attention). Imagine a sliding temporal window that holds two chunks of video frames at different noise levels. As denoising progresses, one chunk moves from being highly noisy to a clean, coherent segment, and then it gracefully exits the window. Simultaneously, a new noisy chunk enters at the other end. Crucially, chunk-wise causal attention ensures that the later chunk can only “see” and attend to the earlier, already processed one, not to unseen future actions. This elegant solution guarantees smooth transitions between chunks and dramatically reduces the accumulation of errors over extended rollouts, keeping the simulated world stable and believable.
Furthermore, the research team implemented a subtle but impactful trick: adding controlled noise to the conditioning frame, rather than using a perfectly sharp one. This might sound counterintuitive, but it helps suppress incidental pixel details that don’t truly matter for the underlying dynamics. Instead, it encourages the model to focus on stable, meaningful structures like objects and their layout, contributing significantly to long-term stability and reducing visual jitter.
The Foundry of Intelligence: Training and Real-World Impact
Developing a model of PAN’s complexity isn’t a walk in the park; it requires immense computational power and meticulously curated data. The training process itself is a testament to sophisticated engineering, conducted in two distinct stages.
Building the Brain: A Two-Stage Training Process
In the initial stage, the research team adapted the Wan2.1 T2V 14B model into the Causal Swin DPM architecture. This stage alone was a monumental undertaking, involving training the decoder in BFloat16 with AdamW, a cosine schedule, gradient clipping, and advanced FlashAttention3 and FlexAttention kernels across a staggering 960 NVIDIA H200 GPUs. They used a flow matching objective, a cutting-edge technique for generative models, to refine the decoder’s ability to reconstruct video from latent states.
The second stage integrated the now-frozen Qwen2.5 VL 7B Instruct backbone with the fine-tuned video diffusion decoder under the GLP objective. Here, the vision-language model itself remained fixed, but the system learned crucial query embeddings and adapted the decoder so that predicted latent states and reconstructed videos remained perfectly consistent. This joint training, utilizing techniques like sequence parallelism and Ulysses-style attention sharding for handling long context sequences, ensures that PAN’s internal understanding of the world aligns flawlessly with its visual output.
Curating Reality: The Data That Powers PAN
A model is only as good as the data it learns from, and PAN is no exception. Its training corpus is derived from widely accessible public video sources, encompassing a rich diversity of everyday activities, human-object interactions, natural environments, and complex multi-agent scenarios. This breadth is crucial for a “general” world model. The raw data underwent a rigorous processing pipeline: long-form videos were segmented into coherent clips using shot boundary detection, and a sophisticated filtering system removed static or overly dynamic clips, those with low aesthetic quality, heavy text overlays, or screen recordings, employing rule-based metrics, pretrained detectors, and a custom VLM filter. Perhaps most importantly, these clips were then re-captioned with dense, temporally grounded descriptions that specifically emphasized motion and causal events—a critical step for teaching PAN action-conditioned, long-range dynamics rather than just isolated short clips.
Proving its Mettle: Benchmarking Performance
The true measure of any advanced AI model lies in its performance against rigorous benchmarks. PAN was evaluated along three crucial axes: action simulation fidelity, long-horizon forecast, and simulative reasoning and planning. It faced off against a formidable array of both open-source and commercial video generators and world models, including established names like WAN 2.1 and 2.2, Cosmos 1 and 2, V JEPA 2, and commercial titans such as KLING, MiniMax Hailuo, and Gen 3.
The results are compelling. For action simulation fidelity, where a VLM-based judge scored how well PAN executed language-specified actions while maintaining a stable background, PAN achieved 70.3% accuracy on agent simulation and 47% on environment simulation, leading to an impressive overall score of 58.6%. This places it at the highest fidelity among open-source models and even surpasses most commercial baselines.
In terms of long-horizon forecasting, measured by Transition Smoothness (quantifying motion smoothness across action boundaries) and Simulation Consistency (monitoring degradation over extended sequences), PAN truly excelled. It scored 53.6% on Transition Smoothness and 64.1% on Simulation Consistency, outperforming all baselines, including industry leaders like KLING and MiniMax. This is a clear indicator of its superior ability to maintain coherent, stable worlds over long periods.
Finally, for simulative reasoning and planning, PAN was integrated as an internal simulator within an OpenAI-o3 based agent loop. In this step-wise simulation, PAN achieved 56.1% accuracy, making it the best among open-source world models for this critical task. These benchmarks not only validate PAN’s technical prowess but also highlight its potential as a practical tool for building more intelligent and adaptive AI agents.
Conclusion
The introduction of PAN by MBZUAI researchers marks a pivotal moment in the evolution of AI. It’s more than just a step forward in video generation; it’s a foundational leap towards truly interactive, dynamic, and persistent AI-driven simulations. By meticulously operationalizing the Generative Latent Prediction architecture with robust components like Qwen2.5-VL-7B and Wan2.1-T2V-14B, and validating it through rigorous, reproducible benchmarks, PAN demonstrates how a vision-language backbone combined with a diffusion video decoder can function as a practical, general world model.
This isn’t merely academic curiosity; the implications are vast. From enabling more realistic and adaptable robotics to powering sophisticated virtual environments, scientific modeling, and ultimately, building AI agents that can learn, reason, and plan within complex, ever-changing digital worlds, PAN offers a glimpse into a future where our interaction with AI is not just observational but deeply experiential. As we continue to bridge the gap between AI understanding and real-world application, models like PAN are precisely what will pave the way for a new era of intelligent systems.
