Imagine you’re watching a movie, but instead of remembering the plot, you’re tasked with counting every blue car that appears across two hours of footage, even if they drive in and out of frame, or the camera cuts to a different scene and then back again. Sounds tedious, right? Now, imagine an AI trying to do that, not just with blue cars, but with countless objects, tracking their positions, sizes, and even predicting where they might go next in a constantly changing, dynamic environment.
For all their incredible advancements, today’s most powerful AI models, even those with “long context windows” designed to remember vast amounts of information, still stumble badly on tasks like these. They might ace a quick text summary or identify objects in a single image, but when it comes to truly understanding and reasoning about what’s happening over extended, messy video streams – tracking specific items, maintaining counts, or remembering spatial layouts across time – they often fall short. This isn’t just a minor glitch; it’s a fundamental limitation that points to a critical missing piece in the puzzle of truly intelligent AI. And that piece, it turns out, is something researchers are calling “Spatial Supersensing.”
The Achilles’ Heel of Current Multimodal AI: Long Video Understanding
We’ve celebrated the rise of multimodal AI, models that can see, hear, and understand language. But dig a little deeper, especially into video understanding, and you’ll find a surprising fragility. Most current video MLLMs (Multimodal Large Language Models) often resort to shortcuts. They might sample sparse frames from a video, relying heavily on language priors or captions to answer questions, rather than genuinely comprehending the continuous visual evidence unfolding before them. Diagnostic tests have even shown that many popular video benchmarks can be solved with surprisingly limited visual input – sometimes just text alone! This reveals a critical truth: these benchmarks aren’t truly stress-testing a model’s ability to sense and reason spatially.
Beyond Simple Context Windows
The prevailing wisdom for improving AI has often been “bigger is better” – more compute, larger context windows, more parameters. But the challenge of long video streams demonstrates this isn’t a universal solution. Even models like Gemini 2.5 Flash, lauded for their expansive context capabilities, degrade sharply when faced with tasks requiring continuous spatial reasoning. It’s not just about how much data an AI can “see” at once, but how intelligently it processes and remembers it. Imagine having an infinite scratchpad but no coherent system for organizing your notes – you’d still get lost.
Exposing the Gaps: The VSI Super Benchmark
To truly expose these limitations, a team of researchers from NYU and Stanford introduced the VSI Super benchmark. Think of it as an extreme stress test designed to break even frontier models. It features arbitrarily long indoor videos and two key challenges:
- VSI Super Recall (VSR): This is your classic “needle in a haystack” task, but stretched to an unprecedented degree. Human annotators take long indoor walkthrough videos (up to 240 minutes!) and subtly insert an unusual object, like a teddy bear, into four frames at different spatial locations. The AI’s job? Report the correct order of locations where the object appeared. It’s a grueling test of long-horizon spatial observation and sequential recall, where models must track visual information over immense time scales.
- VSI Super Count (VSC): This measures continuous counting under constantly changing viewpoints and scene transitions. The benchmark asks for the total number of instances of a target object across multiple rooms and revisits, over durations from 10 to 120 minutes. The AI must maintain a cumulative count, adjusting for objects moving in and out of view, and correctly identifying new instances versus revisited ones.
The results were sobering: a 7B parameter model, Cambrian-S, saw its VSR accuracy plummet from 38.3% at 10 minutes to a mere 6.0% at 60 minutes, hitting zero beyond that. VSC accuracy was near zero across all lengths. Crucially, even Gemini 2.5 Flash, with its vaunted long context, showed similar rapid degradation. This isn’t just about a harder benchmark; it reveals a structural weakness in current long-context multimodal architectures that rely on reactive perception rather than proactive spatial understanding.
Building a Foundation for True Spatial Cognition: Cambrian-S and VSI 590K
So, if brute-force context scaling isn’t the answer, what is? The researchers propose that the next competitive edge lies in models capable of “spatial supersensing,” a progression of capabilities beyond mere linguistic reasoning.
Defining Spatial Supersensing
Spatial supersensing isn’t a single skill but a hierarchy of advanced cognitive abilities:
- Semantic Perception: Understanding what objects are in a scene.
- Streaming Event Cognition: Recognizing and understanding events as they unfold over time.
- Implicit 3D Spatial Cognition: Building an internal, dynamic 3D map of the environment, reasoning about object locations and their relationships.
- Predictive World Modeling: Anticipating changes, predicting what comes next, and selectively remembering surprising or important events.
Current models mostly operate at the lower stages. Cambrian-S, the new model family introduced by the researchers, explicitly targets these higher stages, aiming to remember spatial layouts across time, reason about object locations and counts, and anticipate changes within a dynamic 3D world.
The Data Revolution: VSI 590K
You can’t train a spatial genius without spatial data. To address this, the team constructed VSI 590K, a massive spatially focused instruction corpus. This isn’t just more video; it’s *better* video. It comprises 5,963 videos, 44,858 images, and a staggering 590,667 question-answer pairs drawn from 10 diverse sources. These include richly annotated real indoor scans (like ScanNet and ARKitScenes), simulated scenes, and even pseudo-annotated web data like YouTube room tours.
What makes VSI 590K special is its focus on 12 spatial question types – covering everything from object count and distance to size and appearance order – with questions generated from true 3D annotations or reconstructions. This ensures that the spatial relationships are grounded in actual geometry, not just text-based heuristics that can often mislead. Training on this rich, diverse mix, especially the annotated real videos, showed the largest gains in spatial performance.
Cambrian-S: A New Breed of MLLM
The Cambrian-S model family, building on the Cambrian-1 architecture, leverages Qwen2.5 language backbones and a robust SigLIP2 vision encoder. Its four-stage training pipeline progressively refines its capabilities, culminating in a critical Stage 4: spatial video instruction tuning on a mixture of the VSI 590K dataset and other relevant data. This dedicated focus on spatial learning is key.
The results speak for themselves: Cambrian-S 7B achieved 67.5% accuracy on VSI Bench, outperforming open-source baselines and even proprietary models like Gemini 2.5 Pro by a significant margin. Crucially, this spatial specialization didn’t come at the cost of general capabilities; Cambrian-S maintained strong performance on other general video benchmarks like Perception Test and EgoSchema.
The Leap to Predictive Spatial Sensing: Beyond Reactive Understanding
Even with excellent spatial data and a strong MLLM like Cambrian-S, the “stress test” of VSI Super still highlighted the limitations of purely reactive perception. To truly master unbounded streaming video, models need to do more than just process what they see; they need to predict what’s coming next and intelligently manage their memory. This is where “predictive sensing” comes in.
The Power of Latent Frame Prediction and “Surprise”
The research team’s most innovative step is adding a Latent Frame Prediction head to Cambrian-S. This module predicts the latent representation of the *next* video frame in parallel with predicting the next language token. The magic happens when you compare this prediction to the actual next frame. The cosine distance between the predicted and actual features generates a “surprise score.”
And what do you do with surprise? You use it to drive an intelligent memory system:
- Memory Compression: Frames with low surprise – meaning the model accurately predicted what would happen – are compressed before being stored in long-term memory. They’re predictable, so less detail is needed.
- Detailed Retention: High surprise frames, on the other hand, indicate something unexpected or important has occurred. These are retained with more detail, ensuring critical events aren’t overlooked.
- Event Segmentation: For tasks like VSC, a high surprise frame can signal a scene change or a significant event, prompting the model to summarize accumulated features into a segment-level answer before resetting its buffer.
Real-World Impact: Stable Performance on VSI Super
This surprise-driven memory system dramatically changes the game. For VSR, Cambrian-S, empowered by predictive sensing, maintains its accuracy even as video length increases, all while keeping GPU memory usage stable. It significantly outperforms Gemini 1.5 and 2.5 Flash on VSR at all tested durations, avoiding the sharp degradation that plagues models relying solely on extended context. For VSC, this approach allows Cambrian-S to reach about 38% accuracy at 10 minutes and maintain around 28% at 120 minutes, vastly outperforming baselines like Gemini Live and GPT Realtime, which drop near zero on longer streams.
This isn’t just an incremental improvement; it’s a paradigm shift. It shows that by coupling spatial sensing with internal world modeling, rather than just scaling data and parameters, AI can begin to process and understand the continuous, dynamic information of the real world in a truly intelligent way.
The Future is Predictive, Not Just Reactive
The journey from simple video question answering to genuine spatial supersensing is a profound one. This research signals a clear direction for the future of multimodal AI: away from passive, reactive video understanding and towards active, predictive spatial cognition. It highlights that true intelligence in understanding the world isn’t about brute-force processing of every pixel, but about selectively attending to surprising and important events, building dynamic 3D models of our surroundings, and anticipating what comes next.
As AI moves towards more embodied and real-world applications – from robotics to augmented reality – the ability to continually observe, recall spatial layouts, count objects under changing conditions, and predict future states will become not just a desirable feature, but the core capability. Spatial supersensing, driven by intelligent memory and predictive objectives, is poised to unlock the next generation of truly perceptive and intelligent multimodal AI systems, transforming how they interact with and understand our complex, dynamic world.
