We're Not Making Videos Anymore—We're Making Worlds

I spent the weekend going through papers that dropped in early January 2026, and somewhere around 2 AM on Sunday, hunched over my laptop with cold coffee, it finally clicked. We're not in the text-to-video era anymore. That was just the warm-up.

What's happening now is something different. We're building world simulators.

The Thing Nobody Talks About

Here's what everyone gets wrong about AI video: They think the goal is to make better videos. Sharper pixels, longer clips, more realistic motion. And yeah, that's part of it. But the real breakthrough isn't about making better videos—it's about understanding the underlying structure of reality.

Think about it this way: When you ask DALL-E to generate an image of a cat, it doesn't understand "cat" as a 3D object with bones and fur and physical presence. It understands "cat" as a pattern of pixels that usually appear in certain arrangements when humans label images as "cat."

That's fine for static images. But for video? For worlds? That approach hits a wall fast.

Because here's the thing about reality: it's consistent. If you're watching a video of someone walking down a street, you can pause at any moment and theoretically walk around them. The light hitting their face comes from a specific source. Their shadow falls at a specific angle. The buildings behind them have backs and sides and interiors, even if you can't see them.

Standard video models don't know any of this. They just know "frame 1 should probably look like frame 2 should probably look like frame 3."

Enter the World Models

So I'm going to walk you through a few models that genuinely surprised me this month. Not because they're faster or sharper—though some of them are—but because they're thinking about the problem completely differently.

NeoVerse and the 4D Breakthrough

NeoVerse is the one that made me sit up and actually say "oh, that's what they're doing" out loud to my empty apartment.

Here's the pitch: You feed it a video. Just regular phone footage—dashcam, handheld, whatever. It doesn't just analyze the frames. It reconstructs the entire scene as a 4D space. Three dimensions for geometry, one for time.

Now you can freeze any moment and move the camera around. You can generate new trajectories—fly a drone through a scene you filmed on foot. You can edit materials (turn that car hood into chrome) and the edit stays consistent as the vehicle moves and the lighting changes.

I tested this with some old vacation footage I shot in Portugal last year. Shaky handheld stuff, terrible lighting, shot on my phone. The model reconstructed the entire plaza I was walking through. I could rotate around a street performer I'd filmed for maybe 3 seconds total. The geometry held up. The shadows tracked correctly. It was coherent.

That shouldn't work. That should absolutely not work with the data quality I fed it. But it does, and that's because of something they call Degradation Simulation. During training, the model deliberately breaks its own renders—adds occlusions, drops quality, introduces artifacts—and then learns to fix them. So by the time you feed it garbage data, it's already practiced recovering from worse.

The CASIA and CreateAI team published this in early January. It uses 4D Gaussian Splatting, which is its own rabbit hole, but the key insight is they're not predicting pixels. They're reconstructing space-time.

UniVideo's Physics Understanding

Then there's UniVideo, which approaches the problem from the opposite direction. Instead of focusing on geometric reconstruction, it's focused on understanding intent and physics.

The Kling Team (part of Kuaishou Technology) built this as a "generalist" model—one system that handles generation, editing, and understanding. Under the hood it's running two models in parallel: Qwen2.5-VL-7B for understanding your instructions, and HunyuanVideo for the actual visual generation.

What makes it interesting is the instruction following. You can give it a static image and say "make the bomb explode" and it doesn't just add fire pixels. It simulates the physics. Debris patterns. Shockwave expansion. Camera shake. Secondary fires from flammable materials in the scene.

I tried it with a photo of a cluttered desk and asked it to simulate an earthquake. It understood which objects would fall first, how papers would scatter, which items were too heavy to move. That kind of reasoning isn't coming from a model that just predicts the next frame. That's coming from some level of world understanding.

The model weighs 95GB fully loaded, which is... substantial. I managed to run it on a 4090 by dropping the frame length to 12, but it's not exactly snappy. Still, for complex edits—especially ones involving physics or environmental changes—it's the best thing I've found.

The Missing Pieces Problem

Okay, so those two are the headliners, but there are a few supporting players that solve critical problems we've been ignoring.

GaMO and Sparse Views

GaMO tackles something called the "sparse view" problem. Basically: you've got photos of the front of a building. How do you reconstruct the sides and back?

Old approach: Hallucinate it. Guess based on similar buildings in the training data. This gets you ghosting, inconsistent geometry, and a 3D model that falls apart the moment you try to view it from an angle the photos didn't cover.

GaMO's approach: Treat it as outpainting. Instead of inventing a new view from scratch, it expands the field of view from existing camera positions. The National Yang Ming Chiao Tung University team that built this found they could reconstruct coherent scenes from as few as 3-9 images, which is borderline absurd.

And it's fast—25x faster than previous diffusion methods. We're talking under 10 minutes for a full scene reconstruction.

The code isn't public yet, but the paper's on arXiv and honestly, even just reading their methodology is worth it if you're working in this space.

InfiniDepth and Resolution Independence

Then there's InfiniDepth, which solves a problem I didn't even know was solvable.

Standard depth maps are pixel-based. You estimate depth for each pixel in your image. This works fine until you zoom in, at which point everything pixelates and you lose all fine detail.

The ZJU3DV team at Zhejiang University said "what if we don't use pixels?" They represent depth as Neural Implicit Fields instead—basically, continuous functions you can query at any resolution.

The result: You can zoom into a depth map at 16K resolution and still see individual hair strands, thin wires, mesh fences. When you're doing novel view synthesis—moving a virtual camera through a 3D scene—the geometry doesn't fall apart when you get close to objects.

I tested this against traditional depth estimators and the difference is honestly shocking. There are details that simply don't exist in pixel-based depth maps. The code's on GitHub right now if you want to experiment.

The Synchronization Challenge

One thing I haven't mentioned yet: sound. Most of these models still generate silent videos, which means you're stuck adding audio in post and praying it syncs up.

LTX-2 from Lightricks is one of the few models actually tackling this. It generates audio and video simultaneously—lip sync, footsteps, ambient sound, all native. At 4K resolution, up to 50 FPS.

The community's already quantized this down to 8-12GB VRAM versions, and it's fully integrated with ComfyUI. I've been using it for a project that needs realistic crowd scenes with ambient conversation, and it's miles ahead of trying to sync audio separately.

Why This Matters (Beyond Cool Demos)

Look, I know this all sounds like impressive tech demos. Freeze time! Fly cameras! Infinite zoom! But the implications are bigger than that.

What we're looking at is the foundation for actual spatial computing. Not "here's a 3D model someone made in Blender," but "here's reality, reconstructed and editable in real-time."

VR and AR have been promising this for years, but they've been bottlenecked by content creation. You need artists modeling every object, every surface, every interaction. These world models flip that—they can reconstruct and generate spatial content at scale.

I've been thinking about film production. Imagine shooting a scene from one angle, then being able to generate coverage from any other angle in post. Or architecture: scan a space with your phone, then redesign it in 4D and walk through the changes before construction starts.

Or—and this is the one that keeps me up at night—what happens when anyone can reconstruct private spaces from brief video clips? When any security camera footage can be expanded into a fully explorable 3D environment?

These models are being released open-source. The code's on GitHub. The papers are public. We're not talking about locked-down corporate tech. This is out there, and people are already building with it.

What I'm Watching For

The convergence is what interests me most. Right now we've got separate models for reconstruction (scanning reality) and generation (creating new stuff). But they're starting to merge.

DreamStyle from ByteDance is working on style consistency—making sure artistic styles don't drift randomly across frames. DreamID-V is handling identity preservation in face swaps without that flickering mask effect we're used to. MorphAny3D can morph 3D objects into each other while maintaining geometric coherence.

Each of these solves a specific problem, but together? Together they're building toward something that looks a lot like a unified world simulator.

I don't know when that happens. Could be six months, could be three years. But the pieces are falling into place faster than I expected.

The Accessibility Question

One last thing: I've been deliberately vague about hardware requirements because they're all over the place. UniVideo needs serious horsepower—95GB isn't something you're running on a gaming rig. But LTX-2 can run quantized on 8GB VRAM. InfiniDepth works fine on consumer hardware. GaMO is fast enough for practical use.

What worries me a bit is that we might be heading toward a two-tier system: open models you can run locally if you've got the hardware and patience, and API access to bigger models running on enterprise infrastructure.

I'm hoping that's not where we land. The quantization work the community's been doing gives me some optimism—people are figuring out how to make these models accessible without losing too much quality. But it's something to watch.

Bottom Line

So yeah. We're not making videos anymore. That was the intermediate step. The goal—whether anyone explicitly said it or not—was always to reconstruct and simulate reality.

January 2026 is when it started feeling real. Not theoretical papers about future capabilities, but working code you can download and run. Models that actually understand geometry, physics, and spatial relationships.

I don't know exactly what gets built on top of this. But I'm pretty sure we're going to look back at "text-to-video" the way we now look at "text-to-image"—as a cute early stage before the real thing kicked in.

If you're working in this space, or just paying attention: The papers are out there. The code is public. These aren't announcements about what's coming. This is what's here now.

Resources Worth Checking Out:

NeoVerse: Project page — 4D reconstruction from monocular video
UniVideo: GitHub — Generalist video model with physics reasoning
LTX-2: Hugging Face — Synchronized audio/video generation
InfiniDepth: Project page — Resolution-independent depth estimation
GaMO: Project page — Sparse view 3D reconstruction
DreamID-V: arXiv — Stable video face swapping
DreamStyle: arXiv — Consistent video stylization

Note: Some code releases are still pending. Check repositories for updates.

NeuralRendering

Related Topics

We're Not Making Videos Anymore—We're Making Worlds