Data Filtering for Generative Video Pre-training

Image and video models have gotten a lot better over the last few years, even though the internals of these models haven't changed much since Stable Diffusion 3.Of course, there have been small variants like the auto-regressive diffusion that GPT-Image popularized. But at a high level, it's pretty much all flow matching with a transformer backbone and a v-prediction objective. In our experience, most of the gains are directly attributable to 3 flavors of data improvements:RL has driven some improvement, but it's only started truly working for image and video in the past few months.

1. Data Filtering & Rebalancing: Remove noisy data and resample your data strategically so your model learns more effectively
2. Data Annotation: Gather better annotations like richer captions, bounding boxes, and font details so that it's easier for your model to disambiguate visual conceptsLLMs have gotten infinitely better at image captioning in the last 12 months. Less so for video, but that's for another time.
3. Synthetic Data Generation: Finetune an ensemble of existing generative models to create training data for which there is little-to-no naturally occurring data (e.g. image editing / reference-conditioning for Nano-Banana style models)Often a collection of brittle LoRAs trained for very specific tasks and an LLM finetuned to filter out bad synthetic data.

A couple of years ago, the prevailing wisdom across all generative models (be it text, image, audio) was to aggregate as much data as humanly possible for pre-training. Luckily, the field has gotten a lot smarter about this. If you throw a bunch of low-quality data (e.g. heavily compressed JPEGs) into pre-training, your model is going to waste a significant amount of its capacity learning how to mimic this slice of data. If you filter your dataset well, your model will have a lot easier time learning what you want it to learn.

We know this sounds obvious, but it's a lot harder to do in practice.Then again, all good advice should seem obvious in retrospect.

Today we're going to walk you through how our approach to data filtering has evolved since 2024. And, hopefully we'll save you from a couple of headaches if you end up training your own generative models down the line.

[2024] Filtering on a budget — Traditional CV on CPUs

On the first go around, we decided to push our raw dataset through old-school computer vision algorithms. This way we could get away with a cluster of cheap CPU instances instead of an unholy number of GPUs running a multimodal LLM.Or, spending several million in GPT-4 tokens.

Scene detection

We need to filter down tens of billions of images and videos to create our pre-training dataset. Images don't really require any specific pre-processing, but raw videos do.When you train a generative video model, you first need to pre-train on image generation. If the model learns nouns before it learns verbs, it tends to converge better (and faster).

Next time you watch a television show or movie, track how often the camera cuts. If you're watching something made in the last twenty years, more likely than not you'll see a cut every 5 seconds. When to cut and how to cut is an authorial decision, not something a generative video model should do arbitrarily. So, we need to slice n' dice our videos on shot boundaries into video clips before we can filter them down.

With our cheapskate CPU-only agenda, we picked up PySceneDetect. At a high level it maintains a rolling window of K-frames and if the K+1 frame has significantly different image statistics, it categorizes the frame as a cut. There's no underlying machine learning model. It runs really fast but struggles with common transitions like dissolves, fades, and jitter cuts (which low key is a huge issue).

Getting to know your data

Whenever you get new data, you should spend a few days reviewing random samples, listing what you'd like to keep and what you'd like to throw out. Ideally, you take the time to draft an ontology of categories within "good" and "bad" and track the relative sizes of these categories.

At some point during the data filtering process, your engineer brain will take over, and you'll spend way too much time tuning the knobs of your heuristics (or LLMs), chasing that "perfect" decision boundary. These notes are going to save you from yourself down the line. They'll give you the facts you'll need to talk yourself out of trying "one more idea", when the answer is clearly "no".

Plus, understanding the shape of the data distribution will really help with dataset rebalancing. Certain categories are overrepresented in the natural distribution of all videos. We need to subsample and suppress this signal, otherwise it will dominate training and our model will struggle to learn the long-tail of people/places/things/actions that we need in order to generate anything.

Sieving out the un-captionable

Generative video models are primarily limited by what we can describe correctly and consistently in words.Or more precisely, what LLMs can annotate for us cheaply. Text provides a pretty good scaffold to understand the visual world, but it's by no means the correct conditioning mechanism for all aspects of video generation. Details like camera trajectories in space-time and the nuances of an actor's performance are simply indescribable in natural language.Reference-driven video generation is currently in-vogue. It's the first articulation of a control for these ineffable properties of videos, but we don't think it's a very good control per se.It's sufficient for autogenerated and remixed content like memes and brainrot, but it doesn't allow for the level of fine-grained direction necessary to convey authorial intent. It doesn't enable us to create the compelling stories that this technology should herald.

For now, we need to filter out clips where the primary "thing" that makes the video clip interesting is un-captionable. Without a crystal clear text description, it's just noise to our text-to-video model.

Text-heavy

For example, we want to filter out text-heavy videos. It's still hard for LLMs to caption motion graphics that are constantly changing on screen.Reading the screen isn't the hard part, it's establishing a clear description of when and how the text changes on screen. We don't want to waste capacity in our 2B parameter model learning motion graphics when it could be allocated instead to learning actions.Zooming out, our goal is creative tools for animation. Text and motion graphics aren't a priority right now.

To do this, we sampled frames from each video and ran a tiny EAST Detector to extract bounding boxes for text. From there, we filtered out text heavy videos based on the percentage of the frames that had text and the percentage of each frame covered in text.

Using a CNN for this task was a good idea, but the specific choice was wrong. In order to run tens of billions of frames on CPUs, we had to resize the frames aggressively. So, a lot of text-heavy samples with small fonts fell through the cracks.

EAST is a pretty old model from 2017. It's small and far from the state of the art on text detection. Getting it to run efficiently on CPUs without cache-thrash and thread oversubscription was a challenge. Even after performance optimizations, it was still the largest bottleneck for this version of the data pipeline.It took us weeks to run EAST on our dataset, across a massive cluster of CPUs.

Indescribable actions

When there's not much happening on the screen (e.g. close-up on a person's face), it's hard to describe the specific action taking place. If there's too much happening (e.g. extremely shaky camera, a soccer match with a bunch of folks moving across the pitch at once), LLMs struggle to caption the clip correctly. We lumped these categories of videos together as "indescribable action" clips to be thrown out.

Videos are typically serialized on disk in a compressed format. Codecs like H.264 reduce file size by storing keyframes and motion vectors that describe how the keyframes change over time, rather than RGB values for each pixel over time.We're eliding a bunch of specifics here. In H.264, the encoder computes the residual between two frames (the frame difference) and then applies the DCT (Discrete Cosine Transform) to the residual. This way we get a heavily quantized motion vector that gives us lossy frame reconstructions in exchange for a smaller file size on disk.

We used the motion vectors stored within the mp4 files themselves to isolate and filter out the "indescribable action" videos.In 2024, we also tried running traditional optical flow algorithms like Lucas-Kanade on CPUs. They're extremely brittle and didn't transfer well to open world video. Specifically, we used mv-extractor and computed two heuristics per clip:

• average_frame_energy: L2-norm of all motion vectors averaged across the video
• min(sub_clip_average_frame_energy): Split each clip into a variable number of chunks depending on the video's length, calculate average frame energy for each chunk, and take the minimum across these L2-norms

Then came the decision tree:

• average_frame_energy < 0.1: Throw the clip away. These were essentially static videos (e.g. slideshows, still frames, freeze-frames).
• average_frame_energy > 25: Throw the clip away. The footage was incredibly chaotic.
• min(sub_clip_average_frame_energy) < 0.03: Throw the clip away. A portion of the clip has nothing happening (e.g. a fade, transition to a still image in a documentary).
• Keep everything else.

This works well as a cheap first filter, but it has mediocre recall (i.e., a lot of indescribable action clips are kept in the dataset).

Subsampling talking head video clips

From our initial review of the raw data distribution, it was pretty obvious that talking head clips where folks talk straight to camera were dramatically over-represented. If we let the dataset be, it would have been significantly biased towards this sort of clip. Our model would get disproportionately good at creating them (likely at the expense of others), so we needed to find them and subsample them.Remember the VEO-3 videos that went viral in early 2025? A LOT of them were "man on the street" videos. The fact that it could generate these so much better than other styles of videos is indicative of a significant bias in the underlying training dataset.

Living in an old-school CV world, we naturally burrowed deeper down the engineering tunnel and introduced additional heuristics. We sampled frames from each video clip and ran a Haar-cascade face detector to extract bounding boxes for faces and calculated two numbers:Haar-like features typically fail to detect faces that are positioned at an angle relative to the camera. For us, that was a good thing. This way we could separate true talking head videos from videos of people taking actual actions.

• average_face_frame_energy: L2-norm of motion vectors within face bounding boxes
• average_background_frame_energy: L2-norm of motion vectors, just in the corners of the frame (as a proxy for background motion)

And from there, another decision tree:

• Moderate average_frame_energy + average_face_frame_energy >= average_background_frame_energy: Keep the clip. Usually a really good close-up.
• Low/Moderate average_frame_energy: Subsample these.

[Early 2025] rm -rf — Replacing hand-crafted heuristics with finetuned LLMs

We're starting to sketch a rather complicated decision tree. It's full of lossy proxies that only kind of work, and it's very incomplete.We have omitted a bunch of the categories we tried to filter. For example, clips with the Ken Burns Effect. Plus, we haven't opened the can of worms that is quality-based filtering (e.g. identify and throw out pixelated content).

This approach simply doesn't scale. Every time you have a new idea for a filter you have to re-examine how the new node in the decision tree impacts all the other branches. Everything is intertwined and eventually you end up with a pipeline that's both un-interpretable and uneditable.

When we started in 2024, we were staring down the barrel of tens of billions of samples. Given our limited budget, our gut was to construct the cheapest filters possible. This was fundamentally wrong. Our video model struggled to learn basic motions like guitar strumming after training for several weeks on our 2024 dataset but was able to learn these very actions in less than 24 hours of training, after applying our 2025 filters. Instead of looking for the cheapest filters possible, you should optimize for the best possible filters you can afford.

Migrating to 1000s of GPUs

This brings us to our second takeaway: throw away your "principled" computer vision techniques and adopt black box neural networks wherever you can. There are patterns that humans simply can't describe well, no matter how hard we try. Old school CV methods were the best hand-crafted approximations of their era. They're truly impressive feats of engineering, but a well-trained neural network will learn a non-linear function that will win on precision and recall in 99% of cases.

Once you re-orient yourself around this reality, your job should shift from crafting cheap heuristics to optimizing models for GPU throughput and engineering resilient, parallelizable workloads to run on SPOT instances across providers.In his YC talk from summer 2025, Karpathy discusses a similar realization in his work at Tesla AutoPilot. He describes it as the transition from "software 1.0" to "software 2.0".

Concretely, we replaced PySceneDetect's heuristics with AutoShot and TransNetV2, accelerating inference with custom CUDA kernels.These models still struggle with capturing 100% of shots. For example, they still miss frame jitter cuts. Good enough for now. But, in the future we'll have to train these models with new data to fill in these gaps. We stopped running the ancient EAST Detector on CPUs. Instead, we deployed PaddleOCR with TensorRT across thousands of Nvidia A10Gs and L4s.It only took ~36 hours to rip through tens of billions of samples on GPUs. Less than $10K USD for faster, better results. And, we ripped out the tangled web of computer vision algorithms (Haar Face Detectors, Lucas-Kanade, monocular depth estimators, etc.), replacing them with a set of fine-tuned LLMs.We still use H.264 motion vectors as a cheap initial filter for extremely low and high motion clips.

Iterative dataset labeling (aka self-consistency is harder than you think)

Finetuning is pretty straightforward thanks to the folks at Unsloth.Serving these models is slightly more tricky. We use vLLM like everyone else. But, it's almost impossible to guarantee exact bit-wise compatibility between Unsloth and vLLM. Every time we've had to deploy with vLLM it's eaten up a week or so, tracing through vLLM code and modding the exact ops for image/video patchification (or fixing small bugs in vLLM) so that we can match the accuracy of the results we get out of the box from our LoRAs or fine-tuned LLMs. This means the labeling is the work.

The hard part of training LLMs for data filtration is that the categories are always somewhat fuzzy. You'll have to answer questions like:

• If a sample fits several categories to different degrees, which label do I assign it?
• Should I simplify my categories, so I can label the data more quickly and consistently? Or, do I need to split my category into pieces to make it clearer?
• How easy is this concept for the LLM to learn? How much data do I need for each category?

More often than not, the biggest problem you'll run into is one of self-consistency. Over the course of labeling a couple hundred samples, it's only natural that you'll relax your criteria, mislabel samples, and muddy the signal in your dataset.

Dataset construction & LLM fine-tuning form one continuous process

1. 1Define your categories

   Write clear definitions. Be as specific as humanly possible. You should already have a draft ontology from your dataset study.

2. 2Label ~2K samples

   Draw random samples from your dataset and label them. Revise categories as you see fit.

3. 3Split train and validation

   Hold out a validation set. You'll use it once at the very end to make sure your model generalizes. Don't use it to steer the iterative labeling.

repeat k times

4. 4Finetune the LLM

   LoRA or fully-finetune. We used Qwen-2-VL-2B for our initial filters; smaller models are sufficiently intelligent for these tasks.

5. 5Evaluate on the training set

   Find the categories the model struggled with the most. Either you need more data for the category or more likely than not, the category is poorly constructed and needs to be redefined. For aesthetic scoring, mislabeled samples are usually indicative of inconsistent grading on your part.

6. 6Refine categories and re-label

   Add, drop, or merge categories. Then label again. Stop when you can no longer induce a better decision boundary within your LLM.

In a CMU study from 2025, researchers hired cinematography experts to annotate camera motion in online video clips and train other laypeople to make similar annotations. Even with the criteria in hand, the experts disagreed with "ground truth" ~24% of the time. Only through repeated trials were they able to converge on 96% agreement. It's painful to spend days labeling and re-labeling a dataset, but them's the breaks. At least, we're lucky to live in an era where you only need 2-3K samples to train a good filter.And, not the early deep learning days where you'd need at least tens of thousands of samples to train a CNN from scratch.

If you find yourself working on data filtration, we'd recommend you hack together a simple labeling tool like the one we show below. We just slapped together a super simple React app with Supabase to store labels and R2 to store the samples.

Turning LLMs into categorical classifiers

Ultimately, we supervise-finetuned (SFT) Qwen-2-VL-2B to tag:

• Image Categories: Ugly Product Image, Diagram / Screenshot, Collage, Watermarked, Bad Lighting, Pixelated, Drawing / Illustration, Keep
• Video Categories: Animation, PoV, Bars, Motion Graphics, Ken Burns, Shaky Camera, Little to No Motion, Weird Transition, Keep

We only kept images that our filter predicted as Drawing / Illustration or Keep. For videos, we retained Animation and Keep clips wholesale, while subsampling PoV.

[Late 2025] Reinforcement Learning with Verifiable Rewards (RLVR) for aesthetic filtering

At the start of the data labeling process, we tried to get extremely specific about the properties of the images and videos that divvied up samples into ugly vs. pretty (e.g. overexposed lighting, muted color grades). We thought it would be easier for the LLM to learn the precise reasons why we considered an image ugly than learn an arbitrary "ugliness score".

Once again, our initial intuition turned out to be wrong. The properties that make a particular sample ugly tend to be correlated; you end up assigning K different aesthetic tags to the same sample. And in turn, this poses two significant challenges:

• Sparse Data Signal: The combinatorial explosion of tags makes it harder for the model to disentangle the categories, especially with a small dataset of a few thousand labels.
• Slow, Inconsistent Labeling: It takes a lot longer to label samples (and it's a lot harder to be self-consistent) when you have the cognitive load of weighing several possible tags per sample.

We ended up grading the samples on a scale from 1 to 4 and keeping the samples that our models labeled 3 or 4. This isn't an original idea by any means. LAION Aesthetic Predictor was used to filter data for the original Stable Diffusion. That model took CLIP embeddings and trained a small linear layer to predict an aesthetic score. While useful, it was pretty brittle. If you dig into the predictions, you'll find an abundance of weird ratings. For example, specific types of watercolor illustrations all get a very high score regardless of noticeable JPEG degradation.

Fine-grained aesthetic scorers

Hang with us, as we work through a short history lesson.

The primary way that folks traditionally finetuned LLMs is supervised-finetuning (SFT).Either you can update all the parameters in the network, or train a LoRA where you insert a small number of parameters throughout the network and only train this new set of parameters. We used this approach when training our categorical classifiers. We fed Qwen-2-VL-2B triplets of (image/video frames, prompt Categorize this as A, B, or C, true label) and used the same objective from pre-training (next-word-prediction-with-cross-entropy) to update the model. With SFT, you're essentially extending pre-training, so your model learns your new task.

At the start of 2025, DeepSeek popularized a different finetuning technique called "Reinforcement Learning with Verifiable Rewards" (RLVR) in their R1 paper. Instead of using next-word prediction, they used reinforcement learning (RL) where the model's response is graded for accuracy using a rubric.

It turns out that if your LLM is able to solve a problem even 1 in 1,000 times, we can nudge the network towards finding this solution more consistently. These network updates are much smaller than those provided during SFT, so RLVR allows us to exert much finer grained control on extending the LLM to our tasks. Bluntly put, SFT is model surgery with a butcher knife, while RLVR is model surgery with a scalpel.

By the time we had our aesthetic dataset ready to go, RLVR was gaining adoption by the major labs; so we gave it a go for our aesthetic scorers and it smoked its SFT counterparts.If you look at all the practical progress made in LLMs in the past 18 months (Claude Code, cybersecurity, etc.), you can directly attribute the gains to RLVR. Math and coding rely heavily on reasoning to search the solution space effectively, before issuing an answer. Previously, OpenAI's o-series models were bottlenecked by the amount of data they could gather from humans. With RLVR, a lot less human data is necessary. The rubric stands in for tens of thousands of human-written reasoning traces. It's such an effective method, that some models are able to learn how to reason with zero human traces. Our prompt explicated the different reasons why we'd grade a sample from 1-4, while our rubric penalized the model on the absolute difference between the ground truth and predicted labels. Specifically, we used Group Sequence Policy Optimization (GSPO) as the RLVR algorithm on Qwen-2.5-VL-3B.This is just a small tweak on the GRPO (Group Relative Policy Optimization) algorithm that DeepSeek proposed in their R1 paper.

Filtering end-to-end

After RLVR, we adopted WAFT (a SOTA optical flow predictor) to filter out even more of the long tail of low-motion videos. Instead of relying on heuristics like Haar-cascades, we converted our video captions into tags and used these to edit our training data distribution (e.g., subsample talking heads, oversample animal videos).

Nowadays, there is a lot of jargon being tossed around the internet when it comes to "pre-training", "mid-training", and "post-training". The existence of distinct pre-training and mid-training phases is just a function of the impossibility of creating the perfect filter.

During pre-training, folks are more permissive. They're open to letting some low-quality data into the stack, so that they can guarantee that they cover all the necessary modes of their target data distribution. And by the time they get to mid-training, they're willing to tighten the distribution, even if it means throwing away some good stuff.

Regardless, data filtering is the single biggest lever you can pull outside of scaling to improve the quality of your model. If you take one thing away from this journey, never skimp on your data filtering pipeline. These things are what they eat after all.

Who are we?

We're two brothers training text-to-video models from scratch, trying to make animation accessible to everyone. Stay tuned for more blogs on captioning, synthetic data generation, and our ongoing work exploring better manifolds for pixel-space generative models.

Get Field Notes

Technical deep dives on building generative video models from the ground up, plus updates on new releases from Linum.

Subscribe