An elephant in a refrigerator with a filmstrip

Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding

Shuo GaoRenmin University of China

Chenhao ZhengUniversity of Washington

Zhongzheng RenUNC Chapel Hill

NeurIPS 2026

arXiv

Paper

Code

Hugging Face

ModelScope

Abstract

Hour-long videos, streaming feeds, and movie-length content are pushing vision-language models (VLMs) past their breaking point—the standard recipe of encoding each frame into visual tokens yields sequences that grow linearly with video length, causing prohibitive memory consumption and inference latency. Existing approaches, such as token reduction and keyframe selection, attempt to shorten the token sequence but leave the O(n) scaling intact, failing to resolve the fundamental bottleneck. We propose FramePack-U, a strikingly simple, training-free frame packing framework that compresses arbitrarily long videos into a fixed-size visual representation at constant memory cost. FramePack-U requires no fine-tuning or architectural modification — it plugs directly into off-the-shelf VLMs at inference time, yet consistently surpasses more elaborate baselines. Drawing inspiration from FramePack in generation literature, we propose a new content-aware compression strategy that operates under a fixed token budget for visual understanding. Specifically, FramePack-U proceeds in two steps: (1) scoring each frame by its semantic similarity to the text prompt and ranking frames in a diversity-driven manner, and (2) packing the selected frames at varying resolutions into a fixed-length context window. This design concentrates the memory budget on frames that are both instruction-relevant and visually diverse. While in theory FramePack-U can pack arbitrarily many frames into the fixed budget, we identify an empirical instantiation that generalizes robustly across different settings. Extensive experiments on LongVideoBench, MLVU, and Video-MME show that FramePack-U consistently outperforms full-resolution baselines as well as prior token reduction and keyframe selection methods, across diverse model families, parameter scales, and frame budgets. Notably, on Qwen3.5-35B-A3B with a 128-frame budget, FramePack-U yields gains of +12.9%, +10.6%, and +9.7% over the official model on the three benchmarks, respectively.

Method

FramePack-U is a training-free frame compression and packing method that assigns heterogeneous spatial resolutions under a fixed visual token budget. It operates in two steps: relevance- and diversity-aware sampling ranks frames by instruction relevance and visual diversity; hierarchical resolution packing preserves top-ranked frames at finer resolutions while progressively downsampling lower-ranked frames. This concentrates the memory budget on diverse, instruction-relevant evidence and plugs directly into off-the-shelf VLMs without fine-tuning or architectural modification.

FramePack-U framework: CLIP query relevance scoring, diversity-aware greedy sampling, frame ranking, and hierarchical resolution packing before VLM inference — **Method overview.** Relevance identifies useful evidence; diversity avoids redundant frames; hierarchical packing allocates spatial detail under a fixed budget.

Results

Across LongVideoBench, MLVU, and Video-MME, FramePack-U consistently outperforms training-free compression baselines on both Qwen backbones under the same visual memory budgets. Our top results are competitive with, and often surpass, those of much larger proprietary VLMs.

Backbone

Memory budget

Method	LongVideoBench	MLVU	Video-MME

Runtime Analysis

We evaluate preprocessing, VLM inference, and overall memory and latency across video durations on LongVideoBench, using Qwen3.5-9B with 1 FPS candidate frames on a single NVIDIA A800.

Memory overhead by video duration for preprocessing, VLM inference, and the overall pipeline

Preprocessing, VLM inference, and overall latency across video-duration groups

Fixed visual tokens do not eliminate video decoding and frame-scoring costs. End-to-end latency also depends on candidate-frame loading and the backbone; see the paper for the full runtime analysis.

Qualitative Results

Relevance-only sampling often selects redundant frames, while uniform sampling may miss question-relevant evidence. FramePack-U combines prompt relevance and visual diversity to retain key evidence with broader temporal coverage.

Relevance-only, uniform, and FramePack-U sampling comparison: after the phone call

Relevance-only, uniform, and FramePack-U sampling comparison: the opening scene

t-SNE Visualization

Under the same 128-frame memory budget, FramePack-U packs 848 frames for denser feature-space coverage, prioritizing query-relevant evidence at higher resolution. Uniform sampling retains 128 full-resolution frames.

t-SNE comparison for animal in the snow: FramePack-U and uniform sampling at equal memory

t-SNE comparison for clothing color: FramePack-U and uniform sampling at equal memory

t-SNE comparison for soldiers in the scene: FramePack-U and uniform sampling at equal memory

Limitation & Discussion

Model capacity. Smaller VLMs may struggle to integrate the additional visual context, leading to unstable gains under larger memory budgets. More frames can introduce distraction rather than useful evidence.

Architecture compatibility. FramePack-U naturally fits VLMs that process frames independently. Grid-based or jointly encoded video inputs may require changes to the visual encoder and preprocessing pipeline.

Relevance and diversity scoring. CLIP may miss fine-grained temporal events and reasoning-heavy cues, while visual similarity may not fully capture semantic complementarity. Stronger, temporally aware scoring is a promising direction.

Packing scheme selection. The best balance between temporal coverage and spatial fidelity varies across models, tasks, and video domains. Automatically adapting the scheme to video length, query type, and model capacity could improve robustness.

Citation

If you find FramePack-U useful in your research, please consider citing our paper. 🫶

@inproceedings{gao2026elephant,
    title = {Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding},
    author = {Gao, Shuo and Zheng, Chenhao and Ren, Zhongzheng},
    booktitle = {Neural Information Processing Systems (NeurIPS)},
    year = {2026}
}