1 min read
In short
ReMem shows that the bottleneck in long-video QA is not model capacity, but rather the uniform sampling of frames. An adaptive approach without training yields a double-digit improvement on benchmarks.
The context window is not the main challenge for long-video QA. The main challenge is how models select frames from a long video. Uniform sampling and static selection overlook the temporal context that is critical for answering a specific question. ReMem is a training-free framework that bridges this gap through two-level memory adaptation and boosts LLaVA-Video’s accuracy on LVBench from 42.2% to 54.5%.
The approach operates on two levels. At the query level, the LLM memory analyzes the temporal granularity of the question and extracts semantic entities—that is, it understands exactly which segment of the video is relevant. At the video level, the structural memory aligns frames with the query’s semantics and dynamically allocates the sampling budget across event clusters. Instead of sampling frames at equal intervals, ReMem concentrates sampling where relevant action occurs.
Results on four LongVideoQA benchmarks using three MLLMs demonstrate state-of-the-art performance in zero-shot mode. LLaVA-Video with ReMem achieves 67.1% (+8.2%) on LongVideoBench. This improvement without retraining argues that the data sampling infrastructure is more important than raw model capacity. If your video processing pipeline still uses uniform sampling, this is the first thing you should reconsider.
Source: cs.AI updates on arXiv.org