SIEVE: Preserving Cross-Image Evidence via Depth-Aware Token Compression

SIEVE is a training-free method for multi-image vision-language models that uses up to 16× fewer visual tokens. At one eighth of the tokens, it retains 93–103% of the uncompressed accuracy across four open vision-language backbones and, on LLaVA-OneVision-7B, reduces prefill compute by 6.6× and the KV cache by 6.4×. It requires no fine-tuning, labeled data, or architectural changes.

Nitish Shukla¹ Surgan Jandial^2,*,† Nhat-Tan Bui^3,* Arun Ross¹

¹Michigan State University · ²DatologyAI · ³Carnegie Mellon University
^*Equal contribution · ^†Work done at Carnegie Mellon University

Paper Code (coming soon) Results

up to 16×

fewer visual tokens

averaged over decoder depth

6.6×

less prefill compute

26.46 → 4.02 TFLOPs

6.4×

smaller KV cache

107.4 → 16.7 MB

1.9×

faster prefill

100.4 → 53.5 ms, one H100

training runs required

frozen weights; one forward pass

SIEVE at a glance — **Overview of SIEVE.** **(a)** **Top:** On COCO, attention to queried objects peaks in the latter half of decoder layers across backbones. **Bottom:** Fraction of global top-10% tokens assigned to the queried image (solid) and their overlap across questions (dashed), per backbone. Early rankings are largely query-agnostic, motivating per-image allocation and deferred attention-based selection. **(b)** Cumulative effect of each component at a 6.25% budget, averaged over five benchmarks. Pooling and budget scheduling provide most gains; progressive merging adds 2.27 points. **(c)** At a 12.5% budget, SIEVE leads on all five multi-image benchmarks with LLaVA-OneVision-7B, normalized to the uncompressed model (dashed ring: 100). **(d)** Three-image example: LLaVA-OneVision-7B answers correctly with only 6.25% of visual tokens. After per-image pooling, S(N) = 3 stages retain equal quotas per image and merge discarded tokens into their most similar retained tokens within the same image. Colors indicate the final token absorbing each region.

The challenge of multi-image compression

Vision-language models represent each image with hundreds of tokens, while the attention component of prefill compute scales quadratically with sequence length. A four-image question on LLaVA-OneVision contains 2,916 visual tokens before any text tokens are added. Applications such as document comparison, visual question answering over image galleries, and before-and-after inspection incur this cost on every request.

Token pruning reduces this cost, but when to prune is critical. Selection before the decoder lacks question-conditioned evidence; selection at later layers incurs the cost of processing the full sequence in earlier layers. Under a budget averaged over decoder depth, retaining all tokens through layer 2 already consumes 171% of a one-sixteenth budget. The initial layers alone therefore exceed the target.

Method overview

SIEVE combines three operations in a single forward pass with frozen model weights.

Spatial pooling before the decoder. Average pooling reduces each image's token grid, shortening the sequence processed by the early layers. Every patch contributes to the pooled representation.
Question-conditioned selection. At selected decoder depths, attention from the final prompt token ranks the remaining visual tokens. An equal quota for each image prevents a single image from consuming the entire budget and eliminating evidence from others, a failure mode of global token ranking on multi-image inputs.
Similarity-based merging. Each removed token is merged into its most similar retained token, preserving its contribution even when an early ranking is inaccurate.

Panel (d) above illustrates the pooling and selection stages on a real question. The number of selection stages adapts to the image count, S(N) = clip(N, 2, 4): two stages for one or two images, three for three, and four for four or more.

Interactive token visualization

Select a question to explore SIEVE across decoder depths. Each image begins with 729 visual tokens, is pooled to a 10×10 grid, and is further compressed at each merging stage. Darkened regions indicate tokens that are no longer represented separately. All examples are precomputed from saved forward passes of LLaVA-OneVision-7B at a 6.25% visual-token budget, the setting used for the paper's qualitative figures.

Show prompt attention Darken merged regions 78%

Quantitative results

We evaluate SIEVE on four backbones and eleven benchmarks: five multi-image, four single-image, and two video. One configuration is used for all backbones, budgets, and benchmarks, with no benchmark-specific tuning.

Accuracy retained at one eighth of the visual tokens

percentage of each backbone's uncompressed five-benchmark average

Accuracy against prefill compute — **Accuracy vs. compute** on LLaVA-OneVision at 1/16, 1/8 and 1/4 of the tokens (five-benchmark average). SIEVE has the highest accuracy at every compute level and, at 1/8 and above, exceeds the uncompressed model (dashed, 63.12). x-axis: estimated visual-token prefill TFLOPs.

SIEVE retains 87–99% of each backbone's uncompressed mean at one sixteenth of the tokens and 93–103% at one eighth. On LLaVA-OneVision it exceeds the uncompressed model at one eighth and one quarter (65.13 and 66.64 vs. 63.12). At one sixteenth it has the highest mean on all four backbones, 9.96 points above the strongest baseline on LLaVA-OneVision and 1.72–2.54 on Qwen2.5-VL and Qwen3-VL. At one eighth it leads on three of the four backbones, by 0.76 to 9.03 points, and trails ToDRE by 1.52 points on InternVL3.

Method	LLaVA-OneVision-7B	Qwen2.5-VL-7B	Qwen3-VL-8B	InternVL3-8B
No compression	63.12	70.85	67.91	68.28
One sixteenth of the visual tokens
VisionZip	51.04	58.25	54.78	58.39
FastV	39.48	41.06	41.44	39.07
PyramidDrop	39.48	41.06	41.44	39.07
AVTP	39.72	49.48	42.75	44.96
CLSE	–	–	43.57	–
S²Prune	47.04	56.04	52.28	52.18
ToDRE	52.23	61.95	56.28	63.27
SIEVE	62.19	63.67	58.82	63.57
One eighth of the visual tokens
VisionZip	56.10	62.84	58.68	61.84
FastV	41.16	47.46	52.17	51.04
PyramidDrop	41.68	43.53	49.98	45.77
SparseVLM	43.11	52.36	59.62	54.94
AVTP	44.22	57.30	51.35	52.92
CLSE	46.89	61.37	57.63	59.06
S²Prune	53.72	63.15	57.84	59.28
ToDRE	54.00	65.45	60.04	66.73
SIEVE	65.13	66.21	62.83	65.21
One quarter of the visual tokens
VisionZip	61.39	66.17	63.02	64.62
FastV	51.69	62.26	62.09	65.94
PyramidDrop	55.65	62.15	62.85	65.04
SparseVLM	54.85	65.98	66.14	65.77
AVTP	51.06	63.50	58.53	59.43
CLSE	56.76	66.78	63.76	65.64
S²Prune	58.82	67.91	62.66	63.84
ToDRE	58.88	68.15	63.99	67.71
SIEVE	66.64	68.26	65.14	67.94

Each score is the average over Mantis-Eval, NLVR2, MuirBench, BLINK, and Q-Bench2. Bold marks the best result in each column and underline the second best. A dash marks a method that cannot run at that budget. At one sixteenth, FastV and PyramidDrop are left with a single token after reduction, so their schedules coincide and their rows are identical.

Integration and applicability

Training-free integration. SIEVE uses the original model weights without fine-tuning, distillation, or labeled data.
Single-pass inference. It requires no auxiliary model, additional encoder, or draft-and-verify procedure.
One configuration. The same hyperparameters are used for LLaVA-OneVision-7B, Qwen2.5-VL-7B, Qwen3-VL-8B, and InternVL3-8B, at every budget and on every benchmark, with no benchmark-specific tuning.
User-defined token budget. A closed-form schedule satisfies the chosen retention ratio, providing explicit control over the depth-averaged token budget.
Single-image compatibility. With one image, the per-image quota reduces to global ranking. At one eighth of the tokens, LLaVA-OneVision achieves a mean score of 71.97 across four single-image benchmarks, compared with 80.67 without compression.

Citation

@article{shukla2026sieve,
  title   = {{SIEVE}: Preserving Cross-Image Evidence via Depth-Aware Token Compression},
  author  = {Shukla, Nitish and Jandial, Surgan and Bui, Nhat-Tan and Ross, Arun},
  journal = {arXiv preprint arXiv:2611.00351},
  year    = {2026}
}