vllm - 💡(How to fix) Fix [Feature]: Pre-ViT visual token pruning for VLMs (PixelPrune) [2 comments, 2 participants]

🚀 The feature, motivation and pitch

VLMs for document understanding and GUI interaction require high-resolution image inputs, imposing heavy computational burden on both the Vision Encoder and the LLM. These domains typically exhibit significant pixel-level redundancy (e.g., white margins, uniform backgrounds), yet all patches still go through the full computation pipeline.

PixelPrune (arXiv: 2604.00886) identifies these redundant patches via predictive coding and removes them from the ViT input, accelerating the entire VLM pipeline end-to-end. The method introduces no additional modules and does not alter the internal computation of the ViT or LLM, making it straightforward to integrate into production serving stacks. On document understanding benchmarks, it maintains model accuracy in a fully training-free setting.

Results on Qwen3-VL (training-free, document understanding, τ=0 exact matching):

Retain ratios are content-adaptive per benchmark (50–77%), reflecting domain-specific pixel redundancy. Also validated on Qwen3.5 (2B/4B/9B) with ≤0.8% Avg gap.

Inference efficiency (Qwen3-VL-2B, batch size 1, H20 GPU):

I have integrated PixelPrune into vLLM 0.18.0 via monkey-patching, covering modifications to the processor pipeline and support for pruned-input computation in the vision encoder (Qwen3-VL, Qwen3.5-VL). Would the vLLM team be interested in native support for this feature? I am happy to contribute a PR.

Alternatives

No response

Additional context

• Paper: https://arxiv.org/abs/2604.00886
• Code: https://github.com/OPPO-Mente-Lab/PixelPrune
• User request: https://github.com/OPPO-Mente-Lab/PixelPrune/issues/2

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.