transformers - 💡(How to fix) Fix Add TIPSv2 (Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment) by Google DeepMind

Model description

TIPSv2 is a vision-language encoder that addresses the problem of dense alignment between image regions and text through modification in the pretraining recipe (iBOT++, multi-granularity synthetic captions, and memory-saving head-only EMA scheme). Across 9 tasks and 20 datasets the resulting models set new state-of-the-art results on zero-shot semantic segmentation while generally matching or beating recent encoders like SigLIP2, DINOv3, and Perception Encoder on global and dense tasks.

The model family comes with four sizes (B, L, SO400m, and G) with a variant that comes with DPT head. All of the models are already available in HuggingFace. The code is licensed under Apache 2.0 and the weights have CC-BY 4.0.

I would be glad if I can contribute implementing this model to HuggingFace's Transformer, including model implementation, weight conversion, tests and docs. Happy to coordinate with maintainers about the implementation and integration.

Thank you.

@NielsRogge @molbap @Rocketknight1

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

• Official GitHub Repository
• Weights