transformers - 💡(How to fix) Fix Feature request: Add EfficientViT-SAM (efficientvitsam) to Transformers [1 participants]

Feature request

EfficientViT-SAM combines MIT’s EfficientViT encoder with the SAM-style prompt encoder and mask decoder. It offers a lighter, faster alternative to ViT-based SAM for interactive segmentation while staying close to the same prompting and mask-decoding workflow.

Today, users who want this stack in the Hugging Face ecosystem must rely on custom code or the upstream repo. First-class support in Transformers would:

Enable from_pretrained on official or community checkpoints (e.g. mit-han-lab/efficientvit-sam and converted .pt weights).

Motivation

Provide a single API aligned with existing SAM-style models (AutoModel, processors, post_process_masks). Improve discoverability and CI-tested parity with upstream preprocessing and architecture where feasible.

Your contribution

Add efficientvitsam model code: configuration, image processing, processor, and PyTorch modeling consistent with upstream efficientvit/applications/efficientvit_sam/. Register the model in auto classes and document it in the model hub docs. Include a conversion path from official checkpoints and tests (including optional slow parity checks against the upstream predictor where appropriate).