Issue Station
Sign In
Tools hereToolbox

transformers - ✅(Solved) Fix SAM3 PCS very weird behaviour when providing text and bounding boxes [1 pull requests, 3 comments, 3 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
huggingface/transformers#45059•Fetched 2026-04-08 01:40:51
View on GitHub
Comments
3
Participants
3
Timeline
14
Reactions
0
Author
alex-benealex-bene
Participants
alex-benealex-beneKash6Kash6Rocketknight1Rocketknight1
Timeline (top)
commented ×3mentioned ×3subscribed ×3referenced ×2

Root Cause

So, I was trying to use SAM3 in PCS mode to segment an object for which I have both the bounding box and a textual description. Providing just the text does not find the object (understandable in my case because the text description is not good). Providing both text and a bounding box does find and segment the object correctly (generally, this seems to be a hit or miss in my tests; sometimes, providing the bounding box with the text generates worse masks than providing just the text). However, providing just the text and (after getting the inputs from the Sam3Processor), setting:

inputs["input_boxes"] = inputs["pixel_values"].new_tensor([[[-10.0, -10.0, 0.0, 0.0]]]).expand(len(inputs["pixel_values"]), -1, -1)

finds the object (as well as various other things in the image) and actually generates a good mask for it. The choice of [-10.0, -10.0, 0.0, 0.0] comes from running something like this:

inputs = processor(
    images=[image, image], text=["sheet", None], input_boxes=[None, [[719, 234, 1048, 693]]], return_tensors="pt"
).to(device)

which set inputs["input_boxes"][1] as a tensor of value [[[-10.0, -10.0, 0.0, 0.0]]].

PR fix notes

PR #45171: Fix Sam3Processor missing input_boxes_labels for padded None entries

Description (problem / solution / changelog)

When input_boxes contains a mix of None and real box entries (e.g. input_boxes=[None, [[x1,y1,x2,y2]]]), the processor pads None entries with [-10,-10,0,0] but does not generate corresponding input_boxes_labels. The model's geometry encoder then treats all boxes as valid (box_mask=ones), including the padded ones, causing different behavior compared to input_boxes=None.

Generate default input_boxes_labels when not provided by the user: None entries get label -10 (the pad value) so the model masks them out, real entries get label 1 (positive). User-provided labels are preserved unchanged.

Adds processor tests covering mixed None/real batches, all-real batches, no-boxes, and user-provided label preservation.

What does this PR do?

<!-- Congratulations! You've made it this far! You're not quite done yet though. Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution. Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change. Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost. --> <!-- Remove if not applicable -->

Fixes #45059

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by code agents. We are currently bottlenecked by our ability to review and respond to them. As a result, we ask that new users do not submit pure code agent PRs at this time. You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result, this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

  • I confirm that this is not a pure code agent PR.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline, Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

@yonigozlan @molbap @NielsRogge @zucchini-nlp

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @ If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**. Please tag fewer than 3 people. Models: - text models: @ArthurZucker @Cyrilvallez - vision models: @yonigozlan @molbap - audio models: @eustlb @ebezzam @vasqu - multimodal models: @zucchini-nlp - graph models: @clefourrier Library: - generate: @zucchini-nlp (visual-language models) or @gante (all others) - continuous batching: @remi-or @ArthurZucker @McPatate - pipelines: @Rocketknight1 - tokenizers: @ArthurZucker and @itazap - trainer: @SunMarc - attention: @vasqu @ArthurZucker @CyrilVallez - model loading (from pretrained, etc): @CyrilVallez - distributed: @3outeille @ArthurZucker - CIs: @ydshieh Integrations: - ray/raytune: @richardliaw, @amogkam - Big Model Inference: @SunMarc - quantization: @SunMarc - kernels: @drbh - peft: @BenjaminBossan @githubnemo Devices/Backends: - AMD ROCm: @ivarflakstad - Intel XPU: @IlyasMoutawwakil - Ascend NPU: @ivarflakstad Documentation: @stevhliu Research projects are not maintained and should be taken as is. -->

Changed files

  • src/transformers/models/sam3/processing_sam3.py (modified, +8/-0)
  • tests/models/sam3/test_processing_sam3.py (added, +105/-0)

Code Example

inputs["input_boxes"] = inputs["pixel_values"].new_tensor([[[-10.0, -10.0, 0.0, 0.0]]]).expand(len(inputs["pixel_values"]), -1, -1)

---

inputs = processor(
    images=[image, image], text=["sheet", None], input_boxes=[None, [[719, 234, 1048, 693]]], return_tensors="pt"
).to(device)

---

inputs = processor(
    images=[image, image], text=["sheet", None], input_boxes=[None, [[719, 234, 1048, 693]]], return_tensors="pt"
).to(device)
assert torch.allclose(inputs["input_boxes"][0], inputs["pixel_values"].new_tensor([[[-10.0, -10.0, 0.0, 0.0]]])

---

inputs = processor(images=[image], text=["sheet", input_boxes=None, return_tensors="pt").to(device)
assert "input_boxes" not in inputs
RAW_BUFFERClick to expand / collapse

System Info

So, I was trying to use SAM3 in PCS mode to segment an object for which I have both the bounding box and a textual description. Providing just the text does not find the object (understandable in my case because the text description is not good). Providing both text and a bounding box does find and segment the object correctly (generally, this seems to be a hit or miss in my tests; sometimes, providing the bounding box with the text generates worse masks than providing just the text). However, providing just the text and (after getting the inputs from the Sam3Processor), setting:

inputs["input_boxes"] = inputs["pixel_values"].new_tensor([[[-10.0, -10.0, 0.0, 0.0]]]).expand(len(inputs["pixel_values"]), -1, -1)

finds the object (as well as various other things in the image) and actually generates a good mask for it. The choice of [-10.0, -10.0, 0.0, 0.0] comes from running something like this:

inputs = processor(
    images=[image, image], text=["sheet", None], input_boxes=[None, [[719, 234, 1048, 693]]], return_tensors="pt"
).to(device)

which set inputs["input_boxes"][1] as a tensor of value [[[-10.0, -10.0, 0.0, 0.0]]].

So the question is this:

  1. Should indeed inputs["input_boxes"] be set to [[[-10.0, -10.0, 0.0, 0.0]]] when input_boxes is None? If so, this is a processor bug for not setting it and only setting it when input_boxes is a list and some of the elements of it are None. Also, providing this seems to make instance detection more sensitive overall; in other images where the text-only segmentation worked, adding this "magic" input box led to finding more instances unrelated to the input text.
  2. Is providing a text and a bounding box, intending to complement one another (i.e., the bounding box saying that, hey, the object should be here, try to find what you're looking for in this region), an "intended" use case or did the fact that providing both helped in my case was becase SAM3 independently tried to segment instances based on text and boxes independently?

Examples

Conditioning on the text "human"

<img width="1920" height="1080" alt="Image" src="https://github.com/user-attachments/assets/3b475aee-8d2e-4139-9a40-d8dfdb8cc403" />

Conditioning on the text "human" & magic input box

<img width="1920" height="1080" alt="Image" src="https://github.com/user-attachments/assets/c91855d2-d392-4cfc-ba9c-3589c572a7c6" />

Conditioning on text "sheet" & magic input box (without magic box there is no segmentation)

<img width="1920" height="1080" alt="Image" src="https://github.com/user-attachments/assets/78dda26c-a675-420f-8e91-b3da97ddb050" />

Conditioning on text "umbrella"

<img width="1920" height="1080" alt="Image" src="https://github.com/user-attachments/assets/1fbd0561-f943-428e-b74a-6bfc04224ff9" />

Conditioning on text "umbrella" & magic input box

<img width="1920" height="1080" alt="Image" src="https://github.com/user-attachments/assets/492e89f5-5b58-4fcf-a30b-09c27c6efdf4" />

Who can help?

@zucchini-nlp

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

inputs = processor(
    images=[image, image], text=["sheet", None], input_boxes=[None, [[719, 234, 1048, 693]]], return_tensors="pt"
).to(device)
assert torch.allclose(inputs["input_boxes"][0], inputs["pixel_values"].new_tensor([[[-10.0, -10.0, 0.0, 0.0]]])
inputs = processor(images=[image], text=["sheet", input_boxes=None, return_tensors="pt").to(device)
assert "input_boxes" not in inputs

Expected behavior

Providing or not the object for an image in the batch should definitely not change the behaviour of other samples in the batch that do not condition on bounding boxes. Still, the outputs for the first image on the top code snippet vs the outputs from the bottom one are different!

extent analysis

Fix Plan

To address the issue, we need to ensure that inputs["input_boxes"] is set correctly when input_boxes is None. We can achieve this by modifying the Sam3Processor to handle this case.

Here are the steps:

  • Check if input_boxes is None and if so, set inputs["input_boxes"] to a default value, such as [[[-10.0, -10.0, 0.0, 0.0]]].
  • Update the processor function to handle this case.

Example code:

def processor(images, text, input_boxes=None, return_tensors="pt"):
    # ... existing code ...
    if input_boxes is None:
        input_boxes = [[[-10.0, -10.0, 0.0, 0.0]]]
    # ... existing code ...
    return inputs

Alternatively, you can also modify the Sam3Processor class to handle this case:

class Sam3Processor:
    # ... existing code ...
    def __call__(self, images, text, input_boxes=None, return_tensors="pt"):
        # ... existing code ...
        if input_boxes is None:
            input_boxes = [[[-10.0, -10.0, 0.0, 0.0]]]
        # ... existing code ...
        return inputs

Verification

To verify that the fix worked, you can run the following code:

inputs = processor(
    images=[image, image], text=["sheet", None], input_boxes=[None, [[719, 234, 1048, 693]]], return_tensors="pt"
).to(device)
assert torch.allclose(inputs["input_boxes"][0], inputs["pixel_values"].new_tensor([[[-10.0, -10.0, 0.0, 0.0]]])

And also:

inputs = processor(images=[image], text=["sheet"], input_boxes=None, return_tensors="pt").to(device)
assert "input_boxes" in inputs
assert torch.allclose(inputs["input_boxes"], inputs["pixel_values"].new_tensor([[[-10.0, -10.0, 0.0, 0.0]]])

Extra Tips

  • Make sure to test the updated processor function with different inputs and edge cases to ensure that it works as expected.
  • Consider adding a check to ensure that input_boxes is not None when text is not None, to prevent unexpected behavior.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

FAQ

Expected behavior

Providing or not the object for an image in the batch should definitely not change the behaviour of other samples in the batch that do not condition on bounding boxes. Still, the outputs for the first image on the top code snippet vs the outputs from the bottom one are different!

#api#ssr#model compatibility#GPU setup#container setup

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING