transformers - 💡(How to fix) Fix Qwen3-VL: Halucination/Error with 2D bounding box output [2 comments, 2 participants]

Fix Plan

Aspect Ratio Adjustment

To fix the issue, we need to adjust the aspect ratio of the input images during 2D training and rescale the output bounding boxes.

Steps

1. Update Training Configuration
   • Change the aspect ratio in the training configuration file (e.g., config.yaml) to match the aspect ratio of the KITTI images (3.3:1).
   • Example:

aspect_ratio: 3.3

2. **Update Model Architecture**
   - Modify the model architecture to output 2D bounding boxes instead of 3D bounding boxes.
   - Example (assuming a PyTorch model):
     ```python
class TwoDBoxDetector(nn.Module):
    def forward(self, x):
        # ... (rest of the model architecture)
        boxes_2d = self.output_layer(x)
        return boxes_2d

1. Rescale Output Bounding Boxes
   • Add a post-processing step to rescale the output bounding boxes based on the aspect ratio.
   • Example:

def rescale_boxes(boxes, aspect_ratio): # ... (calculate the rescaled box coordinates) return rescaled_boxes

4. **Update Detection Script**
   - Modify the detection script to use the updated model architecture and rescale the output bounding boxes.
   - Example:
     ```python
detector = TwoDBoxDetector()
boxes = detector(image)
rescaled_boxes = rescale_boxes(boxes, aspect_ratio)

Verification

• Run the detection script on KITTI images and verify that the output JSON contains 9 values per box (3D bbox) is not present.
• Check that the output bounding boxes are correctly rescaled to match the aspect ratio of the input images.

Extra Tips

• Make sure to update the model architecture and detection script to handle the new aspect ratio.
• Test the