vllm - 💡(How to fix) Fix [Feature]: Support multi-image and interleaved multimodal custom datasets in bench serve

🚀 The feature, motivation and pitch

We are using vllm bench serve to test our online multimodal services.

For image-based multimodal models, the current custom image dataset support has two limitations:

1. If a custom sample contains multiple images in image_files, only the first image is used and the remaining images are discarded.
2. The request format does not support arbitrary interleaving of text and images, such as text -> image -> text -> image -> text.

This makes it difficult to reproduce realistic production traffic, where requests may include multiple images, image comparison, document/page sequences, or instructions placed between images.

It would be useful if vllm bench serve could support a custom chat-style multimodal dataset format, where each JSONL row can provide OpenAI-compatible content parts directly, for example:

{
  "content": [
    {"type": "text", "text": "Please inspect this image."},
    {"type": "image_url", "image_url": {"url": "file:///path/to/a.jpg"}},
    {"type": "text", "text": "Now compare it with this one."},
    {"type": "image_url", "image_url": {"url": "file:///path/to/b.jpg"}}
  ],
  "output_tokens": 128
}

I would like to know whether the community considers this a reasonable direction for vllm bench serve, and whether there are any existing plans to support multi-image custom datasets and arbitrary interleaved multimodal content.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.