from vllm import LLM, SamplingParams from PIL import Image import urllib.request, io

url = "https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/280px-PNG_transparency_demonstration_1.png" req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"}) with urllib.request.urlopen(req) as resp: image = Image.open(io.BytesIO(resp.read())).convert("RGB")

This WORKS:

llm = LLM(model="allenai/Molmo2-8B", trust_remote_code=True, max_model_len=4096, max_num_batched_tokens=4096) outputs = llm.generate( {"prompt": "<|image|> Describe this image.", "multi_modal_data": {"image": image}}, sampling_params=SamplingParams(temperature=0.0, max_tokens=64), ) print(outputs[0].outputs[0].text) # Works fine

This CRASHES:

llm2 = LLM(model="allenai/MolmoWeb-8B", trust_remote_code=True, max_model_len=4096, max_num_batched_tokens=4096) outputs2 = llm2.generate( {"prompt": "<|image|> Describe this image.", "multi_modal_data": {"image": image}}, sampling_params=SamplingParams(temperature=0.0, max_tokens=64), )

torch.AcceleratorError: CUDA error: device-side assert triggered

TL;DR

The most likely fix is to increase the max_position_embeddings in the MolmoWeb models to match the value used in the Molmo2 models, which is 36864.

Guidance

• Verify that the crash occurs due to the max_position_embeddings difference by testing with a MolmoWeb model that has its max_position_embeddings set to 36864.
• Update the max_position_embeddings value in the MolmoWeb model configuration to 36864 to potentially resolve the crash.
• If updating the model configuration is not feasible, consider using the allenai/Molmo2-8B model as a workaround, as it does not exhibit the same issue.
• Be cautious when updating model configurations, as changes can have unintended effects on model performance or behavior.

Example

No code snippet is provided, as the issue is related to model configuration rather than code.

Notes

The provided analysis suggests that the max_position_embeddings difference is the likely cause of the crash. However, it is essential to verify this by testing with the updated configuration. Additionally, updating the model configuration may have other effects on the model's performance or behavior, which should be carefully evaluated.

Recommendation

Apply the workaround by using the allenai/Molmo2-8B model until the max_position_embeddings issue is resolved in the MolmoWeb models, as it is a known working configuration.