Issue Station
Sign In
Tools hereToolbox

vllm - ✅(Solved) Fix [New Model]: OmniASR by meta [1 pull requests, 1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#36784•Fetched 2026-04-08 00:34:41
View on GitHub
Comments
1
Participants
2
Timeline
3
Reactions
0
Author
HDprojec8sHDprojec8s
Participants
HDprojec8sHDprojec8sziywang50ziywang50
Assignees
ziywang50ziywang50
Timeline (top)
assigned ×1commented ×1cross-referenced ×1

Fix Action

Fixed

PR fix notes

PR #36864: [Model] Add OmniASR (Meta Omnilingual ASR) model support

Description (problem / solution / changelog)

Purpose

Add initial model support for Meta's OmniASR (Omnilingual ASR) model family. Addresses #28509 and #36784.

OmniASR is an encoder-decoder ASR model supporting 1600+ languages, built on fairseq2:

Audio (16kHz) -> Wav2Vec2 CNN Frontend (7 conv layers, dim=512)
             -> Wav2Vec2 Transformer Encoder (24 layers, dim=1024)
             -> Linear Projection (1024 -> 4096)
             -> LLaMA Decoder (12 layers, dim=4096)
             -> Text Output (vocab=9812)

No cross-attention. Encoder outputs are projected and used as input embeddings (same pattern as Ultravox).

Current status:

  • Wav2Vec2 CNN frontend and 24-layer Transformer encoder implemented
  • Encoder attention uses MMEncoderAttention / QKVParallelLinear / RowParallelLinear
  • Weight loading verified against omniASR-LLM-300M checkpoint (535/535 keys matched)
  • Encoder forward pass tested (16kHz audio -> 49 frames x 4096-dim output)

TODO:

  • Integrate LLaMA decoder with vLLM's existing LlamaForCausalLM
  • Add model registration and processor
  • End-to-end inference support
  • Multi-size model support (300M, 1B, 3B, 7B)

References:

CLOSE #28509 CLOSE #36784

Test Plan

Weight loading and encoder forward pass tested offline with omniASR-LLM-300M checkpoint. Full vLLM integration tests to be added once model registration is complete.

Test Result

Weights loaded successfully!
Loaded 535 parameters
Audio -> Encoder -> Projection: torch.Size([1, 49, 4096])
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update.

Changed files

  • vllm/model_executor/models/omniasr.py (added, +205/-0)
RAW_BUFFERClick to expand / collapse

The model to consider.

please give support to omnilingual model by meta (CTC,LLM,W2V)

The closest model vllm already supports.

No response

What's your difficulty of supporting the model you want?

No response

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

To add support for the omnilingual model by Meta (CTC, LLM, W2V), we need to implement the following steps:

  • Update the model configuration to include the omnilingual model
  • Modify the training loop to accommodate the new model architecture
  • Add support for the CTC, LLM, and W2V loss functions

Example Code

# Import necessary libraries
import torch
import torch.nn as nn
import torch.optim as optim

# Define the omnilingual model configuration
class OmnilingualModel(nn.Module):
    def __init__(self):
        super(OmnilingualModel, self).__init__()
        self.ctc_loss = nn.CTCLoss()
        self.llm_loss = nn.CrossEntropyLoss()
        self.w2v_loss = nn.MSELoss()

    def forward(self, input_ids, attention_mask, labels):
        # Implement the forward pass for the omnilingual model
        pass

# Update the training loop to support the omnilingual model
def train(model, device, loader, optimizer, epoch):
    model.train()
    total_loss = 0
    for batch in loader:
        input_ids = batch["input_ids"].to(device)
        attention_mask = batch["attention_mask"].to(device)
        labels = batch["labels"].to(device)

        optimizer.zero_grad()

        outputs = model(input_ids, attention_mask, labels)
        loss = model.ctc_loss(outputs, labels) + model.llm_loss(outputs, labels) + model.w2v_loss(outputs, labels)

        loss.backward()
        optimizer.step()

        total_loss += loss.item()

    print(f"Epoch {epoch+1}, Loss: {total_loss / len(loader)}")

### Verification
To verify that the fix worked, you can test the model on a sample dataset and check the loss values for the CTC, LLM, and W2V loss functions.

### Extra Tips
* Make sure to update the documentation to reflect the changes made to support the omnilingual model.
* Test the model thoroughly to ensure that it is working as expected.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#autograd error#GPU compatibility#latency issue#model loading#dependency error#configuration error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING