Issue Station
Sign In
Tools hereToolbox

vllm - ✅(Solved) Fix [Feature]: Add apply_with_spec_decode() method to LogitBiasLogitsProcessor for speculative decoding support [2 pull requests, 2 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#38202•Fetched 2026-04-08 01:31:41
View on GitHub
Comments
2
Participants
2
Timeline
7
Reactions
0
Author
ranger2571ranger2571
Participants
NJX-njxNJX-njxranger2571ranger2571
Timeline (top)
commented ×2cross-referenced ×2referenced ×2labeled ×1

Error Message

We have added an apply_with_spec_decode function to LogitBiasLogitsProcessor, included corresponding test cases, and adjusted the current raise error condition.

PR fix notes

PR #38467: [Feature] Add apply_with_spec_decode() to LogitBiasLogitsProcessor

Description (problem / solution / changelog)

Purpose

Fixes #38202

LogitBiasLogitsProcessor was not supported during speculative decoding. When spec_decode was enabled, logit_bias parameters were silently ignored with a warning. This PR adds speculative decoding support so logit biases are correctly applied during rejection sampling.

Changes

  1. vllm/v1/sample/logits_processor/builtin.py: Added apply_with_spec_decode() method to LogitBiasLogitsProcessor that maps request indices to their corresponding rows in the expanded logits tensor (shape [sum(num_draft_tokens), vocab_size]), applying biases to all draft positions for each request.

  2. vllm/v1/sample/logits_processor/__init__.py: Include LogitBiasLogitsProcessor in the spec_decode processor list (previously only MinTokensLogitsProcessor was included).

  3. vllm/v1/sample/rejection_sampler.py: Update the apply_logits_processors dispatch to call apply_with_spec_decode for both LogitBiasLogitsProcessor and MinTokensLogitsProcessor.

Test Plan

  • Run existing logits processor tests: pytest tests/v1/logits_processors/
  • Verify logit_bias works with speculative decoding enabled
  • Verify no regression on non-spec-decode logit_bias behavior

Test Result

The implementation follows the same pattern as MinTokensLogitsProcessor.apply_with_spec_decode() using numpy cumsum for request-to-row mapping and batch tensor construction.

Changed files

  • tests/v1/logits_processors/test_logit_bias_spec_decode.py (added, +179/-0)
  • vllm/v1/sample/logits_processor/__init__.py (modified, +5/-4)
  • vllm/v1/sample/logits_processor/builtin.py (modified, +57/-0)
  • vllm/v1/sample/rejection_sampler.py (modified, +1/-2)

PR #38469: fix: Add apply_with_spec_decode() method to LogitBiasLogitsProcessor

Description (problem / solution / changelog)

for speculative decoding support #38202

Purpose

Test Plan

Test Result

<details> <summary> Essential Elements of an Effective PR Description Checklist </summary>
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.
</details>

Changed files

  • tests/v1/sample/test_rejection_sampler.py (modified, +155/-1)
  • vllm/sampling_params.py (modified, +2/-2)
  • vllm/v1/sample/logits_processor/__init__.py (modified, +5/-2)
  • vllm/v1/sample/logits_processor/builtin.py (modified, +66/-0)
  • vllm/v1/sample/rejection_sampler.py (modified, +8/-1)
RAW_BUFFERClick to expand / collapse

🚀 The feature, motivation and pitch

LogitBiasLogitsProcessor is not supported during speculative decoding. We have added an apply_with_spec_decode function to LogitBiasLogitsProcessor, included corresponding test cases, and adjusted the current raise error condition.

Alternatives

Add apply_with_spec_decode() method to LogitBiasLogitsProcessor for speculative decoding support

Additional context

No response

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

To support LogitBiasLogitsProcessor during speculative decoding, we need to implement the apply_with_spec_decode method. Here are the steps:

  • Add a new method apply_with_spec_decode to the LogitBiasLogitsProcessor class.
  • Implement the logic for speculative decoding in the new method.
  • Update the test cases to cover the new method.

Example Code

class LogitBiasLogitsProcessor:
    # existing methods...

    def apply_with_spec_decode(self, logits, **kwargs):
        # implement speculative decoding logic here
        # for example:
        bias = self.bias
        spec_decode_logits = logits + bias
        return spec_decode_logits

Verification

To verify the fix, run the updated test cases and check that the apply_with_spec_decode method is working correctly.

Extra Tips

  • Make sure to update the documentation to reflect the new method and its usage.
  • Consider adding additional test cases to cover edge cases and ensure the method is working as expected.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#prompt template#agent execution#callback error#memory management#API rate limit

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING