Continuous batching issue and feature tracker

Introduction

Core idea

Continuous batching is the throughput-focused generation API for the transformers library, available through the generate_batch method. It aims to provide a highly optimized generation API to a wide range of transformers models across various hardware. Since this is a non-trivial goal to achieve, we are doing this in an incremental manner. This is because we would rather roll out support slowly, ensuring a decent level of performance, rather than promising support for models that are not correctly optimized.

Current status

Continuous batching currently supports text-only models deployed on a single device or a single node through TP.

Motivation

This document's goal is to centralize:

• current issues with continuous batching, related to either performance or accuracy. The latter is considered the worse of the two
• features that are requested by the community
• features that are already on the roadmap
• improvements that could be made to the existing API, for performance or user-friendliness reasons and track related issues and/or PRs.

Contributing

This document is also a good starting point for anyone wanting to contribute to continuous batching. Please note that some bullet points are marked as MR, which stands for maintainer-reserved: please do not try to submit a PR for those, as it will probably be rejected. The rest is fair game! Some bullet points are marked as Tricky: these may take more time to properly fix / ship than others, be warned!

And of course, please read the transformers guidelines before contributing.

A note on AI

I encourage all contributors to read the part on agent-written PRs in the transformers guidelines. Since reviewing a PR takes time, any PR that was fully written by an agent will be closed for that reason alone. For instance, PRs with an AI-written summary have a higher chance of not passing that filter. So please write your PR summaries yourself! It shows that you clearly understand the issue and the path that was taken to solve it. The quality of your English does not matter: broken English is better than LLM English. The same applies to comments.

Issues, features, experiments

Issues

• (High prio) Rewrite get_result to be more efficient
• (Low hanging fruit) Change the temporary token id to 0 so we can simplify carry-over
• Consolidate the schedulers into two near-identical classes
• Switch back to Python hash for TP processes if that does not break TP
• Fix CPU block leak when H2D copy crashes during request reloading
• Fix CPU request reloading when async batching is turned on
• Address low-priority # TODOs

Features

• (MR) #46376
• (MR) Add speculative decoding support in continuous batching (abstract mechanism first)
• (MR) Support exotic caches like DeepseekV4 (mostly about generalizing the current cache)
• (Tricky) Support models with a multi-device device_map object
• (Tricky) Switch the synchronous inputs_outputs to a HostDevicePair-based system
• Add CB logits processors that are not per-request processors
• Add a temperature + top-p + top-k fused logits processor
• Estimate KV head count by running the model instead of relying on TP keys
• Support sliding window in the decode fast path

Experiments

• (High prio) Is there a way to speed up compile for models?
• Is there a scheduling method that increases the throughput over benchmarks like GSM8K?
• Are there workloads where generate is still faster than generate_batch?