C:\Users\maxch\AppData\Local\Programs\Python\Python313\Lib\site-packages\torch\cuda_init_.py:65: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you. import pynvml # type: ignore[import] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d Loading weights: 100%|█████████████████████████████████████████████| 320/320 [00:00<00:00, 553.59it/s, Materializing param=model.norm.weight] Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. Error in generation loop: Invalid group type: linear_attention Traceback (most recent call last): File "C:\Users\maxch\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\generation\continuous_batching\continuous_api.py", line 767, in _run_generation_loop paged_attention_cache = PagedAttentionCache( self.model.config, ...<4 lines>... allow_block_sharing=self._allow_block_sharing, ) File "C:\Users\maxch\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\generation\continuous_batching\cache.py", line 240, in init raise ValueError(f"Invalid group type: {group_type}") ValueError: Invalid group type: linear_attention Generation thread terminated unexpectedly. Solving 5 requests: 0%| | 0/5 [00:01<?, ?request/s]Returning results of generate_batch despite unexpected termination. Solving 5 requests: 0%| | 0/5 [00:01<?, ?request/s]

Batch processor was not initialized. Request req_0 not found in results. Request req_1 not found in results. Request req_2 not found in results. Request req_3 not found in results. Request req_4 not found in results. --- Running CB Generation Example --- Error in generation loop: Invalid group type: linear_attention
Traceback (most recent call last): File "C:\Users\maxch\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\generation\continuous_batching\continuous_api.py", line 767, in _run_generation_loop paged_attention_cache = PagedAttentionCache( self.model.config, ...<4 lines>... allow_block_sharing=self._allow_block_sharing, ) File "C:\Users\maxch\AppData\Local\Programs\Python\Python313\Lib\site-packages\transformers\generation\continuous_batching\cache.py", line 240, in init raise ValueError(f"Invalid group type: {group_type}") ValueError: Invalid group type: linear_attention Generation thread terminated unexpectedly.
Solving 5 requests: 0%| | 0/5 [00:01<?, ?request/s]Returning results of generate_batch despite unexpected termination. Solving 5 requests: 0%| | 0/5 [00:01<?, ?request/s]

Batch processor was not initialized. Request req_0 not found in results. Request req_1 not found in results. Request req_2 not found in results. Request req_3 not found in results. Request req_4 not found in results. Done with batch generation.

--- Finished CB Generation Example ---

CB generation took: 1.01 seconds for 0 tokens. 0.00tok/s

Fix Plan

Update transformers library to a version that supports hybrid attention models

Step 1: Update transformers library

pip install transformers --upgrade

Step 2: Check transformers library version

import transformers
print(transformers.__version__)

The expected output is a version greater than 5.2.0.

Verify the fix

Step 1: Run the example script again

python examples/pytorch/continuous_batching_simple.py --samples 5

Step 2: Check the output

The script should complete successfully and return generated text for all 5 requests.

Extra Tips

• Make sure to update all dependencies, including transformers, huggingface_hub, and safetensors.
• If you're using a virtual environment, make sure to activate it before running the script.
• If you're still experiencing issues, try resetting the environment and re-running the script.