vllm - 💡(How to fix) Fix [Bug]: qwen3.6-27B claude code使用特别慢 [1 comments, 1 participants]

Your current environment

两三个Claude Code请求并发时，上下文还没有到特别长的时候，日志打印的Avg经常是0： Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.8 tokens/s, Running: 4 reqs

硬件是4090 48G单卡： 全是最新版本 qwen3.6-27B-FP8

启动命令： CUDA_VISIBLE_DEVICES=0 vllm serve /home/ma/work/models/Qwen3.6-27B-FP8 --served-model-name gpt --port 8000 --max-model-len 220000 --reasoning-parser qwen3 --max-num-seqs 32 --gpu-memory-utilization 0.92 --host 0.0.0.0 --tool-call-parser qwen3_coder --enable-auto-tool-choice --async-scheduling --performance-mode throughput --enable-prefix-caching --kv-cache-dtype fp8 --enable-chunked-prefill

(vllm-qwen) ma@gpu4090d:~$ python collect_env.py Traceback (most recent call last): File "/home/ma/collect_env.py", line 20, in <module> from vllm.envs import environment_variables File "/home/ma/miniconda3/envs/vllm-qwen/lib/python3.12/site-packages/vllm/init.py", line 14, in <module> import vllm.env_override # noqa: F401 ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ma/miniconda3/envs/vllm-qwen/lib/python3.12/site-packages/vllm/env_override.py", line 87, in <module> import torch File "/home/ma/miniconda3/envs/vllm-qwen/lib/python3.12/site-packages/torch/init.py", line 431, in <module> from torch._C import * # noqa: F403 ^^^^^^^^^^^^^^^^^^^^^^ ImportError: /home/ma/miniconda3/envs/vllm-qwen/lib/python3.12/site-packages/torch/lib/../../nvidia/cusparse/lib/libcusparse.so.12: undefined symbol: __nvJitLinkGetErrorLogSize_12_9, version libnvJitLink.so.12

🐛 Describe the bug

Apr 22 15:29:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:29:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 46.9%, Prefix cache hit rate: 54.0% Apr 22 15:29:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:29:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.1 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 52.0%, Prefix cache hit rate: 51.3% Apr 22 15:29:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:29:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 56.5%, Prefix cache hit rate: 49.0% Apr 22 15:29:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:29:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 60.9%, Prefix cache hit rate: 47.0% Apr 22 15:29:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:29:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 63.8%, Prefix cache hit rate: 45.7% Apr 22 15:30:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:30:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 66.8%, Prefix cache hit rate: 44.6% Apr 22 15:30:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:30:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 69.7%, Prefix cache hit rate: 43.4% Apr 22 15:30:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:30:21 [loggers.py:259] Engine 000: Avg prompt throughput: 11056.3 tokens/s, Avg generation throughput: 2.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.6%, Prefix cache hit rate: 42.3% Apr 22 15:30:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:30:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.8%, Prefix cache hit rate: 42.3% Apr 22 15:30:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:30:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.2%, Prefix cache hit rate: 42.3% Apr 22 15:30:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:30:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 88.9%, Prefix cache hit rate: 42.3% Apr 22 15:31:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:31:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 91.9%, Prefix cache hit rate: 42.3% Apr 22 15:31:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:31:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 94.8%, Prefix cache hit rate: 42.3% Apr 22 15:31:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:31:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 97.8%, Prefix cache hit rate: 42.3% Apr 22 15:31:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:31:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.6 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 69.7%, Prefix cache hit rate: 42.3% Apr 22 15:31:31 gpu4090d bash[6486]: (EngineCore pid=6486) /home/ma/miniconda3/envs/vllm-qwen/lib/python3.12/site-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (39) < num_heads (48). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...]. Apr 22 15:31:31 gpu4090d bash[6486]: (EngineCore pid=6486) return fn(*contiguous_args, **contiguous_kwargs) Apr 22 15:31:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:31:41 [loggers.py:259] Engine 000: Avg prompt throughput: 12234.2 tokens/s, Avg generation throughput: 65.8 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 96.3%, Prefix cache hit rate: 42.3% Apr 22 15:31:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:31:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 78.3 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.4%, Prefix cache hit rate: 42.3% Apr 22 15:31:58 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:37676 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:32:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:32:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 41.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 47.2%, Prefix cache hit rate: 42.0% Apr 22 15:32:03 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:40342 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:32:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:32:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.1 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 52.4%, Prefix cache hit rate: 40.8% Apr 22 15:32:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:32:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 56.8%, Prefix cache hit rate: 39.4% Apr 22 15:32:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:32:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.5 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 60.5%, Prefix cache hit rate: 38.4% Apr 22 15:32:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:32:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.5 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 64.2%, Prefix cache hit rate: 37.5% Apr 22 15:32:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:32:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 67.2%, Prefix cache hit rate: 36.8% Apr 22 15:33:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:33:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 70.1%, Prefix cache hit rate: 36.0% Apr 22 15:33:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:33:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.6 tokens/s, Running: 3 reqs, Waiting: 2 reqs, GPU KV cache usage: 68.6%, Prefix cache hit rate: 35.3% Apr 22 15:33:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:33:21 [loggers.py:259] Engine 000: Avg prompt throughput: 11806.0 tokens/s, Avg generation throughput: 2.2 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 76.0%, Prefix cache hit rate: 34.9% Apr 22 15:33:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:33:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 80.4%, Prefix cache hit rate: 34.8% Apr 22 15:33:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:33:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 84.9%, Prefix cache hit rate: 34.7% Apr 22 15:33:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:33:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.5 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 88.6%, Prefix cache hit rate: 34.6% Apr 22 15:34:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:34:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 91.5%, Prefix cache hit rate: 34.6% Apr 22 15:34:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:34:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 94.5%, Prefix cache hit rate: 34.5% Apr 22 15:34:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:34:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.9 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 96.7%, Prefix cache hit rate: 34.5% Apr 22 15:34:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:34:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 99.6%, Prefix cache hit rate: 34.4% Apr 22 15:34:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:34:41 [loggers.py:259] Engine 000: Avg prompt throughput: 13591.8 tokens/s, Avg generation throughput: 25.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 98.9%, Prefix cache hit rate: 34.4% Apr 22 15:34:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:34:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 59.3 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 95.9%, Prefix cache hit rate: 34.4% Apr 22 15:35:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:35:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 46.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.5%, Prefix cache hit rate: 34.4% Apr 22 15:35:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:37676 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:35:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:35:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 69.2 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 70.8%, Prefix cache hit rate: 17.2% Apr 22 15:35:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:35:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 70.8%, Prefix cache hit rate: 11.2% Apr 22 15:35:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:35:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 70.8%, Prefix cache hit rate: 8.3% Apr 22 15:35:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:35:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 70.8%, Prefix cache hit rate: 6.6% Apr 22 15:35:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:35:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 70.8%, Prefix cache hit rate: 1.4% Apr 22 15:36:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:36:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 70.8%, Prefix cache hit rate: 0.0% Apr 22 15:36:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:36:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 71.2%, Prefix cache hit rate: 0.0% Apr 22 15:36:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:36:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 71.2%, Prefix cache hit rate: 0.0% Apr 22 15:36:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:36:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 72.0%, Prefix cache hit rate: 0.0% Apr 22 15:36:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:36:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% Apr 22 15:36:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:36:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.4 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% Apr 22 15:37:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:37:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% Apr 22 15:37:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:37:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% Apr 22 15:37:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:37:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% Apr 22 15:37:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:37:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.8 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 0.0% Apr 22 15:37:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:37:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 68.4 tokens/s, Running: 4 reqs, Waiting: 1 reqs, GPU KV cache usage: 72.7%, Prefix cache hit rate: 0.0% Apr 22 15:37:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:37:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.5%, Prefix cache hit rate: 0.0% Apr 22 15:38:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:38:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.1 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 58.7%, Prefix cache hit rate: 0.0% Apr 22 15:38:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:38:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 63.1%, Prefix cache hit rate: 0.0% Apr 22 15:38:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:38:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 66.8%, Prefix cache hit rate: 0.0% Apr 22 15:38:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:38:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 69.7%, Prefix cache hit rate: 0.0% Apr 22 15:38:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:38:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.7%, Prefix cache hit rate: 0.0% Apr 22 15:38:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:38:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.6%, Prefix cache hit rate: 0.0% Apr 22 15:39:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:39:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1.2 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 0.0% Apr 22 15:39:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:39:11 [loggers.py:259] Engine 000: Avg prompt throughput: 12083.7 tokens/s, Avg generation throughput: 47.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 78.6%, Prefix cache hit rate: 0.0% Apr 22 15:39:15 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:44422 - "POST /v1/chat/completions HTTP/1.1" 200 OK Apr 22 15:39:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:39:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 39.9 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.3%, Prefix cache hit rate: 0.0% Apr 22 15:39:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:39:31 [loggers.py:259] Engine 000: Avg prompt throughput: 1310.0 tokens/s, Avg generation throughput: 58.4 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 49.4%, Prefix cache hit rate: 0.0% Apr 22 15:39:33 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:37676 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:39:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:39:41 [loggers.py:259] Engine 000: Avg prompt throughput: 105.6 tokens/s, Avg generation throughput: 57.5 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.6%, Prefix cache hit rate: 0.1% Apr 22 15:39:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:39:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 57.5 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 62.7%, Prefix cache hit rate: 0.1% Apr 22 15:40:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:40:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 52.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 62.7%, Prefix cache hit rate: 0.1% Apr 22 15:40:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:40:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 52.8 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 63.1%, Prefix cache hit rate: 0.1% Apr 22 15:40:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:40:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 53.1 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 63.5%, Prefix cache hit rate: 0.1% Apr 22 15:40:28 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:44422 - "POST /v1/chat/completions HTTP/1.1" 200 OK Apr 22 15:40:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:40:31 [loggers.py:259] Engine 000: Avg prompt throughput: 282.7 tokens/s, Avg generation throughput: 45.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.1%, Prefix cache hit rate: 0.1% Apr 22 15:40:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:40:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.1%, Prefix cache hit rate: 0.1% Apr 22 15:40:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:40:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.1% Apr 22 15:41:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:41:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.1% Apr 22 15:41:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:41:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.1% Apr 22 15:41:12 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:44422 - "POST /v1/chat/completions HTTP/1.1" 200 OK Apr 22 15:41:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:41:21 [loggers.py:259] Engine 000: Avg prompt throughput: 143.9 tokens/s, Avg generation throughput: 35.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.1%, Prefix cache hit rate: 0.1% Apr 22 15:41:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:41:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.1% Apr 22 15:41:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:41:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.1% Apr 22 15:41:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:41:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.1% Apr 22 15:42:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:42:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.1% Apr 22 15:42:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:42:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.1% Apr 22 15:42:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:42:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.1% Apr 22 15:42:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:42:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.1% Apr 22 15:42:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:42:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.5%, Prefix cache hit rate: 0.1% Apr 22 15:42:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:42:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 32.8%, Prefix cache hit rate: 0.1% Apr 22 15:43:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:43:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.2%, Prefix cache hit rate: 0.1% Apr 22 15:43:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:43:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.2%, Prefix cache hit rate: 0.1% Apr 22 15:43:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:43:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.2%, Prefix cache hit rate: 0.1% Apr 22 15:43:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:43:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.2%, Prefix cache hit rate: 0.1% Apr 22 15:43:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:43:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.2%, Prefix cache hit rate: 0.1% Apr 22 15:43:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:43:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.2%, Prefix cache hit rate: 0.1% Apr 22 15:44:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:44:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.2%, Prefix cache hit rate: 0.1% Apr 22 15:44:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:44:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 35.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.6%, Prefix cache hit rate: 0.1% Apr 22 15:44:13 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:37676 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:44:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:44:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.0%, Prefix cache hit rate: 0.1% Apr 22 15:44:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:44:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 0.1% Apr 22 15:44:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:44:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.1% Apr 22 15:44:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:44:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.5%, Prefix cache hit rate: 0.1% Apr 22 15:45:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:45:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.5%, Prefix cache hit rate: 0.1% Apr 22 15:45:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:45:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 28.4%, Prefix cache hit rate: 0.1% Apr 22 15:45:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:45:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 31.4%, Prefix cache hit rate: 0.1% Apr 22 15:45:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:45:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.6%, Prefix cache hit rate: 0.1% Apr 22 15:45:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:45:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.3 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.8%, Prefix cache hit rate: 0.1% Apr 22 15:45:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:45:51 [loggers.py:259] Engine 000: Avg prompt throughput: 13570.5 tokens/s, Avg generation throughput: 27.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.8%, Prefix cache hit rate: 0.1% Apr 22 15:46:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:46:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 37.6 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.8%, Prefix cache hit rate: 0.1% Apr 22 15:46:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:46:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 37.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.8%, Prefix cache hit rate: 0.1% Apr 22 15:46:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:46:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 37.8 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 36.2%, Prefix cache hit rate: 0.1% Apr 22 15:46:29 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:44422 - "POST /v1/chat/completions HTTP/1.1" 200 OK Apr 22 15:46:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:46:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 33.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.6%, Prefix cache hit rate: 0.1% Apr 22 15:46:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:46:41 [loggers.py:259] Engine 000: Avg prompt throughput: 759.4 tokens/s, Avg generation throughput: 12.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 41.7%, Prefix cache hit rate: 0.1% Apr 22 15:46:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:46:51 [loggers.py:259] Engine 000: Avg prompt throughput: 252.3 tokens/s, Avg generation throughput: 95.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 41.3%, Prefix cache hit rate: 0.1% Apr 22 15:47:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:47:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 95.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.1%, Prefix cache hit rate: 0.1% Apr 22 15:47:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:47:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 95.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.1%, Prefix cache hit rate: 0.1% Apr 22 15:47:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:47:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 95.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.1%, Prefix cache hit rate: 0.1% Apr 22 15:47:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:47:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 95.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.1%, Prefix cache hit rate: 0.1% Apr 22 15:47:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:47:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 95.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.1%, Prefix cache hit rate: 0.1% Apr 22 15:47:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:47:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 95.0 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.4%, Prefix cache hit rate: 0.1% Apr 22 15:48:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:48:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 95.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 42.4%, Prefix cache hit rate: 0.1% Apr 22 15:48:07 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:37676 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:48:07 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:36810 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:48:08 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:36800 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:48:08 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:36818 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:48:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:48:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 57.6 tokens/s, Running: 5 reqs, Waiting: 3 reqs, GPU KV cache usage: 13.7%, Prefix cache hit rate: 0.1% Apr 22 15:48:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:48:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.4 tokens/s, Running: 5 reqs, Waiting: 3 reqs, GPU KV cache usage: 14.4%, Prefix cache hit rate: 0.2% Apr 22 15:48:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:48:31 [loggers.py:259] Engine 000: Avg prompt throughput: 1873.3 tokens/s, Avg generation throughput: 100.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 0.2% Apr 22 15:48:37 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:37676 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK Apr 22 15:48:37 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:36818 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK Apr 22 15:48:37 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:37676 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK Apr 22 15:48:38 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:39638 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK Apr 22 15:48:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:48:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 140.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.6%, Prefix cache hit rate: 0.2% Apr 22 15:48:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:36818 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:48:43 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:39638 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:48:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:48:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.8 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 24.0%, Prefix cache hit rate: 0.3% Apr 22 15:49:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:49:01 [loggers.py:259] Engine 000: Avg prompt throughput: 3452.4 tokens/s, Avg generation throughput: 3.7 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 29.9%, Prefix cache hit rate: 0.3% Apr 22 15:49:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:49:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 35.1%, Prefix cache hit rate: 0.3% Apr 22 15:49:14 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:36800 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK Apr 22 15:49:15 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:50256 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK Apr 22 15:49:16 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:50256 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:49:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:49:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 40.2%, Prefix cache hit rate: 0.4% Apr 22 15:49:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:49:31 [loggers.py:259] Engine 000: Avg prompt throughput: 4690.1 tokens/s, Avg generation throughput: 4.8 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 44.3%, Prefix cache hit rate: 0.4% Apr 22 15:49:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:49:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 48.0%, Prefix cache hit rate: 0.4% Apr 22 15:49:44 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:36810 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:49:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:49:51 [loggers.py:259] Engine 000: Avg prompt throughput: 4094.6 tokens/s, Avg generation throughput: 24.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 52.4%, Prefix cache hit rate: 0.4% Apr 22 15:50:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:50:01 [loggers.py:259] Engine 000: Avg prompt throughput: 2968.3 tokens/s, Avg generation throughput: 50.7 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 54.6%, Prefix cache hit rate: 0.4% Apr 22 15:50:08 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:39638 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:50:09 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:50256 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK Apr 22 15:50:09 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:47510 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK Apr 22 15:50:09 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:36818 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:50:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:50:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 92.0 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 36.2%, Prefix cache hit rate: 0.4% Apr 22 15:50:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:47510 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:50:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:50:21 [loggers.py:259] Engine 000: Avg prompt throughput: 1178.6 tokens/s, Avg generation throughput: 3.2 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 42.1%, Prefix cache hit rate: 0.7% Apr 22 15:50:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:50:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.2 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 47.2%, Prefix cache hit rate: 1.0% Apr 22 15:50:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:50:41 [loggers.py:259] Engine 000: Avg prompt throughput: 4274.3 tokens/s, Avg generation throughput: 3.8 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 53.1%, Prefix cache hit rate: 1.0% Apr 22 15:50:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:50:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.2 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 57.6%, Prefix cache hit rate: 1.0% Apr 22 15:50:55 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:36810 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK Apr 22 15:50:57 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:36810 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:51:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:51:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.9 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 53.5%, Prefix cache hit rate: 1.1% Apr 22 15:51:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:51:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 57.2%, Prefix cache hit rate: 1.1% Apr 22 15:51:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:51:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 62.4%, Prefix cache hit rate: 1.2% Apr 22 15:51:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:51:31 [loggers.py:259] Engine 000: Avg prompt throughput: 7459.1 tokens/s, Avg generation throughput: 4.8 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 66.8%, Prefix cache hit rate: 1.2% Apr 22 15:51:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:51:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.5%, Prefix cache hit rate: 1.2% Apr 22 15:51:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:51:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.2%, Prefix cache hit rate: 1.2% Apr 22 15:52:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:52:01 [loggers.py:259] Engine 000: Avg prompt throughput: 6176.3 tokens/s, Avg generation throughput: 80.5 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 74.9%, Prefix cache hit rate: 1.2% Apr 22 15:52:05 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:36818 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK Apr 22 15:52:05 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:39638 - "POST /v1/messages/count_tokens?beta=true HTTP/1.1" 200 OK Apr 22 15:52:07 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:55446 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:52:10 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:39638 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:52:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:52:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 81.2 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 33.9%, Prefix cache hit rate: 1.2% Apr 22 15:52:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO: 192.168.1.108:47510 - "POST /v1/messages?beta=true HTTP/1.1" 200 OK Apr 22 15:52:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:52:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 6 reqs, Waiting: 2 reqs, GPU KV cache usage: 38.4%, Prefix cache hit rate: 1.3% Apr 22 15:52:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:52:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 6 reqs, Waiting: 2 reqs, GPU KV cache usage: 42.8%, Prefix cache hit rate: 1.4% Apr 22 15:52:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:52:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.0 tokens/s, Running: 6 reqs, Waiting: 2 reqs, GPU KV cache usage: 45.8%, Prefix cache hit rate: 1.4% Apr 22 15:52:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:52:51 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.5 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 51.7%, Prefix cache hit rate: 1.5% Apr 22 15:53:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:53:01 [loggers.py:259] Engine 000: Avg prompt throughput: 7372.9 tokens/s, Avg generation throughput: 4.1 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 56.1%, Prefix cache hit rate: 1.6% Apr 22 15:53:11 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:53:11 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 59.8%, Prefix cache hit rate: 1.7% Apr 22 15:53:21 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:53:21 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 63.5%, Prefix cache hit rate: 1.7% Apr 22 15:53:31 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:53:31 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 3.0 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 67.2%, Prefix cache hit rate: 1.8%

Apr 22 15:53:41 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:53:41 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2.4 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.7%, Prefix cache hit rate: 1.9% Apr 22 15:53:51 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:53:51 [loggers.py:259] Engine 000: Avg prompt throughput: 8068.3 tokens/s, Avg generation throughput: 4.9 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 76.8%, Prefix cache hit rate: 1.9% Apr 22 15:54:01 gpu4090d bash[6277]: (APIServer pid=6277) INFO 04-22 15:54:01 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 4.2 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 81.2%, Prefix cache hit rate: 1.9%

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.