vllm - 💡(How to fix) Fix [Usage]: Run:ai S3 streamer crashing when loading model from an S3 compatible object storage [1 comments, 1 participants]

(Worker pid=1254) (Worker_TP0_DCP0 pid=1254) Loading safetensors using Runai Model Streamer: 6% Completed | 12127/208550 [00:13<03:16, 1001.31it/s] (Worker pid=1254) (Worker_TP0_DCP0 pid=1254) Loading safetensors using Runai Model Streamer: 7% Completed | 14247/208550 [00:15<03:20, 970.14it/s] (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) [2026-04-09 08:00:15] ERROR distributed_streamer.py:446: [RunAI Streamer][Distributed] rank 6 error: Could not receive runai_call response (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) Traceback (most recent call last): (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/distributed_streamer/distributed_streamer.py", line 414, in get_chunks (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) current_data_size, chunk_count_in_batch = self.prefill( (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/distributed_streamer/distributed_streamer.py", line 469, in prefill (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) chunk_item = next(chunk_gen) (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/distributed_streamer/distributed_streamer.py", line 403, in chunk_generator (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) yield from ready_chunks_iterator (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/file_streamer/file_streamer.py", line 127, in get_chunks (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) yield from self.request_ready_chunks() (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/file_streamer/file_streamer.py", line 148, in request_ready_chunks (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) file_relative_index, chunk_relative_index = runai_response(self.streamer) (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/libstreamer/libstreamer.py", line 89, in runai_response (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) raise ValueError(f"Could not receive runai_response from libstreamer due to: {error_msg}") (Worker pid=1260) (Worker_TP6_DCP6 pid=1260) ValueError: Could not receive runai_response from libstreamer due to: b'File access error'

(Worker pid=1258) (Worker_TP4_DCP4 pid=1258) [2026-04-09 08:00:15] ERROR distributed_streamer.py:446: [RunAI Streamer][Distributed] rank 4 error: Could not receive runai_call response (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) Traceback (most recent call last): (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/distributed_streamer/distributed_streamer.py", line 414, in get_chunks (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) current_data_size, chunk_count_in_batch = self.prefill( (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/distributed_streamer/distributed_streamer.py", line 469, in prefill (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) chunk_item = next(chunk_gen) (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/distributed_streamer/distributed_streamer.py", line 403, in chunk_generator (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) yield from ready_chunks_iterator (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/file_streamer/file_streamer.py", line 127, in get_chunks (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) yield from self.request_ready_chunks() (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/file_streamer/file_streamer.py", line 148, in request_ready_chunks (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) file_relative_index, chunk_relative_index = runai_response(self.streamer) (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/libstreamer/libstreamer.py", line 89, in runai_response (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) raise ValueError(f"Could not receive runai_response from libstreamer due to: {error_msg}") (Worker pid=1258) (Worker_TP4_DCP4 pid=1258) ValueError: Could not receive runai_response from libstreamer due to: b'File access error'

Loading safetensors using Runai Model Streamer: 7% Completed | 14247/208550 [00:30<03:20, 970.14it/s]

TL;DR

Consider adjusting the --tensor-parallel-size and --decode-context-parallel-size parameters to reduce the load on S3 when launching multiple servers simultaneously.

Guidance

• Review the S3 bucket configuration to ensure it can handle the increased load when multiple servers are launched at the same time.
• Experiment with reducing the --tensor-parallel-size and --decode-context-parallel-size parameters to decrease the parallelism and alleviate the load on S3.
• Monitor the S3 throughput and adjust the parameters accordingly to find a balance between performance and reliability.
• Consider implementing a queueing system or a load balancer to manage the incoming requests and prevent overwhelming the S3 bucket.

Example

No specific code example is provided, but the command used to serve the model can be modified to adjust the parallelism parameters, for example:

vllm serve s3://hf-cache/kimi-k2.5 \
    --port "8000" \
    --host 0.0.0.0 \
    --load-format runai_streamer \
    --model-loader-extra-config '{"distributed":true,"memory_limit":10000000000}' \
    --tool-call-parser kimi_k2 \
    --reasoning-parser kimi_k2 \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tensor-parallel-size 4 \
    --max-model-len 262144 \
    --decode-context-parallel-size 4

Notes

The issue seems to be related to the increased load on S3 when multiple servers are launched simultaneously, causing a decrease in throughput and resulting in errors. Adjusting the parallelism parameters and reviewing the S3 bucket configuration may help alleviate the issue.

Recommendation

Apply a workaround by adjusting the --tensor-parallel-size and --decode-context-parallel-size parameters to reduce the load on S3, and monitor the performance to find the optimal balance between parallelism and reliability.