vllm - 💡(How to fix) Fix Pooling API: expose extra_kwargs and allow nested response data for custom poolers [1 comments, 2 participants]

Fix Plan

To address the issue, we need to modify the PoolingCompletionRequest and PoolingChatRequest to expose extra_kwargs and pass it through to PoolingParams. Additionally, we need to relax the typing of PoolingResponseData.data to allow for 3D nested tensors.

Step-by-Step Solution

• Modify PoolingCompletionRequest and PoolingChatRequest to include extra_kwargs:

class PoolingCompletionRequest:
    def __init__(self, ..., extra_kwargs=None):
        self.extra_kwargs = extra_kwargs

class PoolingChatRequest:
    def __init__(self, ..., extra_kwargs=None):
        self.extra_kwargs = extra_kwargs

• Pass extra_kwargs through to PoolingParams:

class PoolingParams:
    def __init__(self, ..., extra_kwargs=None):
        self.extra_kwargs = extra_kwargs

# In PoolingCompletionRequest and PoolingChatRequest
def to_pooling_params(self):
    return PoolingParams(..., extra_kwargs=self.extra_kwargs)

• Relax typing of PoolingResponseData.data:

from typing import Any

class PoolingResponseData:
    def __init__(self, data: Any):
        self.data = data

Verification

To verify the fix, create a test case that passes a 3D nested tensor through the modified PoolingResponseData and checks that it can be serialized without validation failure.

Extra Tips

• Make sure to update the documentation to reflect the changes to the PoolingCompletionRequest and PoolingChatRequest classes.
• Consider adding additional tests to cover different scenarios and ensure the fix is robust.