vllm - 💡(How to fix) Fix [Feature]: Support Structured Output with Batched Prompts via OpenAI-Compatible API [1 comments, 1 participants]

Fix Plan

To add support for structured output with batched prompts in the OpenAI compatible API, we need to modify the API endpoint to accept and process JSON schema with batched chat compilation prompts.

Steps to Implement the Fix

• Modify the API endpoint to accept a JSON body with a list of prompts and a schema definition.
• Update the API handler to parse the JSON body and extract the prompts and schema.
• Use a library like jsonschema to validate the schema and ensure it conforms to the expected structure.
• Process the batched prompts using the validated schema and return the structured output.

Example Code

import json
from jsonschema import validate

# Define the expected schema
schema = {
    "type": "object",
    "properties": {
        "prompts": {"type": "array", "items": {"type": "string"}},
        "schema": {"type": "object"}
    },
    "required": ["prompts", "schema"]
}

# API endpoint handler
def handle_batched_prompts(request):
    data = json.loads(request.body)
    validate(instance=data, schema=schema)
    prompts = data["prompts"]
    schema_def = data["schema"]
    # Process the batched prompts using the schema
    output = []
    for prompt in prompts:
        # Compile the prompt using the schema
        compiled_prompt = compile_prompt(prompt, schema_def)
        output.append(compiled_prompt)
    return {"output": output}

# Compile a single prompt using the schema
def compile_prompt(prompt, schema_def):
    # Implement the compilation logic here
    pass

Verification

To verify the fix, send a curl request to the modified API endpoint with a JSON body containing a list of prompts and a schema definition. The response should contain the structured output with the compiled prompts.

curl -X POST \
  http://example.com/api/batched-prompts \
  -H 'Content-Type: application/json' \
  -d '{"prompts": ["prompt1", "prompt2"], "schema": {"key": "value"}}'

Extra Tips

• Ensure that the schema definition is properly validated to prevent errors during processing.
• Consider adding error handling to handle cases where the schema is invalid or the prompts cannot be compiled.
• Use a robust JSON schema validation library to ensure that the schema conforms to the expected structure.