hermes - 💡(How to fix) Fix Bug: vision route drops named custom provider and misroutes to default custom endpoint [1 comments, 2 participants]

Summary

When a vision task is configured with a named custom provider like custom:morecode-openai, the vision routing path normalizes it to bare custom, loses the provider name, and ends up using the default custom endpoint instead of the intended named provider.

In the reproduced case this caused requests to go to GitHub Copilot and fail with unauthorized: not authorized to use this Copilot feature, which makes the bug look like a token/license problem even though the real issue is provider misrouting.

Reproduction / evidence

Environment observed at runtime:

• AUXILIARY_VISION_PROVIDER=custom:morecode-openai
• AUXILIARY_VISION_MODEL=gemini-3.1-pro

Observed code path:

• tools/vision_tools.py -> async_call_llm(task="vision")
• agent/auxiliary_client.py:resolve_vision_provider_client()
• agent/auxiliary_client.py:_normalize_vision_provider()
• agent/auxiliary_client.py:_normalize_aux_provider(provider, for_vision=True)

Relevant current logic:

if normalized.startswith("custom:"):
    suffix = normalized.split(":", 1)[1].strip()
    if not suffix:
        return "custom"
    normalized = suffix if not for_vision else "custom"

Effect:

• text path preserves morecode-openai
• vision path collapses custom:morecode-openai -> custom
• vision then goes through _resolve_strict_vision_backend("custom") -> _try_custom_endpoint()
• the named custom provider branch is skipped entirely

Direct repro evidence from local instrumentation:

• RESOLVE_VISION_RESULT= custom https://api.githubcopilot.com gemini-3.1-pro
• subsequent error: unauthorized: not authorized to use this Copilot feature

Cross-check:

• directly resolving _get_cached_client('morecode-openai', 'gemini-3.1-pro', async_mode=True) returns the expected named provider base URL:
  • http://www.1314mc.net:1314/v1/

Root cause

The vision-specific provider normalization drops the named custom provider suffix, so the resolver cannot route through resolve_provider_client("morecode-openai", ...) and instead falls back to the generic custom endpoint.

Expected behavior

For vision tasks, custom:<name> should preserve <name> and resolve through the same named custom provider path used by non-vision tasks.

Minimal fix direction

In agent/auxiliary_client.py, stop collapsing custom:<name> to bare custom when for_vision=True.

Minimal candidate change:

if normalized.startswith("custom:"):
    suffix = normalized.split(":", 1)[1].strip()
    if not suffix:
        return "custom"
    normalized = suffix

Regression checks

1. With AUXILIARY_VISION_PROVIDER=custom:morecode-openai, resolve_vision_provider_client() should resolve to http://www.1314mc.net:1314/v1/, not https://api.githubcopilot.com.
2. Bare AUXILIARY_VISION_PROVIDER=custom with explicit base URL/key should still use the direct custom endpoint path.
3. AUXILIARY_VISION_PROVIDER=auto behavior should remain unchanged.

Notes

A local bug writeup with the detailed evidence chain exists here:

• ~/self-improving/bugs/vision-provider-token-401.md