Issue Station
Sign In
Tools hereToolbox

crewai - ✅(Solved) Fix [BUG] Remote Ollama url ignored by litellm [1 pull requests, 1 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
crewAIInc/crewAI#4694•Fetched 2026-04-08 00:40:40
View on GitHub
Comments
1
Participants
2
Timeline
8
Reactions
0
Author
Killian-falKillian-fal
Participants
github-actions[bot]github-actions[bot]Killian-falKillian-fal
Timeline (top)
referenced ×3cross-referenced ×2labeled ×2commented ×1

When using CrewAI with a remote Ollama server (not localhost) and a Task with output_pydantic, two failures occur:

  1. The api_base/base_url is not forward to LiteLLM. Even though the LLM object is configured with a remote URL. (I analyse and found that InternalInstructor calls litellm.completion without those parameters, causing LiteLLM to fall back to http://localhost:11434)

  2. LLM.supports_function_calling() always returns False for remote Ollama, because LiteLLM internally calls http://localhost:11434/api/show (ignoring api_base) to check model capabilities. This forces CrewAI to use the ReAct text-based loop instead of native function calling, even for models that fully support it.

Error Message

<last_exception> litellm.APIConnectionError: Ollama_chatException - {"error":"model 'mistral-small3.2:24b' not found"} </last_exception> Traceback (most recent call last): File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 174, in _make_common_sync_call response = sync_httpx_client.post( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py", line 780, in post raise e File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py", line 762, in post response.raise_for_status() File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/httpx/_models.py", line 829, in raise_for_status raise HTTPStatusError(message, request=request, response=self) httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://localhost:11434/api/chat' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

Root Cause

  1. LLM.supports_function_calling() always returns False for remote Ollama, because LiteLLM internally calls http://localhost:11434/api/show (ignoring api_base) to check model capabilities. This forces CrewAI to use the ReAct text-based loop instead of native function calling, even for models that fully support it.

Fix Action

Fixed

PR fix notes

PR #4696: fix: support remote Ollama server for function calling and pydantic output (#4694)

Description (problem / solution / changelog)

fix: support remote Ollama server for function calling and pydantic output

Fixes #4694

Summary

Two bugs prevented CrewAI from working with remote (non-localhost) Ollama servers:

Bug 1 — InternalInstructor.to_pydantic() loses remote URL: The instructor's completion call to litellm did not forward api_base/base_url/api_key from the LLM object, so litellm defaulted to localhost and failed to reach the remote Ollama server during structured output parsing. Fixed by adding _get_llm_extra_kwargs() which extracts these params and splats them into the create() call.

Bug 2 — LLM.supports_function_calling() always returns False for remote Ollama: litellm.utils.supports_function_calling() internally tries to query localhost:11434 to check model capabilities, which fails when Ollama runs on a different host. Fixed by adding a fallback that queries the remote Ollama /api/show endpoint directly, inspecting model_info keys and the template string for tool support indicators.

Files changed

  • lib/crewai/src/crewai/utilities/internal_instructor.py — new _get_llm_extra_kwargs() helper + forwarding in to_pydantic()
  • lib/crewai/src/crewai/llm.py — new _is_ollama_model(), _get_ollama_base_url(), _check_ollama_function_calling() helpers + fallback in supports_function_calling()
  • lib/crewai/tests/test_remote_ollama.py — 31 new unit tests covering both bugs

Review & Testing Checklist for Human

  • Verify tool-detection heuristics against real Ollama responses: _check_ollama_function_calling() checks for "tool" in key.lower() and value is True in model_info and "tools"/".ToolCalls" in the template string. Confirm these patterns match actual /api/show responses for tool-capable models (e.g., mistral-small3.2, qwen2.5) and don't false-positive on non-tool models.
  • Synchronous HTTP call in hot path: _check_ollama_function_calling() makes a blocking httpx.post with 5s timeout. This only fires when litellm returns False for an Ollama model with a remote URL, but verify this doesn't introduce unacceptable latency in your workflows.
  • Test with a real remote Ollama server: All tests use mocks. Spin up an Ollama server on a different host (or Docker container with exposed port) and verify:
    • LLM.supports_function_calling() returns True for tool-capable models like ollama_chat/mistral-small3.2:24b
    • InternalInstructor.to_pydantic() successfully parses structured output from a remote Ollama model
    • Non-tool models (e.g., ollama/llama2) correctly return False
  • Edge cases: Test with trailing slashes in api_base (should be stripped), unreachable servers (should return False without crashing), and localhost URLs (should behave as before).

Notes

  • Test file uses object.__new__() to bypass InternalInstructor.__init__ and avoid the dynamic instructor import; this doesn't test the real initialization path but allows testing the helper methods in isolation.
  • The unused pytest import can be removed if desired (no parametrize or raises used).
  • Session: https://app.devin.ai/sessions/1aecc957ee64482b984d1bb06283ac54
  • Requested by: João
<!-- CURSOR_SUMMARY -->

[!NOTE] Medium Risk Adds a new synchronous HTTP capability probe to LLM.supports_function_calling() for remote Ollama and changes instructor completion kwargs forwarding; risk is moderate due to potential latency/false positives and altered request parameters.

Overview Fixes remote Ollama usage by ensuring structured-output (InternalInstructor.to_pydantic) forwards provider connection kwargs (api_base, base_url, api_key) into the litellm create() call.

Adds an Ollama-specific fallback in LLM.supports_function_calling() that, when litellm reports no tool support and a remote URL is configured, queries the remote Ollama /api/show endpoint to infer tool/function-calling capability.

Introduces a new test_remote_ollama.py suite covering both behaviors (kwargs forwarding and remote capability probing) with mocked httpx/litellm interactions.

<sup>Written by Cursor Bugbot for commit 9bdc7b9eefd2cc5e2000eede026f92eec4f0ee35. This will update automatically on new commits. Configure here.</sup>

<!-- /CURSOR_SUMMARY -->

Changed files

  • lib/crewai/src/crewai/llm.py (modified, +72/-1)
  • lib/crewai/src/crewai/utilities/internal_instructor.py (modified, +29/-1)
  • lib/crewai/tests/test_remote_ollama.py (added, +473/-0)

Code Example

from crewai import LLM, Agent, Task, Crew
from pydantic import BaseModel

class MyOutput(BaseModel):
    result: str

llm = LLM(
    model="ollama_chat/mistral-small3.2:24b",
    base_url="https://my-remote-ollama.example.com",
    api_base="https://my-remote-ollama.example.com",
)

agent = Agent(role="Assistant", goal="Answer", backstory="...", llm=llm)
task = Task(description="Say hello", expected_output="A greeting", agent=agent, output_pydantic=MyOutput)
crew = Crew(agents=[agent], tasks=[task])
crew.kickoff() # same with async

---

<last_exception>
      litellm.APIConnectionError: Ollama_chatException - {"error":"model 'mistral-small3.2:24b' not found"}
  </last_exception> Traceback (most recent call last):
    File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 174, in _make_common_sync_call
      response = sync_httpx_client.post(
                 ^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py", line 780, in post
      raise e
    File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py", line 762, in post
      response.raise_for_status()
    File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/httpx/_models.py", line 829, in raise_for_status
      raise HTTPStatusError(message, request=request, response=self)
  httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://localhost:11434/api/chat'
  For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

---

def _get_llm_extra_kwargs(self) -> dict[str, Any]:
    if self.llm is None:
        return {}
    extra: dict[str, Any] = {}
    for attr in ("api_base", "base_url"): # add other maybe ?
        value = getattr(self.llm, attr, None)
        if value is not None:
            extra[attr] = value
    return extra

def to_pydantic(self) -> T:
    # ...
    return self._client.chat.completions.create(
        model=model_name, response_model=self.model, messages=messages,
        **self._get_llm_extra_kwargs(),  # FIX HERE
    )

---

llm.supports_function_calling = lambda: True
RAW_BUFFERClick to expand / collapse

Description

When using CrewAI with a remote Ollama server (not localhost) and a Task with output_pydantic, two failures occur:

  1. The api_base/base_url is not forward to LiteLLM. Even though the LLM object is configured with a remote URL. (I analyse and found that InternalInstructor calls litellm.completion without those parameters, causing LiteLLM to fall back to http://localhost:11434)

  2. LLM.supports_function_calling() always returns False for remote Ollama, because LiteLLM internally calls http://localhost:11434/api/show (ignoring api_base) to check model capabilities. This forces CrewAI to use the ReAct text-based loop instead of native function calling, even for models that fully support it.

Steps to Reproduce

from crewai import LLM, Agent, Task, Crew
from pydantic import BaseModel

class MyOutput(BaseModel):
    result: str

llm = LLM(
    model="ollama_chat/mistral-small3.2:24b",
    base_url="https://my-remote-ollama.example.com",
    api_base="https://my-remote-ollama.example.com",
)

agent = Agent(role="Assistant", goal="Answer", backstory="...", llm=llm)
task = Task(description="Say hello", expected_output="A greeting", agent=agent, output_pydantic=MyOutput)
crew = Crew(agents=[agent], tasks=[task])
crew.kickoff() # same with async

Expected behavior

  1. Bug 1 (api_base): Use a remote Ollama server via base_url/api_base Run the crew -> observe that InternalInstructor hits localhost:11434 instead of the remote URL

  2. Bug 2 (supports_function_calling): Use a remote Ollama server via base_url/api_base Run the crew -> check llm.supports_function_calling() -> returns False even for models like mistral-small3.2 that natively support function calling

Screenshots/Code snippets

X

Operating System

Ubuntu 20.04

Python Version

3.11

crewAI Version

1.10.0

crewAI Tools Version

1.10.0

Virtual Environment

Venv

Evidence

<last_exception>
      litellm.APIConnectionError: Ollama_chatException - {"error":"model 'mistral-small3.2:24b' not found"}
  </last_exception> Traceback (most recent call last):
    File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/litellm/llms/custom_httpx/llm_http_handler.py", line 174, in _make_common_sync_call
      response = sync_httpx_client.post(
                 ^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py", line 780, in post
      raise e
    File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py", line 762, in post
      response.raise_for_status()
    File "/home/kfalguiere/Workspace/Beyond/afe/interfaceafe_final/.venv/lib/python3.11/site-packages/httpx/_models.py", line 829, in raise_for_status
      raise HTTPStatusError(message, request=request, response=self)
  httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://localhost:11434/api/chat'
  For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

Possible Solution

To fix in my project I did that :

  1. Bug 1 - in internal_instructor.py
def _get_llm_extra_kwargs(self) -> dict[str, Any]:
    if self.llm is None:
        return {}
    extra: dict[str, Any] = {}
    for attr in ("api_base", "base_url"): # add other maybe ?
        value = getattr(self.llm, attr, None)
        if value is not None:
            extra[attr] = value
    return extra

def to_pydantic(self) -> T:
    # ...
    return self._client.chat.completions.create(
        model=model_name, response_model=self.model, messages=messages,
        **self._get_llm_extra_kwargs(),  # FIX HERE
    )
  1. Bug 2 I found nothing sorry, I juste override the function in my code
llm.supports_function_calling = lambda: True

Additional context

X

extent analysis

Fix Plan

To fix the issues with CrewAI and remote Ollama server, follow these steps:

  1. Fix Bug 1 (api_base): Modify the internal_instructor.py file to include api_base and base_url in the extra dictionary.

def _get_llm_extra_kwargs(self) -> dict[str, Any]: if self.llm is None: return {} extra: dict[str, Any] = {} for attr in ("api_base", "base_url"): value = getattr(self.llm, attr, None) if value is not None: extra[attr] = value return extra

def to_pydantic(self) -> T: # ... return self._client.chat.completions.create( model=model_name, response_model=self.model, messages=messages, **self._get_llm_extra_kwargs(),
)


2. **Fix Bug 2 (supports_function_calling)**: Override the `supports_function_calling` method to return `True` for models that support function calling.
   ```python
llm.supports_function_calling = lambda: True

Alternatively, you can modify the supports_function_calling method to check the model capabilities correctly.

Verification

To verify that the fixes work:

  1. Run the crew with the modified internal_instructor.py file.
  2. Check that the InternalInstructor hits the remote URL instead of http://localhost:11434.
  3. Verify that llm.supports_function_calling() returns True for models that support function calling.

Extra Tips

  • Make sure to update the crewAI and crewAI Tools versions to the latest ones.
  • If you encounter any issues, check the documentation for crewAI and Ollama for troubleshooting guides.
  • Consider submitting a pull request to the crewAI repository with the fixes to help others who may encounter the same issues.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

FAQ

Expected behavior

  1. Bug 1 (api_base): Use a remote Ollama server via base_url/api_base Run the crew -> observe that InternalInstructor hits localhost:11434 instead of the remote URL

  2. Bug 2 (supports_function_calling): Use a remote Ollama server via base_url/api_base Run the crew -> check llm.supports_function_calling() -> returns False even for models like mistral-small3.2 that natively support function calling

#api#ssr#installation#tensor shape#autograd error#environment setup#docker error#permission error#memory optimization

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING