ollama - 💡(How to fix) Fix is there something wrong with the models folder setting [1 participants]

Fix Plan

To resolve the issue of ollama.exe loading models into 100% CPU and the discrepancy in model listing between ollama.exe and ollama serve, follow these steps:

• Configure Ollama to use GPU: Ensure that Ollama is configured to utilize the GPU for model loading and inference. This can be achieved by setting the OLLAMA_DEVICE environment variable to cuda (for Nvidia GPUs) before running ollama.exe or ollama serve.
• Specify model directory: Use the --models-dir flag with ollama serve to specify the custom model directory (D:\.ollama\models\) to ensure that all models are listed and loaded correctly.
• Update Ollama configuration: Modify the Ollama configuration file to include the custom model directory and GPU settings. This can be done by adding the following lines to the configuration file:

device: cuda
models_dir: D:\.ollama\models\

• Verify GPU usage: Use tools like nvidia-smi to monitor GPU usage and ensure that Ollama is utilizing the GPU for model loading and inference.

Example Code

To set the OLLAMA_DEVICE environment variable and specify the custom model directory, use the following commands:

set OLLAMA_DEVICE=cuda
ollama serve --models-dir D:\.ollama\models\

Alternatively, you can modify the Ollama configuration file to include the custom settings:

# ollama.yml
device: cuda
models_dir: D:\.ollama\models\

Then, run ollama serve without specifying the --models-dir flag.

Verification

To verify that the fix worked, monitor the CPU and GPU usage using tools like taskmgr and nvidia-smi. Ensure that Ollama is utilizing the GPU for model loading and inference, and that all models are listed correctly when running ollama ls.