ollama - 💡(How to fix) Fix Running deepseek-r1:32b doesn't use my 3090s for inferencing anymore [3 comments, 2 participants]

Fix Plan

The issue seems to be related to the model not utilizing the GPU for inferencing. To fix this, we need to ensure that the model is configured to use the GPU.

Steps to Fix

• Check if the CUDA version is compatible with the Nvidia drivers.
• Verify that the deepseek-r1:32b model is configured to use the GPU.
• Update the Ollama configuration to specify the GPU devices.

Example Code

import torch

# Set the device to use the GPU
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Move the model to the GPU
model.to(device)

# Specify the GPU devices in the Ollama configuration
config = {
    "model": "deepseek-r1:32b",
    "device": "cuda:0",
    "gpu_devices": [0, 1]  # Use both GPU devices
}

Verification

To verify that the fix worked, check the GPU utilization using tools like nvidia-smi while running the model. The GPU usage should increase when the model is running.

Extra Tips

• Ensure that the Nvidia drivers are up-to-date.
• Check the Ollama documentation for any specific configuration options related to GPU usage.
• If using a multi-GPU setup, ensure that the model is configured to use both GPUs.