ollama - 💡(How to fix) Fix Ollama fails to use a coding model that doesn't fit into Vulkan memory [4 comments, 2 participants]

Fix Plan

To resolve the issue, we need to implement a hybrid approach where Ollama uses both the Vulkan device and CPU for model computation. This can be achieved by:

• Modifying the Ollama configuration to specify the number of layers to be loaded onto the Vulkan device
• Implementing a fallback mechanism to use the CPU for the remaining layers

Step-by-Step Solution

1. Update Ollama configuration: Modify the ollama.yaml file to include the vulkan_layers parameter, specifying the number of layers to be loaded onto the Vulkan device. For example:

   vulkan_layers: 10

2. Implement hybrid computation: Update the Ollama code to use the Vulkan device for the specified number of layers and fall back to CPU for the remaining layers. This can be achieved by modifying the load_model function to:

   def load_model(model_name, vulkan_layers):
       # Load vulkan_layers onto the Vulkan device
       vulkan_model = load_vulkan_model(model_name, vulkan_layers)
       
       # Load remaining layers onto the CPU
       cpu_model = load_cpu_model(model_name, vulkan_layers)
       
       # Combine Vulkan and CPU models for hybrid computation
       hybrid_model = combine_models(vulkan_model, cpu_model)
       
       return hybrid_model

3. Verify the fix: Restart the Ollama server and monitor the logs to ensure that the model is loaded correctly and computation starts within a reasonable time frame.

Example Code

import torch

def load_vulkan_model(model_name, vulkan_layers):
    # Load the specified number of layers onto the Vulkan device
    model = torch.load(model_name, map_location='cuda:0')
    vulkan_model = model[:vulkan_layers]
    return vulkan_model

def load_cpu_model(model_name, vulkan_layers):
    # Load the remaining layers onto the CPU
    model = torch.load(model_name, map_location='cpu')
    cpu_model = model[vulkan_layers:]
    return cpu_model

def combine_models(vulkan_model, cpu_model):
    # Combine the Vulkan and CPU models for hybrid computation
    hybrid_model = torch.nn.Sequential(vulkan_model, cpu_model)
    return hybrid_model