ollama - 💡(How to fix) Fix Not work on rx9060xt [2 comments, 3 participants]

Even with OLLAMA_VULKAN=1, the runner falls back to CPU: level=INFO source=ggml.go:494 msg="offloaded 0/35 layers to GPU" vulkaninfo correctly identifies the device: deviceName = AMD Radeon RX 9060 XT (RADV GFX1200)

Fix Plan

To resolve the issue of Ollama failing to offload layers to the GPU on the new AMD Radeon RX 9060 XT, we need to update the supported GPU discovery list in the llama.cpp/ggml integration to include the GFX1200 architecture.

Steps to Fix

• Update the llama.cpp file to include the GFX1200 architecture in the supported GPU list.
• Recompile the ggml library with the updated llama.cpp file.
• Set the OLLAMA_VULKAN environment variable to 1 to enable Vulkan support.

Example Code Changes

// In llama.cpp, add the following line to the supported GPU list
supported_gpus.push_back("GFX1200"); // Add GFX1200 architecture

// Ensure the Vulkan library is linked correctly
#include <vulkan/vulkan.h>

// Initialize the Vulkan instance
VkInstance instance;
VkApplicationInfo app_info = {};
app_info.sType = VK_STRUCTURE_TYPE_APPLICATION_INFO;
app_info.pApplicationName = "Ollama";
app_info.applicationVersion = VK_MAKE_VERSION(1, 0, 0);
app_info.pEngineName = "Ollama Engine";
app_info.engineVersion = VK_MAKE_VERSION(1, 0, 0);
app_info.apiVersion = VK_API_VERSION_1_2;

VkInstanceCreateInfo create_info = {};
create_info.sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO;
create_info.pApplicationInfo = &app_info;

vkCreateInstance(&create_info, nullptr, &instance);

Verification

• Run vulkaninfo to verify that the device is correctly identified.
• Set OLLAMA_VULKAN=1 and run Ollama to verify that layers are offloaded to the GPU.
• Check the log output for level=INFO source=ggml.go:494 msg="offloaded X/35 layers to GPU" where X is greater than 0.

Extra Tips

• Ensure that the mesa-git package is up-to-date, as it may contain fixes for the GFX1200 architecture.
• If issues persist, try setting the VK_LAYER_KHRONOS_validation environment variable to 1 to enable Vulkan validation layers.