ollama - 💡(How to fix) Fix Vulkan runtime, allow more than 64GB VRAM with splitted memory heaps. [1 participants]

TL;DR

• The issue can be mitigated by ensuring Ollama correctly handles the split memory heap reported by Vulkan, potentially by accounting for the device-local memory heap.

Guidance

• Review the Vulkan documentation to understand how to properly handle split memory heaps, especially when the device-local heap is involved.
• Verify that Ollama's memory allocation logic is compatible with the Vulkan memory model, particularly for devices with multiple memory heaps.
• Consider implementing a check to detect when the allocated memory exceeds the size of the first memory heap (27.36 GiB in this case), and adjust the allocation strategy accordingly to utilize the device-local heap.
• Investigate if there are any Vulkan extensions or features that can help manage or merge the memory heaps, potentially simplifying the allocation process.

Example

• No specific code example can be provided without more context on Ollama's implementation, but ensuring that memory allocations are made with awareness of the MEMORY_HEAP_DEVICE_LOCAL_BIT flag may be crucial.

Notes

• The exact solution may depend on the specifics of Ollama's Vulkan integration and how it manages memory allocations across different heaps.
• Understanding the implications of the MEMORY_HEAP_DEVICE_LOCAL_BIT flag on memory allocation and access patterns is essential for a proper fix.

Recommendation

• Apply workaround: Adjust Ollama's memory allocation logic to correctly handle the split memory heap scenario, ensuring it can utilize the device-local memory heap efficiently. This approach is recommended because it directly addresses the observed issue without requiring external updates or upgrades.