ollama - 💡(How to fix) Fix is it possibel to run both m60 gpus in linux? [7 comments, 2 participants]

CUDA error: an illegal memory access was encountered current device: 0, in function ggml_backend_cuda_synchronize at /ml/backend/ggml/ggml/src/ggml-cuda/ggml-cuda.cu:2981 cudaStreamSynchronize(cuda_ctx->stream())

Fix Plan

The issue seems to be related to CUDA compatibility and memory access. To fix this, we'll try updating CUDA and cuDNN versions, and also modify the code to handle memory access correctly.

• Update CUDA to the latest version (e.g., 12.0) and install the corresponding cuDNN version.
• Modify the ggml-cuda.cu file to add error checking for cudaStreamSynchronize:

cudaError_t err = cudaStreamSynchronize(cuda_ctx->stream());
if (err != cudaSuccess) {
    printf("CUDA error: %s\n", cudaGetErrorString(err));
    // Handle the error or exit the program
}

• Try running the program with the CUDA_DEVICE_ORDER environment variable set to PCI_BUS_ID to ensure the GPUs are detected correctly:

export CUDA_DEVICE_ORDER=PCI_BUS_ID

• If using Docker, make sure to install the NVIDIA driver and CUDA toolkit inside the container, and mount the GPU devices correctly.

Verification

To verify the fix, run the program with the modified code and check for any CUDA errors. If the program runs without errors, it should be able to utilize both GPUs correctly.

Extra Tips

• Make sure to install the correct version of cuDNN for the CUDA version being used.
• If using a Docker container, ensure that the container has the necessary dependencies and environment variables set.
• Consider using a tool like nvidia-smi to monitor GPU usage and detect any potential issues.