ollama - 💡(How to fix) Fix Jetson Orin Nano 8GB: improve defaults and diagnostics for reliable CUDA inference [4 comments, 3 participants]

Summary

Jetson users still run into recurring CUDA-init and runtime stability issues despite successful installs. I’d like to contribute a focused set of improvements for Orin-class devices (especially 8GB) that reduce setup friction and improve reliability without changing behavior on non-Jetson systems.

Motivation

There are multiple historical and active Jetson/Nvidia reports around detection, compatibility, and stability under load (for example #9503 and related Nvidia-labeled issues).

Proposed scope

1. Docker JetPack auto-detection fallback

   • Infer JetPack major from /etc/nv_tegra_release
   • Preserve explicit JETSON_JETPACK override
   • Keep current behavior if detection is unavailable

2. Scheduler warning guardrails for low-memory devices

   • Warn when effective memory pressure is likely unsafe (for example high OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH combinations)
   • Suggest safer values in warning text
   • Non-breaking: warning-only, no hard fail

3. Jetson docs refresh (native + Docker)

   • Clear CUDA verification commands
   • expected log signatures for successful Jetson CUDA path
   • profile guidance for 8GB-class systems

Validation environment available

• Device: Jetson Orin Nano 8GB
• JetPack: 6.x
• Ollama: 0.21.0
• Verified CUDA runtime path via journalctl (inference compute ... library=CUDA ... Orin)

Contribution plan

If this direction is welcome, I can submit PRs in small sequence:

1. docs + verification improvements
2. Docker JetPack autodetect fallback
3. scheduler warning guardrails

Happy to adjust scope to maintainer preference before opening PR 1.