Problem Summary

Memory usage is higher for Ollama 0.17.7 compared to Ollama 0.9.5 for the same model under the same concurrency.

Root Cause Analysis

The root cause is likely due to changes in the underlying implementation or optimizations in the newer version of Ollama.

Fix Plan

1. Check for Memory Leaks

• Run the application under a memory profiler (e.g., Valgrind, VisualVM) to identify memory leaks.
• Use tools like mmap or pmap to check for memory usage.

2. Optimize Model Loading

• Check if the model is being loaded multiple times, causing memory duplication.
• Use a caching mechanism to load the model only once.

3. Reduce Data Structures

• Review the data structures used to store model data and reduce their size.
• Use more memory-efficient data structures like std::vector instead of std::list.

4. Optimize Concurrency

• Review the concurrency implementation and ensure it is optimized for memory usage.
• Use thread-local storage to reduce memory usage.

Example Code (Optimize Model Loading)

// Before
std::string loadModel() {
  // Load model from file
  return model_data;
}

// After
std::string loadModel() {
  static std::string model_data; // Use static to cache the model
  if (model_data.empty()) {
    // Load model from file
    model_data = ...;
  }
  return model_data;
}

Example Code (Reduce Data Structures)

// Before
std::list<std::pair<int, float>> model_data;

// After
std::vector<std::pair<int, float>> model_data;

Verification

• Monitor memory usage under the same concurrency as before.
• Compare memory usage between Ollama 0.17.7