Fix Plan

To implement hot-swappable model loading in Ollama, follow these steps:

• Step 1: API Endpoints
  • Create API endpoints for loading, unloading, and listing models:

from flask import Flask, request, jsonify app = Flask(name)

Model management

models = {}

@app.route('/api/model/load', methods=['POST']) def load_model(): model_name = request.json['model'] # Load model into memory models[model_name] = load_model_into_memory(model_name) return jsonify({'message': f'Model {model_name} loaded'})

@app.route('/api/model/unload', methods=['POST']) def unload_model(): model_name = request.json['model'] # Unload model from memory del models[model_name] return jsonify({'message': f'Model {model_name} unloaded'})

@app.route('/api/models/loaded', methods=['GET']) def list_loaded_models(): return jsonify([{'name': model, 'memory': get_model_memory_usage(model)} for model in models])

* **Step 2: Configuration**
  * Add configuration options for maximum concurrent models and max VRAM per model:
    ```python
max_concurrent_models = 2
max_vram_per_model = 16  # GB

• Step 3: Memory Management
  • Implement LRU eviction when VRAM is exhausted:

from collections import OrderedDict

model_lru = OrderedDict()

def load_model_into_memory(model_name): # Load model into memory # ... model_lru[model_name] = time.time() if len(model_lru) > max_concurrent_models: # Evict least recently used model lru_model = next(iter(model_lru)) del model_lru[lru_model] del models[lru_model]

* **Step 4: Auto-loading**
  * Add parameter to API endpoint to auto-load model if not loaded:
    ```python
@app.route('/api/chat', methods=['POST'])
def chat():
    model_name = request.json.get('model')
    if model_name and model_name not in models:
        # Auto-load model
        load_model_into_memory(model_name)
    # ...

Verification

To verify that the fix worked, test the following scenarios:

• Load multiple models into memory using the /api/model/load endpoint
• List loaded models using the /api/models/loaded endpoint
• Unload a model using the /api/model/unload endpoint
• Test auto-loading by specifying a model in the /api/chat endpoint that is not loaded

Extra Tips

• Monitor VRAM usage and adjust configuration options as needed
• Implement priority system for sticky models that shouldn't be evicted
• Expose memory usage stats via API endpoint for monitoring and debugging purposes