Support for Multiple Ollama Backends (Cluster-Aware Routing & Scheduling)

[celevra]

Feature Request: Support for Multiple Ollama Backends

Currently, ollama_smart_proxy appears to operate against a single Ollama instance. This limits its usefulness in distributed setups where multiple Ollama servers are available.

Motivation

In real-world deployments, users often run multiple Ollama instances across different machines, GPUs, or hosts.

Typical challenges:

• Models are loaded dynamically on different nodes
• Some nodes are temporarily saturated
• Model reloads are expensive
• Requests should ideally go to a node where the requested model is already loaded

Proposed Feature

Add first-class support for multiple Ollama backends.

Example configuration:

backends:
  - name: ollama1
    api_base: http://ollama1:11434
    total_vram_mb: 24000

  - name: ollama2
    api_base: http://ollama2:11434
    total_vram_mb: 48000

  - name: ollama3
    api_base: http://ollama3:11434
    total_vram_mb: 16000

Desired Routing Strategies

• least-loaded
• model-affinity
• fastest-response
• weighted

Expected Behavior

The proxy should track across all backends:

• Which models are currently loaded
• VRAM usage per node
• Active request count
• Queue length
• Backend health

This would allow the proxy to:

• Prefer nodes where the requested model is already loaded
• Avoid unnecessary model reloads
• Fail over automatically if a backend is unavailable
• Route new requests to the most suitable Ollama instance

Observability

It would be useful if the UI/metrics exposed:

• Per-node utilization
• Model distribution across nodes
• Queue length per node
• Routing decisions
• Backend health status

Why This Fits the Project

ollama_smart_proxy already provides:

• VRAM-aware scheduling
• Queue management
• Model affinity

Extending these capabilities across multiple Ollama backends would make it a true cluster-aware scheduler for Ollama.

Alternatives Considered

• LiteLLM Proxy: good API gateway, but not deeply VRAM/model-aware
• External load balancers: lack model-level intelligence
• Custom orchestration: duplicates logic already present in this project

Impact

This would enable:

• Horizontal scaling of Ollama deployments
• Better utilization of multiple machines/GPUs
• Lower latency by reducing unnecessary model reloads
• More production-ready Ollama cluster deployments

Happy to contribute feedback or help test this if it aligns with the project roadmap.