LLM Providers & Routing

Octos supports 16 LLM providers out of the box. Each provider needs an API key stored in an environment variable (except local providers like Ollama and Vertex AI, which uses a service-account JSON).

Supported Providers

vertex authenticates with a Google service-account JSON (resolved via VERTEX_SA_JSON — keychain marker, config value, or env) instead of an API key; the GCP project is read from the JSON and the region is fixed to global. It must be selected explicitly (provider: "vertex") — bare gemini-* model names still resolve to the AI Studio gemini provider. r9s is a multi-protocol proxy that auto-detects the Anthropic Messages API for claude-* models and OpenAI Chat Completions otherwise.

Any other OpenAI- or Anthropic-compatible endpoint (e.g. wisemodel, Together, Fireworks, Azure) is reachable by setting base_url on a provider — see Custom Endpoints.

Configuration Methods

Config File

Set provider and model in your config.json:

{
  "provider": "moonshot",
  "model": "kimi-2.5",
  "api_key_env": "KIMI_API_KEY"
}

The api_key_env field overrides the default environment variable name for the provider. For example, Moonshot defaults to MOONSHOT_API_KEY, but you can point it at KIMI_API_KEY instead.

CLI Flags

Every provider setting is also an octos chat flag, for one-off runs without touching config:

# Known vendor (shorthand: supplies its default base URL + key env var)
octos chat --provider deepseek --model deepseek-chat

# Auto-detect the provider from the model name
octos chat --model gpt-4o

# Custom endpoint — name the real vendor, pick the wire protocol explicitly
octos chat --provider zai --api-type anthropic \
  --base-url https://api.z.ai/api/anthropic --model glm-5.2

# Full autonomy (bypass approvals + sandbox) alongside model selection
octos chat --yolo --provider zai --api-type anthropic \
  --base-url https://api.z.ai/api/anthropic --model glm-5.2

CLI flags override config, which overrides the built-in default. The API key is not a CLI flag — it comes from the auth store, config, or an environment variable (see Providing the API Key).

Auth Store

Instead of environment variables, you can store API keys through the auth CLI:

# OAuth PKCE (OpenAI)
octos auth login --provider openai

# Device code flow (OpenAI)
octos auth login --provider openai --device-code

# Paste-token (all other providers)
octos auth login --provider anthropic
# -> prompts: "Paste your API key:"

# Check stored credentials
octos auth status

# Remove credentials
octos auth logout --provider openai

Credentials are stored in ~/.octos/auth.json (file mode 0600). The auth store is checked before environment variables when resolving API keys.

Providing the API Key

There is no --api-key flag — the key is resolved, in order:

1. Auth store — octos auth login --provider <name> (stored once, in ~/.octos/auth.json).
2. Config — the env_vars map in config.json (below), or an api_key_env pointing at a variable.
3. Environment variable — whose name is the provider’s default: zai → ZAI_API_KEY, anthropic → ANTHROPIC_API_KEY, openai → OPENAI_API_KEY, deepseek → DEEPSEEK_API_KEY, … (see the table above).

# Quickest — export the provider's env var, then run
export ZAI_API_KEY=<your-key>
octos chat --provider zai --api-type anthropic \
  --base-url https://api.z.ai/api/anthropic --model glm-5.2

# Or log in once (no env var afterward)
octos auth login --provider zai      # prompts: "Paste your API key:"
octos auth status                    # which providers are logged in
octos auth keys                      # keys + keychain vs plaintext

Or bake it into config.json so nothing is needed at runtime:

{
  "provider": "zai",
  "model": "glm-5.2",
  "base_url": "https://api.z.ai/api/anthropic",
  "api_type": "anthropic",
  "env_vars": { "ZAI_API_KEY": "<your-key>" }
}

Passing a secret on the command line would land it in shell history and the process list; prefer octos auth login, an env var, or the config env_vars map.

Auto-Detection

When --provider is omitted, Octos infers the provider from the model name:

octos chat --model gpt-4o           # -> openai
octos chat --model claude-sonnet-4-20250514  # -> anthropic
octos chat --model deepseek-chat    # -> deepseek
octos chat --model glm-4-plus       # -> zhipu
octos chat --model qwen-max         # -> dashscope

Custom Endpoints

Use base_url to point at self-hosted or proxy endpoints:

{
  "provider": "openai",
  "model": "gpt-4o",
  "base_url": "https://your-azure-endpoint.openai.azure.com/v1"
}

{
  "provider": "ollama",
  "model": "llama3.2",
  "base_url": "http://localhost:11434/v1"
}

{
  "provider": "vllm",
  "model": "meta-llama/Llama-3-70b",
  "base_url": "http://localhost:8000/v1"
}

API Type Override

api_type forces a specific wire protocol when a custom base_url speaks a known format under a non-matching provider name — so you name the real vendor with provider and pick the protocol with api_type, rather than overloading provider with a protocol name.

In config:

{
  "provider": "zai",
  "model": "glm-5-turbo",
  "api_type": "anthropic"
}

Or on the command line with --api-type (alias --api-style), which overrides the config value:

octos chat --provider zai --api-type anthropic \
  --base-url https://api.z.ai/api/anthropic --model glm-5.2

• "openai" – OpenAI Chat Completions format (default for most providers)
• "anthropic" – Anthropic Messages format (for Anthropic-compatible proxies, e.g. z.ai/GLM)
• "responses" – OpenAI Responses API format

Fallback Chains

Configure a priority-ordered fallback chain. If the primary provider fails, the next provider in the list is tried automatically:

{
  "provider": "moonshot",
  "model": "kimi-2.5",
  "fallback_models": [
    {
      "provider": "deepseek",
      "model": "deepseek-chat",
      "api_key_env": "DEEPSEEK_API_KEY"
    },
    {
      "provider": "gemini",
      "model": "gemini-2.5-flash",
      "api_key_env": "GEMINI_API_KEY"
    }
  ]
}

Failover rules:

• 401/403 (authentication errors) – failover immediately, no retry on the same provider
• 429 (rate limit) / 5xx (server errors) – retry with exponential backoff, then failover
• 400 (content-format errors) – failover if the error contains “must not be empty”, “reasoning_content”, “API key not valid”, or “invalid_value”
• Timeouts – failover immediately, no retry (don’t waste 120s × retries on an unresponsive provider)
• Circuit breaker – 3 consecutive failures marks a provider as degraded

Adaptive Routing

When multiple fallback models are configured, adaptive routing dynamically selects the best provider based on real-time performance metrics instead of following the static priority order. Three mutually exclusive modes are available:

{
  "adaptive_routing": {
    "mode": "hedge",
    "qos_ranking": true,
    "latency_threshold_ms": 30000,
    "error_rate_threshold": 0.3,
    "probe_probability": 0.1,
    "probe_interval_secs": 60,
    "failure_threshold": 3,
    "weight_latency": 0.3,
    "weight_error_rate": 0.3,
    "weight_priority": 0.2,
    "weight_cost": 0.2
  }
}

Adaptive Modes

QoS Ranking

Setting qos_ranking: true enables quality-of-service ranking using a unified model catalog (model_catalog.json). The catalog provides baseline metrics (stability, latency, output quality) that blend with live traffic data via EMA:

• Cold start: Baseline catalog values are used (10 synthetic samples seeded).
• Warm state: Live metrics gradually replace baselines (weight ramps from 0 to 1 over 10 calls).
• Export: Live catalog is exported to model_catalog.json for observability.

Scoring Formula

Each provider is scored on 4 factors (lower score = better). All weights are configurable via adaptive_routing:

Provider Metadata

Hedge Mode Details

When Hedge is active:

1. The primary provider and the cheapest alternate are raced via tokio::select!.
2. The winner’s response is returned; the loser is cancelled.
3. Both completed requests record metrics (cancelled requests do not).
4. If the primary fails, the alternate is tried sequentially (it was cancelled by the race).

Auto-Escalation

When sustained latency degradation is detected (3 consecutive responses exceeding 3× baseline), the session actor auto-activates Hedge mode + Speculative queue. The ResponsivenessObserver learns a median baseline from the first 5 requests (robust to outliers), then adapts every 20 samples via 80/20 EMA blend with the current window median. When the provider recovers (one normal-latency response), both revert to normal.

Provider Wrappers

The routing stack is composed of layered wrappers: