Skip to content
UMKIMDocs

fleet-llm

Experimental: LLM, GPU placement, proxy, and billing-related behavior are not production guarantees.

Fleet LLM is an optional hub LiteLLM proxy. GPU inference is the llama-cpp-helm chart installed with FleetApp onto a selected Ready cluster; see FleetApp.

Components

Section titled “Components”
Component Role
FleetLLM Optional configuration and reconciliation of the hub proxy
LiteLLM Hub-side OpenAI-compatible proxy
llama-cpp-helm GPU inference workload discovered as a LiteLLM backend
Redis (fleet-llm) Router cache for session affinity

The workload is llama.cpp using a CUDA llama-server image and a GGUF model source. Review model alias, context size, GPU-layer setting, resource requests, persistence, and runtime class for each deployment.

Multi-spoke routing

Section titled “Multi-spoke routing”

Use the same chart model value (OpenAI alias) on multiple GPU clusters when you want one logical model id on the hub. Ready llama-cpp-helm FleetApps sync as multiple LiteLLM backends with the same model_name; LiteLLM load-balances (router_settings.routing_strategy: simple-shuffle).

deployment_affinity is not enabled on the hub proxy: LiteLLM pins that mode by API key hash, which would send all traffic for one Fleet LLM key to a single spoke. For sticky multi-turn chats, use session_affinity (Redis-backed): send header x-litellm-session-id or metadata.session_id on each chat/completions call (TTL deployment_affinity_ttl_seconds in LiteLLM config, default 3600). Hub reconcile clears legacy deployment_affinity:* Redis keys when deployment affinity is off.

Auto backends (backendsSource: auto) rebuild from Ready registered llama-cpp FleetApps. Use umkim llm backends sync-from-apps or console Sync backends from FleetApps. Manual JSON sets backendsSource: manual.

For Qwen3.x GGUFs, set chart values jinja: true and chatTemplateKwargs: '{"enable_thinking":false}' so replies land in message.content. Shared values live under deploy/llama-cpp-helm/ in the UMKIM repo (runtime overlay, profiles, render-llama-cpp-values.sh).

FleetApp route (hub/site ingress) is separate: hub LiteLLM reaches spokes via WireGuard upstream_host, not route hostnames.

Access control

Section titled “Access control”

The public LLM API requires authenticated access through fleet-console-auth (platform-admin session or scoped Fleet LLM API key). Do not expose LiteLLM as an anonymous endpoint. Inference workloads may use host networking — validate firewall and ingress before WAN exposure.

Enable and verify

Section titled “Enable and verify”

Not installed by install-hub.sh. After GPU-capable clusters exist:

  1. GPU host prep and fleet.capability/gpu=true — GPU host prep.
  2. umkim llm set --hub-enabled then umkim llm reconcile --wait (or console Infrastructure → LLM platform).
  3. Install llama-cpp-helm with profile or values; enable hub registration (--llm-backend / console checkbox unless annotation fleet.umkim.com/llm-backend: "false").
  4. umkim llm reconcile --wait after backend changes.
  5. Scoped API key: umkim auth llm-api-keys create or console Access control → LLM API keys.

Ansible:

Terminal window
ansible-playbook fleet-llm.yml -l hub --tags fleet_llm_hub -e fleet_llm_enabled=true

Verification scripts in the UMKIM repo: verify-fleet-llm.sh, verify-litellm-routing.sh, verify-fleet-llama-profile.sh, verify-llm-inference-e2e.sh. See Verification.

Section titled “Related”