Skip to content
UMKIMDocs

Operational introduction — lab site, HA, and ops

Experimental and unsupported: YAIDAP provisioning, BM install, and GPU passthrough change bare metal and hypervisors. Use reviewed lab inventory only.

This is Phase B of the operational introduction. Goal: provision site homelab through YAIDAP, bring up lab-single-0 and HA lab-ha-0, prove site-routed apps and Authelia, exercise node lifecycle, then KubeVirt, GPU clusters, and hub LiteLLM inference.

Examples use jumphost linux@jump.otthol.hu:30200, lab LAN 172.16.10.0/24, and zone homelab.otthol.hu. Complete Phase A first, or start from a hub that is already verified. See YAIDAP and UMKIM integration.

B1 — Lab site and ingress

Section titled “B1 — Lab site and ingress”

umkim site create -f takes a FleetSite manifest. The published umkim tree does not include examples/homelab/. The lab manifest sets jumphost to linux@jump.otthol.hu:30200 and ingress rows homelab-pf, homelab-gpu-pf (port_forward, wanAddressMode: dns), and edge-rp (reverse_proxy). Omitting spec.rockyVersion on a FleetSite stores the CRD default "9". Guest OS is inventory os: rocky-10. Cluster create goes through RockyMajor, which stores 10 when rockyVersion is omitted (FLEET_SITE_ROCKY_MAJOR defaults to 10).

Terminal window
umkim site create -f site-homelab.json
umkim site ingress set homelab homelab-pf --address 172.16.10.100
umkim site ingress add homelab --id homelab-ha-pf --method port_forward \
  --address 172.16.10.100 --zone homelab.otthol.hu \
  --wildcard '*.homelab.otthol.hu'
umkim site get homelab
umkim site ingress list homelab

WAN DNAT for *.homelab.otthol.hu is HTTP 172.16.10.100:30080 and HTTPS 172.16.10.100:30443 (Traefik NodePorts on the lab-ha-0 VIP). OpenWRT already has that DNAT.

Verify: site Ready. homelab-pf and homelab-ha-pf show address 172.16.10.100 and zone homelab.otthol.hu. homelab-gpu-pf stays DNS mode (*.gpu.homelab.otthol.hu) until a later cluster bind.

B2 — YAIDAP provisioner + inventory

Section titled “B2 — YAIDAP provisioner + inventory”

umkim site provision set takes --inventory (API field inventoryPath). umkim site provision yaidap-sync returns when the job is accepted. https:// and git@ remotes require --yaidap-git-secret: a Secret in fleet-console whose key is id_ed25519, deploy_key, or id_rsa. This demo uses the SSH remote git@github.com:rborso/yaidap.git at ref main.

Terminal window
# Secret fleet-site-homelab-yaidap-git must already exist (deploy key; do not commit it).
umkim site provision set homelab --provider yaidap \
  --jumphost linux@jump.otthol.hu:30200 \
  --ssh-secret fleet-site-homelab-provision-ssh \
  --yaidap-repo git@github.com:rborso/yaidap.git \
  --yaidap-ref main \
  --yaidap-root /home/linux/yaidap \
  --inventory /home/linux/yaidap/inventory/homelab \
  --yaidap-git-secret fleet-site-homelab-yaidap-git
umkim site provision yaidap-sync homelab
umkim site provision get homelab
umkim site inventory migrate homelab
# The published checkout has no examples/homelab/inventory/. Put files from a local inventory directory.
# Skip lab-gpu-0.yml — empty hosts: {} is rejected.
INV_DIR=/path/to/site-inventory
for p in \
  hypervisors.yml bare_metals.yml guests.yml \
  group_vars/all/all.yml \
  lab-ha-0.yml group_vars/lab-ha-0.yml \
  lab-ha-0-w1.yml group_vars/lab-ha-0-w1.yml \
  group_vars/lab-gpu-0.yml \
  lab-gpu-pt-0.yml group_vars/lab-gpu-pt-0.yml \
  host_vars/virt01.yml host_vars/virt02.yml host_vars/virt03.yml host_vars/virt04.yml \
  host_vars/lab-ha-0-cp1.yml host_vars/lab-ha-0-cp2.yml host_vars/lab-ha-0-cp3.yml \
  host_vars/lab-ha-0-w1.yml host_vars/lab-gpu-pt-0-cp1.yml
do
  umkim site inventory files put homelab --path "$p" -f "$INV_DIR/$p"
done
umkim site inventory sync homelab --wait

Inventory shape: hypervisors.yml (virt01 172.16.10.2, virt02 172.16.10.3, virt04 172.16.10.6), bare_metals.yml (virt03 172.16.10.4), guests.yml plus matching host_vars/*.yml (os: rocky-10). Root {cluster}.yml needs hosts or children; empty hosts: {} is rejected.

Verify: provisioner yaidap; umkim site provision get shows the synced git revision; inventory sync Succeeded; jumphost can reach virt01–virt04.

B3 — virt03: discover the existing host

Section titled “B3 — virt03: discover the existing host”

install_bm reinstalls a machine and requires --confirm. virt03 already runs Rocky Linux 10.2 at 172.16.10.4 and is listed in bare_metals.yml, so this demo discovers the host and imports inventory.

Terminal window
umkim site host get homelab virt03
ssh -J linux@jump.otthol.hu:30200 linux@172.16.10.4 'cat /etc/redhat-release; hostname'
umkim site host provision homelab virt03 --action discover --wait
umkim site inventory import preview homelab --wait
umkim site inventory import apply homelab --confirm \
  --paths host_vars/virt03.yml,bare_metals.yml
umkim site host get homelab virt03

Verify: discover Succeeded; host Ready; ansible_host 172.16.10.4, osFamily rocky-10 (GTX 1050 Ti present for later GPU stages).

B4 — Hypervisors prepare

Section titled “B4 — Hypervisors prepare”
Terminal window
umkim site inventory sync homelab --wait
umkim site host prepare homelab virt01 --wait   # 172.16.10.2 — action prepare_hypervisor
umkim site host prepare homelab virt02 --wait   # 172.16.10.3

Verify: virt01 / virt02 prepared as hypervisors; SSH from jumphost; libvirt healthy.

B5 — Single-node cluster lab-single-0

Section titled “B5 — Single-node cluster lab-single-0”
Terminal window
umkim cluster create -f cluster-lab-single-0.json
# bootstrapHost 172.16.10.4, site homelab, ingressId homelab-pf
umkim cluster bootstrap lab-single-0 --bootstrap-host 172.16.10.4 --ssh-user linux \
  --jumphost linux@jump.otthol.hu:30200
umkim cluster wait lab-single-0 --phase Validated --timeout 45m
umkim cluster provision lab-single-0
umkim cluster wait lab-single-0 --phase Ready --timeout 45m
umkim cluster get lab-single-0

Verify: phase Ready; spoke overlay 10.10.0.10.

B6 — HA cluster lab-ha-0 (VIP 172.16.10.100)

Section titled “B6 — HA cluster lab-ha-0 (VIP 172.16.10.100)”

Create guests then the HA cluster:

Terminal window
for h in lab-ha-0-cp1 lab-ha-0-cp2 lab-ha-0-cp3 lab-ha-0-w1; do
  umkim site host provision homelab "$h" --action create_vm --wait
done
# Addresses: cp1 172.16.10.101, cp2 .102, cp3 .103, w1 .140
# vm_location in the example host_vars: least-utilized for the CPs, virt02 for w1.
# Example placement: cp1 and cp3 on virt01 (172.16.10.2), cp2 and w1 on virt02 (172.16.10.3).


umkim cluster create -f cluster-lab-ha-0.json
# controlPlaneVip 172.16.10.100; the file's edge.ingressId is edge-rp (B7 rebinds homelab-ha-pf)
umkim cluster bootstrap lab-ha-0 --bootstrap-host 172.16.10.101 --ssh-user linux \
  --jumphost linux@jump.otthol.hu:30200
umkim cluster wait lab-ha-0 --phase Validated --timeout 45m
umkim cluster provision lab-ha-0
umkim cluster wait lab-ha-0 --phase Ready --timeout 90m

Verify: cluster Ready; three control planes + worker; VIP 172.16.10.100 Traefik NodePorts respond when site-routed ingress is configured; spoke overlay 10.10.0.12.

B7 / B7b — Site-routed apps and spoke Authelia

Section titled “B7 / B7b — Site-routed apps and spoke Authelia”

The lab HA create body binds edge.ingressId to edge-rp. Point the cluster at the port-forward row instead. Spoke ACME for port_forward uses Traefik TLS-ALPN (tlschallenge); HTTP-01 is off because the spoke redirects port 80 to HTTPS. Hub ACME stays HTTP-01. FleetIngressAuth on the cluster is separate from hub ingress auth.

Terminal window
umkim cluster patch lab-ha-0 --ingress-id homelab-ha-pf
umkim app create --cluster lab-ha-0 --chart whoami --version 0.1.0 \
  --name whoami-site --release whoami-site \
  --route-mode site --route-host whoami.homelab.otthol.hu
umkim cluster ingress-auth enable lab-ha-0 \
  --lldap-password-file /path/to/lldap-user-secret   # REDACTED path; password stays in the file
umkim cluster acme enable lab-ha-0
cat > /tmp/whoami-site-auth.json <<'EOF'
{
  "name": "whoami-site-auth",
  "clusterName": "lab-ha-0",
  "chart": "whoami",
  "version": "0.1.0",
  "releaseName": "whoami-site-auth",
  "route": {
    "mode": "site",
    "siteHost": "whoami-auth.homelab.otthol.hu",
    "requireAuth": true
  }
}
EOF
umkim app create -f /tmp/whoami-site-auth.json
curl -4 -sS https://whoami.homelab.otthol.hu/ | head
curl -4 -sSI https://whoami-auth.homelab.otthol.hu/ | head

Verify:

  • FleetApp whoami-site is on lab-ha-0, host whoami.homelab.otthol.hu, route mode site, no auth. tlsPhase Ready.
  • The certificate is a production Let’s Encrypt certificate, issuer CN YR1 (Let’s Encrypt). HTTP returns 308 to HTTPS. HTTPS verifies.
  • auth.homelab.otthol.hu has a production Let’s Encrypt certificate (issuer CN YR1).
  • Public HTTPS returns whoami for whoami.homelab.otthol.hu.
  • Unauthenticated whoami-auth.homelab.otthol.hu challenges Authelia at https://auth.homelab.otthol.hu/?rd=…; authenticated session returns 200.

Spoke firewalld

Section titled “Spoke firewalld”

configure_firewalld stays false, so YAIDAP prepare does not start firewalld or lock SSH.

The fleet_spoke_firewall role stays off unless fleet_spoke_firewall_enabled is true. That flag defaults to configure_firewalld (false). The role does not disable firewalld. When it is enabled it starts firewalld, trusts the WireGuard interface, and allows the overlay to TCP 6443, 10250, and hostNetwork LLM TCP 8080. Spoke Traefik NodePorts 30080, 30443, and 30081 are opened when spoke ingress is enabled.

When an operator turns YAIDAP firewalld on, firewalld_cluster_profile (default true) trusts 10.10.0.0/16 (UMKIM WireGuard), 192.168.0.0/16 (Calico; flannel is disabled), and 10.43.0.0/16, and trusts wg-umkim and wg0 when those interfaces exist.

Inventory zones also get TCP 80, 443, 30080, 30443, and 30081. Zone targets are DROP. TCP 6443 and 10250 are not opened on those DROP zones; trusted overlay and pod sources can reach them. TCP 179 (Calico BGP) is not in that zone list. host_prepare opens TCP 179, 6443, and 10250 only on its opt-in firewalld path (configure_firewalld true), because Calico peers are management-LAN addresses.

When firewalld is already running (firewall-cmd --state succeeds), K3s overlay tasks open spoke Traefik NodePorts if spoke ingress is enabled and trust discovered pod CIDRs on every cluster node, not only the bootstrap node. Those tasks do not start firewalld.

B8 — Cluster node operations

Section titled “B8 — Cluster node operations”

On lab-ha-0:

Step Action Verify
B8a Worker lab-ha-0-w1 (172.16.10.140) is already in cluster-lab-ha-0.json nodes[] and was created with create_vm in B6 umkim node list --cluster lab-ha-0 shows cp1–cp3 + w1 Ready
B8b Remove the worker. group_vars/lab-ha-0-w1.yml sets mode: none and autoApply: false so inventory reconcile does not addWorker it back Worker gone from the API; three CPs Ready
B8c Provision lab-ha-0-cp5 @ 172.16.10.144, then umkim node replace Ready nodes cp1 / cp2 / cp5
Terminal window
# B8b — put the autoApply:false group_vars, sync, then remove.
# A remove while another node operation is in progress returns HTTP 422.
umkim site inventory files put homelab --path group_vars/lab-ha-0-w1.yml \
  -f group_vars/lab-ha-0-w1.yml
umkim site inventory sync homelab --wait
umkim node remove lab-ha-0-w1 --cluster lab-ha-0 --force


# B8c — node name is --old-node. The published examples stop at cp1–cp3 and w1.
# Add replacement guest lab-ha-0-cp5 (172.16.10.144) to guests.yml
# and host_vars/lab-ha-0-cp5.yml in the same shape as lab-ha-0-cp1.yml, then:
umkim site inventory sync homelab --wait
umkim site host provision homelab lab-ha-0-cp5 --action create_vm --wait
umkim node replace --cluster lab-ha-0 \
  --old-node lab-ha-0-cp4 \
  --address 172.16.10.144 \
  --hostname lab-ha-0-cp5

While a replace is Pending or Running, the controller stores oldNodeAddress and injects that control-plane host into the leave inventory on every rebuild. This example leaves lab-ha-0-cp4 and joins lab-ha-0-cp5. Ready nodes afterward: lab-ha-0-cp1 172.16.10.101, lab-ha-0-cp2 172.16.10.102, lab-ha-0-cp5 172.16.10.144. Check umkim node-op history --cluster lab-ha-0 if the phase stays non-Ready.

B9 — KubeVirt and Multus

Section titled “B9 — KubeVirt and Multus”

The command is umkim cluster platform enable-kubevirt. --lan-bridge configures host bridge n0p0_mgmt_nw (override with --bridge) and Multus NAD default/mgmt-l2. --wait is a duration; 0 returns after the API accepts the job. Helm values use --values-file. -f on app create is a JSON body.

Terminal window
umkim cluster platform enable-kubevirt lab-ha-0 --lan-bridge --wait 45m
umkim cluster platform status lab-ha-0
umkim app create --cluster lab-ha-0 --chart rocky-10-vm --version 0.1.3 \
  --name rocky-kv-pod --release rocky-kv-pod --route-mode none
umkim app create --cluster lab-ha-0 --chart rocky-10-vm --version 0.1.3 \
  --name rocky-kv-lan --release rocky-kv-lan --route-mode none \
  --values-file rocky-lan-ha-values.yaml

Verify: KubeVirt platform Ready; capabilities show kubevirt / multus; rocky-kv-pod and rocky-kv-lan Ready/Running. See Fleet KubeVirt.

B10 — virt04 GPU passthrough guest

Section titled “B10 — virt04 GPU passthrough guest”

virt04 is 172.16.10.6 in hypervisors.yml. GPU prepare requires --confirm. The combined flag is umkim site host prepare … --gpu-passthrough, which posts action prepare_hypervisor_gpu (prepare plus VFIO). This demo uses the granular action prepare_gpu_passthrough. A prepare against a name that is not in the hub inventory returns HTTP 404 (guest not found / not found).

Terminal window
umkim site inventory sync homelab --wait
umkim site host prepare homelab virt04 --wait
umkim site host provision homelab virt04 --action prepare_gpu_passthrough --confirm --wait
# equivalent combined action, once virt04 is in inventory:
# umkim site host prepare homelab virt04 --gpu-passthrough --confirm --wait
umkim site host provision homelab lab-gpu-pt-0-cp1 --action create_vm --wait
# guest lab-gpu-pt-0-cp1 @ 172.16.10.141, vm_location virt04 (GTX 1650)
ssh -J linux@jump.otthol.hu:30200 linux@172.16.10.141 'hostname; uptime'

Verify: guest SSH reachable; GPU hostdev attached (driver install continues in B13 with umkim cluster gpu host-prep).

B11 — GPU clusters Ready

Section titled “B11 — GPU clusters Ready”

lab-single-0 on virt03 was already Ready from B5. Provision the passthrough guest as its own cluster:

Terminal window
umkim cluster create -f cluster-lab-gpu-pt-0.json
# bootstrapHost 172.16.10.141, ingressId homelab-gpu-pf
umkim cluster bootstrap lab-gpu-pt-0 --bootstrap-host 172.16.10.141 --ssh-user linux \
  --jumphost linux@jump.otthol.hu:30200
umkim cluster provision lab-gpu-pt-0
umkim cluster wait lab-gpu-pt-0 --phase Ready --timeout 120m
umkim cluster list -o wide

Verify: umkim cluster list -o wide shows lab-single-0 and lab-gpu-pt-0 Ready (spoke overlays 10.10.0.10 and 10.10.0.14). homelab-gpu-pf is DNS mode for *.gpu.homelab.otthol.hu (no fixed WAN address in the example). Bind with umkim cluster patch lab-gpu-pt-0 --ingress-id homelab-gpu-pf when the create body did not already set edge.ingressId. In this lab, ACME is enabled on lab-ha-0 and lab-single-0, and not on lab-gpu-pt-0.

B12 — Hub LiteLLM

Section titled “B12 — Hub LiteLLM”

umkim llm set writes FleetLLM. umkim llm reconcile --wait polls until phase Ready; --timeout is seconds (default 600), not a Go duration.

Terminal window
umkim llm set --hub-enabled --public-host llm.api.demo.otthol.hu
umkim llm reconcile --wait --timeout 600
umkim llm get
# The token is printed once on stderr. Save it yourself, mode 600, outside git
# This demo saves it at ~/.umkim-llm-intro.key:
umkim auth llm-api-keys create --name intro-lab --display-name "Intro lab"
curl -4 -sS -H "Authorization: Bearer $(cat ~/.umkim-llm-intro.key)" \
  https://llm.api.demo.otthol.hu/v1/models | head

Verify: phase Ready; endpoint https://llm.api.demo.otthol.hu/v1; litellm + redis pods Running in fleet-llm. See fleet-llm.

B13 — llama.cpp on GPU clusters + inference

Section titled “B13 — llama.cpp on GPU clusters + inference”

--wait on gpu host-prep and gpu enable takes a duration string (45m, 30m). Host prep on a cluster that is already GPU-Ready needs --force. --llm-backend is true or false (it sets annotation fleet.umkim.com/llm-backend). --llama-profile and --values-file are mutually exclusive on create. umkim app update takes --values-file and replaces spec.valuesYaml as a whole.

Terminal window
umkim cluster gpu host-prep lab-single-0 --wait 45m
umkim cluster gpu enable lab-single-0 --wait 30m
umkim cluster gpu discover lab-single-0
umkim cluster gpu status lab-single-0
# GTX 1050 Ti, driver 535, clusterGpuCapability=true, gpuAvailable=1


umkim cluster gpu host-prep lab-gpu-pt-0 --wait 45m
umkim cluster gpu enable lab-gpu-pt-0 --wait 30m
umkim cluster gpu discover lab-gpu-pt-0
umkim cluster gpu status lab-gpu-pt-0
# GTX 1650 / TU117 (pci 10de:1f82), same capability flags


umkim app create --cluster lab-single-0 --chart llama-cpp-helm --version 0.1.4 \
  --llama-profile qwen35-2b-ud-4g --namespace llm --release llama-cpp \
  --name llama-virt03 --llm-backend true --route-mode none


umkim app create --cluster lab-gpu-pt-0 --chart llama-cpp-helm --version 0.1.4 \
  --llama-profile qwen35-2b-ud-4g --namespace llm --release llama-cpp \
  --name llama-gpu-pt --llm-backend true --route-mode none
# Guest has ~3.6Gi RAM. Profile qwen35-2b-ud-4g requests memory 4Gi / limits 6Gi (CPU 2/4),
# which does not fit this guest. --values-file replaces spec.valuesYaml entirely (not a merge).
umkim llm profile values qwen35-2b-ud-4g > /tmp/llama-gpu-pt-mem.yaml
# Edit that file: resources.requests memory 1500Mi / cpu "1", limits memory 2800Mi / cpu "3".
umkim app update llama-gpu-pt --values-file /tmp/llama-gpu-pt-mem.yaml


umkim llm backends sync-from-apps
umkim llm reconcile --wait --timeout 600
# backends: 10.10.0.10:8080 + 10.10.0.14:8080 model Qwen3.5-2B-UD
curl -sS http://10.10.0.10:8080/health   # {"status":"ok"}
curl -sS http://10.10.0.14:8080/health   # {"status":"ok"}


curl -4 -sS https://llm.api.demo.otthol.hu/v1/chat/completions \
  -H "Authorization: Bearer $(cat ~/.umkim-llm-intro.key)" \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3.5-2B-UD","messages":[{"role":"user","content":"Say Hi!"}]}'

Verify: both backends /health return ok; hub LiteLLM chat returns HTTP 200 with assistant content “Hi!”.

What you should see

Section titled “What you should see”
  • Site homelab Ready
  • lab-single-0 Ready (virt03, GTX 1050 Ti)
  • lab-ha-0 Ready with control planes cp1, cp2, and cp5
  • Site-routed whoami-site on lab-ha-0 (whoami.homelab.otthol.hu, no auth, tlsPhase Ready) and whoami-site-auth Ready
  • KubeVirt rocky-kv-pod and rocky-kv-lan Ready
  • lab-gpu-pt-0 Ready (guest 172.16.10.141)
  • Hub LiteLLM Ready at llm.api.demo.otthol.hu with synced llama backends and a successful chat

Next / related

Section titled “Next / related”