Verification

Experimental: verification increases confidence; it does not certify production readiness.

scripts/verify-fleet.sh runs layered checks from the repository root.

Start with the read-only hub audit:

./scripts/verify-fleet.sh --layer 1 --suite hub

Layers

Layer	Purpose	Safety
`1`	Read-only audit and health checks	Safe after deployment
`2`	Synthetic create, verify, and cleanup	Changes test resources
`3`	Rebuild workflow followed by checks	Destructive; approved maintenance window only

Registered suite names include hub, core, edge, backup, tenancy (console auth / RBAC), llm, apps, regression, and all. Legacy names (nested, catalog) may appear in --help; use apps for chart-library checks.

App and chart smoke tests

Script	Purpose
`verify-fleet-app.sh`	whoami install + pod probe (`FLEET_APP_CLUSTER`)
`verify-fleet-chart.sh`	Hub library lists seeded charts
`verify-fleet-llama-app-lifecycle.sh`	llama-cpp install, PATCH, LLM backend sync (`FLEET_LLM_CLUSTER` on GPU Ready cluster)
`verify-fleet-llama-profile.sh`	Profile CRUD, quick/extended verify, hub-proxied chat (check id `llm.profile-verify`)
`verify-fleet-llm.sh`	Hub LiteLLM enabled + smoke
`verify-litellm-routing.sh`	Multi-spoke load balance + session sticky (`x-litellm-model-api-base`)
`verify-llm-inference-e2e.sh`	End-to-end LLM API through console auth

See FleetApp, fleet-llm, and Fleet Llama profiles.

Cluster reinstall presets

Cluster reinstall is an L2 operation. Built-in preset names: lab-sn01, lab-3n01, lab-ha01 (homelab profile in the check registry). Example:

./scripts/verify-fleet.sh --layer 2 --cluster-preset lab-sn01 --cluster-preset-only

A reinstall can delete cluster state; confirm the target and recovery plan first.

For HA presets, hub finalize Jobs allow up to 90 minutes. Default flow polling is 90 × 20s (30 minutes). Override fleet_flow_poll_retries (for example 270 for 90 minutes) for HA flows.

Reports land under reports/verify-<layer>-<timestamp>/. Protect reports if they contain operational identifiers.

Use layers 2 and 3 only with a reviewed target and recovery plan. See scripts/fleet-verify/README.md for the check registry.