SimpleVirt Architecture

This is the authoritative technical architecture of SimpleVirt: how the two appliances are built, how they cooperate, who owns which state, and how the platform behaves under failure. It is a reference document — it names internal components and notes that SimpleVirt HCI uses Ceph internally. The task-oriented guides (user guide, operations, troubleshooting) omit those internals.

Related reading: HA lease fence — the single-writer disk-fencing contract HA depends on; Known limitations — what this release does not do; Backup responsibility — what SimpleVirt protects and what you must protect yourself.

1. Product shape

SimpleVirt is a two-tier virtualization platform delivered as bootable appliance ISOs. The Go module is simplevirt.

• SimpleVirt-Hypervisor — the per-host appliance. Binary simplevirtd (internal/server, internal/hypervisor). It turns a physical machine into a locked-down KVM/QEMU host that runs Virtual Machines through libvirt and exposes a host agent API over HTTPS. The on-host operator interface is the simplevirt-console TUI; there is no general shell to secure. One appliance per physical Host.
• SimpleVirt-Command — the controller appliance. Binary simplevirt-command (internal/commandserver, internal/registry). It manages a fleet of Hosts grouped into Clusters, holds desired state, and drives orchestration (HA, migration, networking, HCI, lifecycle). Its state store is a bundled PostgreSQL (internal/db; most stores have both a file and a *_pg.go implementation). Appliance TUI is simplevirt-command-console. Command ships as its own ISO and runs as a guest VM on a SimpleVirt Host.

graph TB
  subgraph Command["SimpleVirt-Command  (simplevirt-command)"]
    REG["Registry / Inventory<br/>(hosts, clusters, desired state)"]
    ORCH["Orchestration engines<br/>HA · Workload Distributor · Reconcilers"]
    PG[("Bundled PostgreSQL")]
    UI["Web UI + REST API"]
    REG --- PG
    ORCH --- REG
    UI --- REG
  end

  subgraph H1["SimpleVirt Host — host01"]
    A1["simplevirtd agent API"]
    L1["libvirt / KVM-QEMU"]
    V1["VMs"]
    A1 --- L1 --- V1
  end
  subgraph H2["SimpleVirt Host — host02"]
    A2["simplevirtd agent API"]
    L2["libvirt / KVM-QEMU"]
    V2["VMs"]
    A2 --- L2 --- V2
  end
  subgraph H3["SimpleVirt Host — host03"]
    A3["simplevirtd agent API"]
    L3["libvirt / KVM-QEMU"]
    V3["VMs"]
    A3 --- L3 --- V3
  end

  Command -- "HTTPS agent API<br/>(TOFU-pinned cert + per-host bearer token)" --> H1
  Command --> H2
  Command --> H3

Command runs as a VM on the fleet it manages; it is not in any VM's data path, so a Command outage never stops running VMs — only orchestration and the central UI pause.

2. SimpleVirt-Hypervisor (the Host)

simplevirtd owns everything local to one physical machine. Its responsibilities:

• VM execution — creates and runs domains on KVM/QEMU via libvirt (internal/hypervisor/libvirt.go, domxml.go). It builds domain XML for CPU topology, CPU pinning, memory, disks/bus, NICs, PCI passthrough, static MAC, boot order, and cloud-init.
• Host networking — realizes the local networking model (hostnet.go, netmodel.go, nics.go): Logical Switches bound to physical uplink NICs, and Networks (port groups) as additive 802.1Q sub-interfaces + bridges on those uplinks. The management interface is a derived, read-only service adapter configured through internal/netcfg.
• Storage — mounts and presents Datastores (datastore_*.go): NFS shared datastores and the SimpleVirt HCI pooled datastore. Handles disks, snapshots, CD-ROM/ISO, offline and live storage migration (storagemigrate.go, storagecopy.go).
• Local agent API — serves /api/v1/agent/* for Command and a small unauthenticated /healthz liveness probe. Self-signed TLS; no external CA.
• Health & self-report — reports host info, capacity, datastore status, alerts, and per-NIC/VLAN state (server/handlers_health.go, svcprobe.go). Liveness reads are deliberately isolated from sick storage so one wedged mount cannot make a live host look dead.
• HA participation — runs the disk-lease renewer, watchdog pet, and fence logic (lease*.go, wdmd.go, simplevirt-wdmd.service); libvirt lockd is the final disk-lock arbiter.
• SimpleVirt HCI daemon — simplevirt-cephd participates in the pooled storage backend (see §9).
• Cluster-services safety plane — simplevirt-clusterd (internal/clusterd) runs a Command-independent observe-only safety plane (see §8). It is enabled and shipped but disarmed by default (shadow mode), with Command remaining the live HA decider.

A Host is fully functional standalone. Command adds fleet coordination on top; it never becomes a required runtime dependency for a Host to keep serving its VMs.

3. SimpleVirt-Command (the Controller)

simplevirt-command is the fleet brain. Its responsibilities:

• Inventory (internal/registry) — the persisted set of managed Hosts (address, per-host bearer token, TOFU-pinned cert fingerprint, cluster membership) and the Clusters that group them. This store holds live host tokens and is the highest-value secret in the system.
• Live view / poller (registry/poller.go) — polls every Host and maintains an in-memory view. A failed poll never erases inventory: last-known info/VMs are carried forward and marked stale. Liveness is three-valued: online (agent API answered), degraded (/healthz answered but agent API did not — host up, hypervisor layer sick), unreachable (even /healthz silent). Failures are classed terminal (auth, tls-pin) vs retryable.
• Desired state & reconciliation — Clusters record intent (declared Datastores, HA intent, distributed switches). Reconcilers apply that intent onto members (§5).
• Orchestration engines — HA restart (ha_engine.go), Workload Distributor / load balancing (drs_engine.go, recommend + automatic modes), admission control / HA reserve (admission.go), host maintenance & evacuation (maintenance.go, handlers_evacuate.go), migration.
• Networking control — declares and realizes Logical Distributed Switches across a Cluster (handlers_clusterswitch.go).
• HCI control — orchestrates the pooled cluster storage backend and its multi-tab management console (handlers_ceph.go, hci_*.go).
• Lifecycle — signed installer-ISO update flow (handlers_lifecycle.go, internal/update).
• RBAC & directory — default-deny per-route capabilities (internal/authz), users and roles (internal/auth), LDAP/AD integration (auth/ldap.go).
• UI/API — an embedded, dependency-free web UI and REST API, with a durable Tasks log and an SSE Event stream.
• State — persisted to bundled PostgreSQL (simplevirt-postgres.service, simplevirt-postgres-init.service).

4. Control-plane communication

Command talks to Hosts and only to Hosts; Hosts never call Command.

• Transport — HTTPS to each Host's agent API under /api/v1/agent/*. Hosts serve self-signed TLS, so there is no CA to verify against. Each connection is pinned to the certificate fingerprint captured when the Host was added (trust-on-first-use, registry/client.go, pinning.go). A changed fingerprint is a terminal tls-pin failure, surfaced honestly rather than retried.
• Authentication — every agent call carries a per-host bearer token minted at pairing time and stored in the registry.
• Pairing — an operator adds a Host by address + admin credentials; Command pairs via registry.Pair (pair.go, handlers_pairing.go), mints the token, and pins the cert. No tokens or CLI on the Host side.
• Controller→host proxy — host-admin operations that the UI needs (host networking, disks, certificates, etc.) are proxied through Command to the Host's admin endpoints via a fixed-path agent proxy, so the browser never talks to a Host directly.
• Distinct client tiers — the agent client keeps separate HTTP clients per pinned fingerprint and per timeout profile: short-timeout poll clients, normal clients, and generous long clients for slow operations (create, migrate).

5. State ownership

The single most important architectural rule: desired state lives on Command; live facts live on the Host; the two are reconciled, never conflated.

Two deliberate consequences:

• Live facts are never persisted as desired state. Mount status and HA readiness are derived from the poller each time, so the store can never drift "stale-but-confident" out of agreement with the fleet.
• Reconciliation is edge-triggered and additive (cluster_reconciler.go). Declaring or editing a cluster Datastore fans it out to current members; a Host joining a Cluster receives declared Datastores once. There is no continuous convergence loop and no "converging…" nag. Applies are strictly additive and idempotent — reconciliation never tears anything down (removal is always an explicit operator action), so it can never disconnect a VM or unmount storage out from under one. Logical Distributed Switches are attached to a Host only by explicit operator action, never silently re-asserted on rejoin.

6. Networking architecture

SimpleVirt networking is identity-first and ownership-scoped. Every switch and Network has a stable random ID independent of its (mutable, possibly colliding) name (netmodel.go: newNetID), and every object records an owner so a local Host action can never mutate a Command-owned object and vice versa.

• Logical Switch — host-local. A named binding of one or more physical uplink NICs. It is realized on exactly one Host.
• Networks / port groups — VLAN segments on a switch that VMs attach to. Each provisions an additive 802.1Q sub-interface + bridge on the uplink (the uplink's own traffic is untouched) plus a libvirt bridge network of the same name.
• Uplinks — logical uplink slots on a switch, mapped to physical NICs; supports multiple uplinks (kernel bond, active-backup) with live add/remove.
• Logical Distributed Switch — declared once per Cluster on Command and realized identically on every member Host, so a VM's Network exists on the same VLAN everywhere and migrates without a per-host port-group mismatch (handlers_clusterswitch.go). Declaration is desired state; realization is per-host.
• Realization — Command fans the declaration out to members as bounded background work, one Task per Host. A live GET reports realized-vs-declared per member rather than pretending to know the end state; a wedged member degrades to "could not be read" instead of stalling the whole view.
• Drift / reconciliation — the Networking view continuously compares declared intent against each member's realized state. Reconciliation is explicit and additive; a member that was unreachable when a switch was declared simply lacks it until re-applied. Migrate prechecks verify VLAN presence on the destination before a VM moves.

graph TB
  subgraph CMD["SimpleVirt-Command"]
    LDS["Logical Distributed Switch<br/>(declared once per Cluster)<br/>stable IDs · uplink slots · Networks/VLANs"]
  end
  subgraph M1["Host host01 (realization)"]
    S1["Logical Switch"] --> U1["Uplinks → vmnicX"]
    S1 --> P1["Networks / port groups (VLAN)"]
  end
  subgraph M2["Host host02 (realization)"]
    S2["Logical Switch"] --> U2["Uplinks → vmnicX"]
    S2 --> P2["Networks / port groups (VLAN)"]
  end
  LDS -- "fan-out, one Task per host" --> M1
  LDS -- "fan-out, one Task per host" --> M2
  M1 -. "realized-vs-declared" .-> LDS
  M2 -. "realized-vs-declared" .-> LDS

7. VM lifecycle

VM operations are Host-executed and Command-orchestrated.

• Create — a wizard collects CPU topology, CPU pinning, memory, disk bus, PCI passthrough, static MAC, boot order, and cloud-init; the Host builds the domain XML (domxml.go) and defines the domain.
• Power — start, stop (graceful/guest-agent), reset, suspend, force-stop.
• Edit — CPU/memory, NIC attach/detach, CD-ROM/ISO, disk add/resize.
• Snapshots, templates/clone, folders — snapshot lifecycle; clone/deploy from a template (full disk copy, no installer boot); folder organization for inventory.
• Console — VNC via noVNC, proxied through Command (console.go, console-page.js).
• Migration
  • Live migration and storage migration use libvirt's peer-to-peer transport across a dedicated migration network: a control channel on TCP 16509 and the qemu data channel on TCP 49152–49215. The data channel is encrypted and mutually authenticated by default (qemu+tls / VIR_MIGRATE_TLS): guest RAM and the domain XML cross the wire under TLS, not in the clear. Command stands up a per-fleet migration CA (cfg.MigrationCADir) and auto-issues short-lived per-host leaf certificates at controller startup, host-add, and renewal; every migration requires mTLS, and the handler refuses rather than downgrades if a peer lacks a valid leaf. Encryption is on unless migration.encryption is explicitly set false, which selects a plaintext qemu+tcp opt-out.
  • Migration access control is still network-scoped, and this is the residual boundary that stays true regardless of the encryption setting: the libvirt control listener on TCP 16509 runs with auth_tcp = "none" (unauthenticated), and simplevirt-migration.nft opens it to migration peers on the migration adapter. Anything that can reach 16509 has root-equivalent control of that Host, so a dedicated, isolated migration VLAN with the per-peer nft roster remains a deployment requirement — see Installation. Each Host firewalls 16509 and the data ports closed by default (simplevirt-migration.nft) and opens them only on a migration adapter and to the fleet peer addresses Command pushes (simplevirt-migration-tls.service, simplevirt-migration-peers.service). The listener is ordered Requires=/After=nftables.service, so a firewall that fails to load takes 16509 down with it rather than exposing it (fail-closed).
  • Cold / offline migration copies disks off local storage for stopped VMs (handlers_coldmigrate.go, OfflineStorageMigrate).
  • All migrations run prechecks (VLAN presence, storage reachability) before moving.

8. Clusters & HA

A Cluster groups Hosts so CPU/memory/Datastores aggregate and VMs can migrate and fail over between members. Everything on a Cluster object is desired state.

HA restart engine (ha_engine.go) restarts a dead Host's protected VMs on a survivor. Its design turns on a few load-bearing refusals:

• Enable-time gate vs per-pass gate. The full readiness check (all hosts online, capacity, a live cross-host lock round-trip) runs only at enable time, while every Host is up. Re-imposing it every pass would make failover never fire, because those checks go false the instant a member is unreachable — exactly when failover must run. The per-pass gate is structural only: is there still somewhere to fail over to. Safety is enforced per target at the moment of each restart.
• Acts only on true death. A VM is restarted elsewhere only when its Host is unreachable on both planes (agent API and /healthz) for a sustained hold-down. A degraded Host (up, libvirt sick, VMs still running and holding disk locks) is never failed over — that would be the exact double-write the lock exists to prevent.
• lockd is the final arbiter. An adopt on the survivor either succeeds — itself proof the dead Host released the fcntl disk lock — or is refused "disk still held," which the engine surfaces and re-polls, never forces. There is no lock-override path anywhere, by design.
• Placement by identity. Targets are chosen by Datastore identity + capacity, never by name.
• Ghost cleanup is safe. A stale definition is undefined only after the survivor copy is confirmed running.

Disk fencing has two paths (lease*.go, ./HA_LEASE_FENCE.md): NFSv4.1 auto-expiring fcntl leases where available, and a heartbeat-lease record on the datastore (owner graceful self-fence + survivor observe-then-steal, watchdog-backed) where the mount cannot provide them. A single RBD watcher keeps HCI data safe.

Supporting engines: admission control reserves HA failover capacity and rejects power-on/create that would break the reserve; the Workload Distributor rebalances VMs (recommend or automatic); host maintenance/evacuation drains a Host with zero downtime before planned work.

Command-independent safety plane (simplevirt-clusterd). simplevirt-clusterd (internal/clusterd) is a real, shipped, enabled cluster-services daemon that gives the fleet a Command-independent view of member liveness. It is observe-only by construction: an arch_test.go import ban forbids internal/clusterd from linking internal/hypervisor, internal/ceph, internal/registry, or internal/server, so it structurally cannot name a fence, adopt, power, or storage operation — it can observe and warn, never act. At HEAD its arming was made durable and operator-controllable (armed.json, POST .../cluster-services/arm), but the default is disarmed/shadow: Command remains the live HA decider, and the plane runs behind the cluster_services experimental flag plus a quorum requirement. It is a redundant observer today, not a second actor.

9. Storage & SimpleVirt HCI

Two Datastore kinds serve VMs:

• NFS shared datastores — external shared storage mounted read-write on cluster members; identity is server:/export (a stable ID), never the display name, so a rename/re-add never strands VMs.
• SimpleVirt HCI — a supported, always-on pooled storage tier (not a preview or experimental feature): resilient cluster storage built from Hosts' local disks and presented as a single Datastore reachable from every member. SimpleVirt HCI uses Ceph internally as its storage engine (replicated pools, MON quorum, OSDs). The host-side helper is simplevirt-cephd (simplevirt-cephd.service, simplevirt-ceph-boot.service) — a root-run POSIX shell privilege-helper, not a compiled Go daemon: it watches path-triggered request files (/run/simplevirt/ceph/*.req) and executes the privileged Ceph operations they name on behalf of the unprivileged simplevirtd. Customer-facing surfaces say only "SimpleVirt HCI" and never expose raw Ceph/CRUSH commands. A witness MON can run on Command. Management is a multi-tab console (hci_*.go, hciview.js), including shipped day-2 operations: safe data-disk drain/replace (POST /api/v1/clusters/{id}/hci/osd/{osd}/remove) and safe host decommission (POST /api/v1/clusters/{id}/hci/host/{host}/decommission). HCI-backed VMs cannot take internal snapshots — a raw RBD image cannot hold a qcow2 internal snapshot, and SimpleVirt refuses rather than silently dropping the request.

graph TB
  subgraph HCI["SimpleVirt HCI — one pooled Datastore"]
    direction LR
    O1["Host host01<br/>simplevirt-cephd · local disks (OSDs)"]
    O2["Host host02<br/>simplevirt-cephd · local disks (OSDs)"]
    O3["Host host03<br/>simplevirt-cephd · local disks (OSDs)"]
    O1 --- O2 --- O3
  end
  W["Witness MON<br/>(may run on SimpleVirt-Command)"]
  HCI --- W
  VMS["VMs on every member<br/>see the same Datastore"] --- HCI

Replication and quorum give the pooled Datastore resilience to a Host or disk loss without VM data loss; VMs on any member address the same Datastore, which is what makes live migration and HA restart possible across the Cluster.

10. Stretched Cluster

A Stretched Cluster spans two independent sites plus a witness — a safety-critical topology. It is software-complete with live activation gated: the orchestration engine, health model, state machine, failure-simulation harness, and guided wizard are implemented and unit-tested against mock/recorded backends; the one remaining piece is command execution against a real two-site cluster plus live network/witness probes. It is not proven on live two-DC hardware, and enable is gated on readiness so it never activates in a lab.

Topology — exactly two DATA sites (PRIMARY, SECONDARY) and one WITNESS site (quorum/tiebreaker only — never customer VMs or OSDs). A Site is a failure domain identified by a stable ID, never a name convention.

Pluggable storage backends (registry/stretch.go, stretch_backends.go) — the cluster asks the backend "can the surviving site safely reach this VM's storage?", never "is Ceph healthy?":

• SIMPLEVIRT_HCI — SimpleVirt owns the stretched pool (CRUSH datacenter buckets, MON stretch mode). Requires a Witness, which hosts the tiebreaker MON. Site-protection health is rolled up live (stretch_hci_health.go).
• NFS_SHARED — an external shared Datastore reachable from both sites; SimpleVirt owns compute HA + fencing only. Readiness is real and evaluable: the same Datastore identity (server:/export, not the name) must be mounted read-write in both sites. Storage-site resilience is reported separately and honestly — reachable-from-both-sites does not claim the array survives a storage-site loss, and SimpleVirt never fakes replication it cannot prove.

State & reconciliation — the config records intent only; live facts (per-site reachability, backend health, real state-machine position) are derived, not persisted, because a persisted "healthy" that outlives the truth is the worst failure mode here. Only the state field is persisted, and it is reconciled against real storage/network/host state on restart. Positions include STRETCH_HEALTHY, DEGRADED_SITE_A/B, NETSPLIT, planned MAINTENANCE_A/B, STRETCH_RECOVERING, and STRETCH_ERROR. enable/disable activate or exit stretch mode and are gated on readiness.

Fencing & failure handling — the standard HA engine and its fencing rules are shared unchanged; stretch adds site-aware placement. On a site loss, the surviving site restarts protected VMs subject to the same "true death only" and lockd-arbiter rules; the Witness provides the quorum tiebreaker that keeps a network partition from becoming split-brain.

graph TB
  subgraph SA["Site A (PRIMARY)"]
    HA1["Hosts + OSDs / shared DS mount"]
  end
  subgraph SB["Site B (SECONDARY)"]
    HB1["Hosts + OSDs / shared DS mount"]
  end
  subgraph SW["Witness"]
    WMON["Tiebreaker MON (HCI) /<br/>quorum health role"]
  end
  HA1 <-- "inter-site replication / shared access" --> HB1
  SA --- SW
  SB --- SW

11. Tasks, Events & observability

• Tasks (internal/tasks) — a durable, bounded, server-side log of long-running operations, the truth behind the operator's Recent Tasks dock. It captures work the controller does on its own (a health transition, an automatic failover) and work other operators start — neither of which a browser-local list could show. Tasks mutate running → success/failed and carry a stable ID, error text, and a finished timestamp.
• Events (handlers_events.go) — an SSE stream that pushes live task/health changes to the UI. The SSE fan-out is in-process (a live convenience), separate from the durable store.
• Health & Alarms — per-host health and alerts roll up into a fleet health center, including network-health rollup across the distributed switch.
• Audit (internal/audit) — an append-only record of every mutation, distinct from Tasks (audit is a fact about the past; Tasks back the live dock).

Both Tasks and most other stores have file and PostgreSQL backends; the pgStore runs the identical apply/validate logic inside SELECT … FOR UPDATE, so file and PG paths validate and mutate byte-identically.

12. Security boundaries

• No shell on the appliances. Operators use the web UI/API or the on-host TUI; there is no general-purpose login to secure.
• Host↔Command trust — self-signed host TLS with TOFU cert pinning + per-host bearer tokens. A changed cert is terminal, not silently retried.
• The registry is the crown jewel — it holds every host's live token; it is written atomically (tmp+rename, 0600) and must be protected like fleet-wide root keys.
• AuthZ is default-deny (internal/authz) — every route requires a specific capability; LDAP/AD groups map to roles (auth/ldap.go).
• Credentials — argon2id password hashing, CSRF-protected sessions, strict CSP, embedded assets (no external fetch).
• Updates are signed — the only supported upgrade artifact is the signed SimpleVirt installer ISO (simplevirt-update-sign / simplevirt-update-verify, internal/update); VerifyUpdateImage decides the format from the file's bytes and refuses anything else, including a legacy .auf bundle. There is no user-facing dev-lab install path.
• Backups — Command state is exported as an audit-logged gzip-tar (backup_pg.go, handlers_backup.go).
• Certificates — self-signed by default; operators can upload a signed cert or generate a CSR and install the result (internal/certs, handlers_certificate.go).

13. Failure & recovery model

The architecture is designed so that no single failure silently corrupts data or lies to the operator.

• Command down — VMs keep running (Command is not in the data path). Orchestration and the central UI pause; on restart the poller rebuilds its live view and the stretch state field is re-reconciled against reality. Inventory and tokens survive in PostgreSQL.
• Host unreachable — inventory is preserved and marked stale, never erased; the operator still sees what was on the Host. If the Host is truly dead (both planes silent past the hold-down) and HA is enabled, protected VMs restart on a survivor gated by lockd.
• Host degraded — treated as alive-but-sick; VMs are left running and are never failed over, because they still hold their disk locks.
• Network partition / split-brain — disk locking + fencing (fcntl leases or heartbeat-lease records, watchdog-backed) prevent two Hosts from writing the same disk. In a Stretched Cluster the Witness breaks quorum ties.
• Storage sickness — liveness reads are isolated from storage RPCs so a bad mount cannot make a healthy Host report as dead; datastore identity binding survives rename/re-add without stranding VMs.
• Partial reconciliation — reconciliation is additive and idempotent, so a failed or partial apply leaves existing VMs and mounts intact and is simply re-applied later; it never tears down.
• Poll failure classification — terminal failures (auth, tls-pin) are surfaced honestly instead of retried behind a hopeful spinner; retryable ones (e.g. a Host regenerating its cert on boot) clear on their own.

14. Binaries & services

Shipping binaries: simplevirtd, simplevirt-command, simplevirt-clusterd, simplevirt-console, simplevirt-command-console, simplevirt-installer, simplevirt-update-sign, simplevirt-update-verify. The guest-tools binaries simplevirt-guestd / simplevirt-guestprobe ship via the guest-tools ISO, not the appliance rootfs. simplevirt-keygen and simplevirt-license are build/vendor tools — they are not built by make appliance-binaries and are installed in neither image.

Core services: simplevirtd.service, simplevirt-command.service, simplevirt-postgres.service, simplevirt-postgres-init.service.

Host appliance services: simplevirt-cephd.service, simplevirt-ceph-boot.service, simplevirt-clusterd.service, simplevirt-console.service, simplevirt-installer.service, simplevirt-migration-host.service, simplevirt-migration-peers.service, simplevirt-migration-tls.service, simplevirt-netcheck.service, simplevirt-updater.service, simplevirt-usermgr.service, simplevirt-wdmd.service.

Command appliance services: simplevirt-command-console.service, simplevirt-netcheck.service.