SimpleVirt Operations Guide

Day-2 procedures for running a SimpleVirt fleet: routine health checks, planned maintenance, upgrades, storage capacity, and where to look when something needs attention.

This guide assumes SimpleVirt is installed and clusters exist. For first-time setup see installation; for concepts see architecture; for problems see troubleshooting.

Daily health check

SimpleVirt-Command aggregates the state of the whole fleet. Start every day-2 session here.

• Health — a single dashboard of Host CPU/memory/load, Datastore capacity, service up/down, VM state counts, and computed Alarms. The overall status is ok, warning, or critical. A Host is only reported healthy when its core services (simplevirtd, libvirtd, networking) are active.
• Tasks — every mutating operation (migration, evacuation, update, stretch enable) runs as a tracked Task with progress. Watch long operations here.
• Events — an append-only activity stream, delivered live over SSE. Use it to see what happened and in what order.
• Alarms — computed warnings/criticals (capacity thresholds, unreachable Hosts, degraded storage, network-health rollups).

flowchart LR
  A[SimpleVirt-Command] --> H[Health dashboard]
  A --> T[Tasks]
  A --> E[Events stream]
  H --> AL[Alarms]

A green Health status with no open Alarms and no stuck Tasks is the normal steady state.

Host maintenance

SimpleVirt separates three distinct workflows so a planned service window never behaves like an unplanned failure.

Preview before you act

Before entering maintenance, run the read-only precheck (Host → Maintenance → Preview). It plans — but never performs — the evacuation of each running VM and reports:

• Per-VM fate: migrate (to a named survivor) or blocked (with the reason).
• SimpleVirt HCI impact: storage health, OSDs up/in, monitor quorum — read from the background sample, so it is fast and never disturbs live storage.
• A single verdict: ok, warn, or blocked.

The preview mutates nothing. Use it to confirm the window is safe before committing.

Evacuate a Host (keep VMs running)

Host → Evacuate. Every running VM with shared storage (SimpleVirt HCI or a shared Datastore) is live-migrated onto surviving cluster members. Each move is its own tracked Task; the response lists what started and what could not be placed. A VM with no valid target is reported and left running — evacuation never powers a VM off to make progress.

Live migration crosses the dedicated migration network when the destination has a migration adapter on one. If it has none, the move is refused unless you explicitly allow it to cross the management network (fail-closed by default). Guest data is encrypted and mutually authenticated in flight by default (qemu+tls / VIR_MIGRATE_TLS); the reason that opt-in exists, and why a dedicated, isolated migration VLAN is a deployment requirement, is that the libvirt control listener on TCP 16509 is unauthenticated — see Migration network.

Enter / exit maintenance mode

• Enter — Host → Enter Maintenance. Running VMs are gracefully powered down; the Host stops accepting new VMs and power-ons. To keep workloads running, evacuate first, then enter maintenance on the now-empty Host.
• Exit — Host → Exit Maintenance. The flag is persisted, so a Host in maintenance stays in maintenance across a simplevirtd restart until you explicitly exit.

flowchart LR
  P[Preview precheck] --> EV[Evacuate VMs]
  EV --> M[Enter maintenance]
  M --> W[Service / reboot Host]
  W --> X[Exit maintenance]

VM migration

• Live migration — moves a running VM's memory and device state to another Host with near-zero downtime; disks stay on shared storage. It uses libvirt's migration transport on the dedicated migration network: a control channel on TCP 16509 and data on TCP 49152–49215. Guest RAM and domain XML are encrypted and mutually authenticated in flight by default (qemu+tls / VIR_MIGRATE_TLS), but the libvirt control listener on 16509 is itself unauthenticated, so that channel is restricted by each Host's migration firewall rather than by a credential (Migration network).
• Storage migration — moves a running VM's disks between Datastores (block-copy + pivot) with the VM online.
• Cold / offline migration — moves a stopped VM, copying disks off local storage when needed.

Migration runs a precheck (target reachability, network/port-group availability, storage) before it starts, and each migration is a tracked Task. To move many VMs at once, use Host Evacuation rather than migrating VMs individually.

Cluster maintenance

For a rolling service window across a Cluster:

1. Confirm cluster Health is green and HA readiness passes while all members are up.
2. For each member in turn: run the maintenance preview, evacuate, enter maintenance, service, exit maintenance, and confirm the member rejoins before moving to the next.
3. Keep enough spare capacity that the surviving members can hold the evacuated workload. Admission control reserves failover headroom; a member that cannot be evacuated without breaching it is reported blocked in the preview.

Never take more than one member offline at a time unless the preview confirms the remaining members can carry everything.

Backups (controller state)

SimpleVirt-Command backs up its own state — the fleet inventory, users, library metadata, tasks, and HA state. (VM data lives on Datastores and is not part of this backup.)

• Back up — Administration → Backup → Download. Produces a single gzip-tar of the controller's state directory, streamed as a download. It contains secrets (Host credentials, join tokens, the local password store), so every backup is audit-logged.
• Restore — upload a backup. The controller swaps in the restored state and restarts; systemd brings it back up on the restored data. Restore is also audit-logged.

Take a backup before any controller upgrade and after significant fleet changes (adding Hosts/Clusters, RBAC or directory changes).

Certificate maintenance

Administration → Certificates manages the HTTPS certificate a browser sees when reaching SimpleVirt-Command. The default is a self-signed pair (browsers flag it). To install a trusted certificate:

• Upload a certificate your CA already signed, together with its key, or
• Generate a CSR on the appliance, have your CA sign it, and install the result against a key that never leaves the appliance.

Every install is validated before it can replace the running certificate, so a bad upload can never take the console offline. Certificate management requires the admin.users privilege.

If you pinned explicit certificate/key files in the controller's configuration, that is treated as deliberate: the management endpoints report that the certificate is externally managed rather than overriding your choice. Manage it in the config file in that case.

Lifecycle and updates

SimpleVirt appliances are updated by uploading the signed SimpleVirt installer ISO through the Lifecycle view — the same .iso that installs an appliance is the one that upgrades it. There is no separate reinstall step, and no second artifact: an upload that is not an installer ISO is refused, including an older .auf update bundle.

An ISO is built for exactly one appliance flavor — Command or Hypervisor — and carries a new appliance root filesystem, kernel, and initramfs. Its manifest is signed; the signature and a per-payload checksum are verified twice (by the unprivileged daemon on upload, and again by the privileged apply helper before anything is written), so a tampered or truncated upload can never reach a reboot. The format is decided by the file's contents, not its name, so renaming a file does not get it past the check. A Command ISO applied to a Hypervisor (or vice-versa) is rejected.

The update flow

flowchart LR
  U[Upload signed installer ISO] --> V[Verify signature + checksums]
  V --> S[Stage onto target disk]
  S --> A[Apply: one-shot boot into in-place upgrade]
  A --> R[Reboot; VMs, storage, config preserved]

1. Upload the .iso in Lifecycle. The controller verifies it and marks it staged.
2. Apply. Staging writes the new image to the target's own disk and arms a one-shot boot into the already-proven unattended in-place upgrade, which overlays only the OS — VMs, storage, and configuration are preserved. The Host reboots into the new version.

The Lifecycle table shows the whole fleet in one place: each Host's running version, what it has staged, whether it can apply, and the outcome of the last apply (so a Host that verified an image but had its apply refused is distinguishable from one never touched). SimpleVirt-Command updates itself the same way.

Sequencing: update SimpleVirt-Command first (take a controller backup beforehand), then update Hosts. Evacuate a Host before applying its update so its VMs keep running during the reboot.

SimpleVirt HCI: health and capacity

SimpleVirt HCI is pooled, resilient cluster storage presented as a Datastore reachable from every member. The HCI console (Datastore → HCI) reports protection and capacity in plain terms; you do not manage the storage engine directly.

Health

• Protection is proven from the live failure-domain topology, not inferred from configuration. Each Datastore reports its number of copies, minimum copies, whether copies are separated across Hosts, and how many Host failures it can currently tolerate. State is healthy, reduced, or unknown (reported honestly when it cannot be proven).
• Watch for reduced protection after a Host or disk loss — it means the pool is rebuilding redundancy and can tolerate fewer further failures until it finishes.

Capacity

Capacity is layered, not a single division:

• Raw — total physical.
• Usable — theoretical after replication.
• Available — the honest "how much can I actually still write," accounting for balance across Hosts.

Plan against Available, not raw. Add capacity (or investigate imbalance) as Available approaches your headroom threshold; capacity Alarms surface on the Health dashboard.

Stretched Cluster operations

A Stretched Cluster spans two sites plus a Witness for tiebreaking. It has two storage backends:

• SimpleVirt HCI — SimpleVirt owns a synchronous cross-site copy of the data (Ceph stretch mode + a tiebreaker monitor at the Witness) and requires a Witness. This backend does survive the loss of an entire storage site, because a complete copy already lives in the surviving site.
• NFS Shared — an external shared Datastore reachable from both sites, which your storage layer has already replicated between them. Here SimpleVirt owns compute HA and fencing only; it does not make the external array itself survive a storage-site loss. Use it only when your storage is already stretched by other means.

Status: validated in simulation and single-site lab; full two-datacenter hardware validation is pending. The feature has graduated out of the experimental preview gate (its tab and endpoints need no flag), but graduation did not relax any safety check — enable still refuses on any cluster that is not a genuine two-site topology with a Witness. Treat a first activation on real two-site hardware as a carefully monitored change. See docs/SIMPLEVIRT_STRETCHED_CLUSTER.md.

Readiness (always start here)

Stretched Cluster → Readiness. The engine answers one question honestly: is this cluster a valid, safe stretched cluster right now, and if not, exactly which requirements are unmet.

• It blocks, it does not warn-through: the cluster is Ready only when every required item is provably met. A required item that cannot be evaluated is unknown, and unknown never counts as ready (fail-closed).
• Requirements (reachability, MTU, two-site topology, Witness) block. Recommendations (latency, bandwidth) are advisory and never block.
• A single-site cluster correctly comes out Not Ready, with the exact missing topology named.

Configure and enable

1. Configure intent (PUT). Records the desired stretched topology — Site A / Site B / Witness — as intent only. It does not activate storage stretch mode or change site networking.
2. Preview the activation plan. Non-mutating; never implies enablement.
3. Enable. Refused unless the cluster is Ready. On the SimpleVirt HCI backend, enable activates stretch mode (site-separated placement across both sites with a tiebreaker) idempotently and rolls back on any partial failure; activation is audit-logged.
4. Disable withdraws stretch arbitration and clears the enabled flag (it does not, by itself, relax redundancy settings).

Site maintenance

Draining a whole site is a controlled, reversible site "loss." Stretched Cluster → Site Maintenance → Precheck proves the surviving site can safely carry everything before you drain: it reports the number of VMs that must move, the surviving site, the memory to relocate versus the surviving site's free memory, and any blockers (worst-first). The precheck is fail-closed — any blocker refuses the drain. It mutates nothing; confirm before proceeding.

Degraded and recovery

The site-protection rollup reports one of: protected, degraded, recovering, or unknown.

• Degraded — a site or link is impaired; the surviving site continues to run workloads, arbitrated by the Witness. Do not start a planned site drain while degraded.
• Recovering — the impaired site is back and redundancy is rebuilding across sites. Let it reach protected before the next planned maintenance.
• Unknown — SimpleVirt cannot currently prove the posture; investigate reachability to the sites and the Witness before making changes.

Logs and events

Where to look when you need detail beyond the dashboards.

In-product

• Events — the live, append-only fleet activity stream (SSE).

• Tasks — per-operation progress and outcome.

• Alarms — persisted health alarms. An alarm fires when one of the health thresholds first breaches, stays until the condition actually goes away, and can be acknowledged by name so the rest of the team can see somebody owns it. GET /api/v1/alarms is what is open now; GET /api/v1/alarms/history is what has been open — including alarms that have since cleared, which is how you answer "was anything wrong overnight" the next morning. Acknowledging an alarm never clears it: only the condition clearing does that.

  A configured webhook is notified on every fire, acknowledge and clear (see known-limitations §7 — it is off unless you configure it).

Support bundle

GET /api/v1/support-bundle (administrators only) downloads one .tar.gz holding what is needed to diagnose a fault after the fact: build versions for the controller and every host, the controller's configuration, the fleet and cluster inventory, recent tasks, HA state, the alarm list and history, and the tails of the controller's audit and service logs.

Secrets are redacted. Host bearer tokens, the directory bind password, Kubernetes join tokens and kubeconfigs, the database password, private keys, session cookies and Authorization headers are replaced with [REDACTED] before anything is written, so the bundle is safe to send on. It is also size-bounded — logs are included as tails — so producing one on a busy controller is cheap.

Collecting one. There is no button for this in the console yet; fetch it from the API. Sign in as an administrator, then GET /api/v1/support-bundle with that session's cookie — a GET carries no CSRF token, see api-reference § Authentication. The reply is an attachment named simplevirt-support-<UTC timestamp>.tar.gz. Collect it while the fault is still live: it is a point-in-time snapshot, and the logs it carries are tails, so a bundle taken after a restart may no longer contain the window you care about.

Where to send it. SimpleVirt does not ship a support address, and this guide will not invent one — use whatever channel your subscription or support agreement names. Until then, keep the bundle: it is the evidence, and it is the part that cannot be reconstructed later.

On the appliances (systemd)

Use systemctl status <unit> and journalctl -u <unit> on the appliance.

SimpleVirt-Command:

SimpleVirt-Hypervisor (Host):

The Host TUI (simplevirt-console) and the Command TUI (simplevirt-command-console) also give on-appliance status and recovery entry points when the web console is unreachable.

Quick reference

See also: installation · architecture · troubleshooting · known limitations · backup responsibility