SimpleVirt Troubleshooting

This guide covers the failures operators hit most often, worst-first. Each entry is written as Symptom → What it means → Checks → Resolution.

Two places you act from:

• A web UI — SimpleVirt-Command (the controller, which manages the whole fleet) or a single Host's own web interface.
• The appliance console — the local text console reached at the machine's keyboard/monitor or its VM console. A few recovery actions live only here, on purpose, so they survive a web-UI outage.

A guiding principle throughout: Hosts keep running their Virtual Machines even when SimpleVirt-Command is down. A controller outage means you cannot manage the fleet; it does not stop your workloads. And SimpleVirt never fails a VM over while its original copy might still be holding the disk — most "stuck" states are SimpleVirt protecting your data, not losing it.

flowchart TD
    A[Something is wrong] --> B{Can you reach<br/>SimpleVirt-Command?}
    B -- No --> C[Command service unhealthy §7]
    B -- Yes --> D{What is affected?}
    D -- A Host --> E[Host offline/unreachable §1]
    D -- A VM --> F[VM won't start §2<br/>Migration failed §3]
    D -- Networking --> G[Distributed Switch not realized §4]
    D -- Storage --> H[Datastore unavailable §5<br/>SimpleVirt HCI degraded §6]
    D -- Certificates --> I[Certificate issues §8]
    D -- Stretched Cluster --> J[Not ready §9]

1. A Host shows offline, unreachable, or degraded

Symptom. In SimpleVirt-Command a Host reads unreachable (its Virtual Machines dim to last-known state) or degraded (VMs render but every Start / Shut down / Power off / Reset returns "host not answering"). HA does not restart the VMs from a degraded Host elsewhere.

What it means. SimpleVirt-Command polls each Host's health endpoint and reports one of connecting, online, degraded, unreachable:

• unreachable — the Host does not answer at all (powered off, management network down, or a genuine crash). If it truly crashed, HA takes over (see below).
• degraded — the Host answers its health check but its hypervisor layer is wedged, most often parked on a dead storage mount. SimpleVirt deliberately does not fail these VMs over, because they are most likely still running and holding their disk locks; failing them over would risk a second writer.

Checks.

• Confirm reachability of the Host's management address from the controller (ping/port check). A degraded Host still answers /healthz while its VM operations time out — that distinction is why SimpleVirt separates "degraded" from "unreachable".
• On the Host's local console, check whether the hypervisor service (simplevirtd) is active or stuck, and whether a Datastore mount is hung (see §5).
• Watch the status: a Host that is merely rebooting flips back to online within a short poll window; a lasting unreachable is a real outage.

Resolution.

1. Do not try to start a degraded Host's VMs on another Host — they are probably still running.
2. If the management network is the fault, restore it; the Host re-registers on the next poll.
3. If the hypervisor layer is wedged, open the Host console, and once storage responds (§5) restart the Host agent. It usually unwedges when the mount answers.
4. Back in SimpleVirt-Command, use Retry now to re-probe once the Host recovers.
5. For a genuinely dead Host, confirm HA has adopted its protected VMs elsewhere before powering the Host back on, so the same VM never runs twice.

2. A Virtual Machine won't start

Symptom. A VM refuses to power on. Common messages point to a held disk lease ("failed to get lock" / "resource busy"), missing storage ("cannot access storage" / "no such file"), or an admission refusal.

What it means. A start is blocked when one of the following is true:

• The VM's disk lease is still held — a crashed peer left it behind, or an HA-adopted copy is genuinely running elsewhere. SimpleVirt keeps a single writer per disk on purpose.
• The Datastore is not mounted on this Host (see §5). SimpleVirt auto-heals a renamed Datastore by re-binding the disk, but not an unmounted one.
• The backing disk image is gone — deleted or moved out-of-band while the VM definition still references it.
• Admission control refused the power-on because the cluster could not hold the HA memory reserve for it.

Checks.

• In SimpleVirt-Command, confirm no other Host shows this VM running or paused — watch for an HA-adopted copy after a failover.
• Confirm the Datastore holding the VM's disk is mounted on the target Host.
• Read the Task/Event detail for the exact reason (lease, missing file, or admission).

Resolution.

1. If the Datastore is simply unmounted or was renamed, re-add/mount it and retry Start — SimpleVirt re-binds the disk automatically.
2. If a peer crashed holding the lease and you have confirmed no live copy exists anywhere, recover the peer or let its lease expire, then Start again. Never clear a lock while another Host might be writing the disk.
3. If the backing disk was deleted, restore it from backup (or re-run the storage migration that moved it) so the referenced volume exists again. Prevent recurrence by removing VMs through the Delete VM action rather than deleting disk files directly.
4. If admission control refused the start, free reserve capacity (power off or migrate other VMs) or adjust the HA reserve policy, then retry.

3. Live migration fails or is refused

Symptom. A live migration is refused up front by the precheck, or it starts and then errors partway.

What it means. SimpleVirt runs a migrate precheck before moving a running VM and refuses (rather than risk a broken move) when it cannot prove the move is safe. The most common refusals:

• A disk is on local storage — a live move needs every disk on a shared Datastore both Hosts mount. Use cold migration to copy the disks instead.
• The destination doesn't mount the shared Datastore holding the VM's disk.
• Port-group VLAN mismatch — a Network with the same name on the destination is on a different VLAN, which would land the VM in the wrong broadcast domain and silently lose connectivity. A same-name match alone is not "same network".
• Source unreachable — the precheck cannot read the source Host's networks, Datastores, or PCI devices to confirm compatibility, so it fails closed and asks you to retry when the source is reachable.
• PCI passthrough incompatibility between the two Hosts.

Once the precheck passes, the transfer itself rides libvirt's migration ports on the migration network — a control channel on TCP 16509 and the data channel on TCP 49152–49215. The data channel is mutually authenticated TLS by default: the controller auto-issues a short-lived migration leaf certificate to each Host, while the 16509 control channel stays plaintext and is restricted by each Host's firewall (see the Migration network prerequisite in Installation). A failure after a clean precheck is therefore usually reachability on those ports — a connection that is refused, times out, or resets — though it can also be a missing or not-yet-provisioned migration leaf on a Host that has just booted or just been added, which typically resolves on retry once the controller has issued the leaf.

Checks.

• Read the precheck verdict — it names the exact blocker (which disk, which port group, which VLAN).
• Confirm both Hosts mount the same Datastore and that the destination has a matching Network on the same VLAN.
• Confirm the source Host is online (not degraded/unreachable) so the precheck can read it.
• For a mid-transfer connection error, confirm both Hosts have a migration adapter up on the same migration VLAN (Host → Settings → Networking) and can reach each other on it, and that both are still paired to this controller — the controller pushes each Host's allowed-peer list, so a Host that was re-addressed or re-paired may not yet be in its peer's firewall.

Resolution.

1. For a local-disk or unshared-Datastore refusal, use cold migration (which copies the disks) or first storage-migrate the disk onto shared storage.
2. Align the destination's port group to the source VLAN, then retry.
3. If the source is unreachable, restore it and retry — the precheck re-runs.
4. If the transfer fails after a clean precheck with a connection error, treat it as a network/firewall problem. Confirm both Hosts' migration adapters are up on the same VLAN and can reach each other, and that both are online in this controller. Each Host re-applies its migration firewall shortly after boot and on every membership change, so a Host that has just rebooted or just been added may need a short moment (or a re-poll from Retry now) before it accepts its peers. Then retry the migration.

4. A Logical Distributed Switch is not realized on a Host

Symptom. A Logical Distributed Switch is declared on the Cluster, but one member Host shows it unrealized: its Networks/port groups are missing there, and VMs on that Host can't reach the expected VLAN. Migration into that Host is refused on a network check (§3).

What it means. A Logical Distributed Switch is declared once per Cluster and realized on each member — each Host must map the switch's uplink slots to its own physical NICs (Uplinks) and stand up the port groups locally. Realization fails on a member when its Uplinks aren't mapped, the chosen physical NICs are missing or down, or the Host was unreachable when the switch was pushed.

Checks.

• In the Cluster's Networks view, look at per-member realization state and the network-health rollup; the unrealized member is flagged.
• On that Host, confirm the physical NICs backing the switch's Uplinks exist and are up, and that the uplink slots are mapped.
• Confirm the Host is online; an unreachable Host cannot realize a pushed switch until it returns.

Resolution.

1. Map the member's Uplinks to real, live physical NICs, then re-push/realize the Distributed Switch to that Host.
2. If the Host was offline during the change, bring it back online and re-realize — declaration is intent; realization catches up per member.
3. Verify the port groups now exist on the Host and the VLAN-health rollup is green before relying on migrations into it.

5. A Datastore is unavailable

Symptom. A Datastore shows unavailable on one or more Hosts. VMs backed by it won't start (§2), a running VM keeps working but can't be power-cycled, and "Power off (force)" may hang for a spell.

What it means. The Datastore's backing storage is unreachable (an NFS export is down or its server is unreachable), or the mount is wedged on one Host while healthy on others. A guest whose storage vanished blocks in the kernel until the storage times out and cannot be signalled before then — and that stuck operation holds the Host's per-VM job lock, so later force-off/delete/reset attempts queue behind it.

Checks.

• From the affected Host, confirm the storage server/export is reachable and the mount responds.
• Check whether it's one Host (a wedged mount) or all Hosts (the backing store is down).
• Look for a full Datastore — a full backing filesystem behaves like an unavailable one for writes.

Resolution.

1. Fix the storage first — restore the export/server or free space. Once storage responds, blocked guests unblock and normal power operations work again.
2. Re-add or remount the Datastore where the mount was wedged; SimpleVirt re-binds VM disks to the active Datastore automatically after a rename/re-add.
3. Only if storage cannot be recovered and a VM must be forced down, do so from the Host console at the hypervisor level — and confirm the VM is not still running on its original Host before powering it on anywhere else.
4. Prefer shared Datastores mounted with server-side locking so leases expire on their own after a Host failure.

6. SimpleVirt HCI reports degraded

Symptom. The SimpleVirt HCI console shows the pooled Datastore degraded or recovering rather than healthy, and may raise Alarms about reduced protection or an imbalance between Hosts.

What it means. SimpleVirt HCI presents pooled, resilient cluster storage as a single Datastore reachable from every member; it keeps multiple copies of your data across Hosts. Degraded means it is still serving reads and writes but one or more copies are missing (a Host or its disks are down), so it is temporarily below full redundancy. Recovering means it is rebuilding those copies after a fault; protection is returning on its own. This is the resilience working — not data loss — but it should be resolved promptly so a second failure doesn't overlap the window.

Checks.

• In the SimpleVirt HCI console, confirm how many member Hosts and their disks are currently participating versus expected.
• Confirm quorum among the storage monitors — a lost monitor majority is more serious than a single missing disk.
• Check per-Host capacity balance; a large imbalance is called out as an Alarm.

Resolution.

1. Bring the missing Host or disks back online — SimpleVirt HCI re-adds them and rebuilds copies automatically; watch the state move degraded → recovering → healthy.
2. If a disk is genuinely dead, replace it and let the pool rebuild onto the replacement.
3. Do not remove Hosts or disks while the pool is already degraded — wait for full protection to return first.
4. If the pool is short on space, add capacity before it fills; a full pool blocks writes.

Operators manage SimpleVirt HCI entirely through the SimpleVirt HCI console — never run raw storage-engine commands against a production pool. (Its internal engine is described in architecture.md.)

7. SimpleVirt-Command is unhealthy or won't start

Symptom. The SimpleVirt-Command web UI is unreachable or errors, and the whole fleet shows unmanaged. A health probe to GET /healthz does not return {"status":"ok"}.

What it means. The controller process is down or unhealthy. SimpleVirt-Command persists its state in a bundled PostgreSQL database; if that database is unavailable, the controller cannot serve. Managed Hosts keep running their VMs throughout — only fleet management is affected.

Checks.

• Probe GET /healthz on the controller. 200 {"status":"ok"} means the web tier is up; no response means the service is down.
• On the controller appliance, check that the controller service and its bundled PostgreSQL service are both active.
• Check free space on the controller's data volume — a full or read-only volume causes persistence warnings and can stop the service from starting cleanly.

Resolution.

1. If PostgreSQL is not running, start it, then the controller; the controller depends on the database being up first.
2. If the data volume is full or read-only, free space or remount read-write, then restart the controller so persistence resumes.
3. If the controller VM's disk is lost, reinstall the SimpleVirt-Command appliance, seed an operator account at the console, and restore your most recent backup under Administration → Backup/Restore. Without a backup, reinstall and re-pair each Host from Hosts → Add.
4. Prevention: on a schedule, download a controller backup and store it off the controller VM, keeping several generations. All controller state — including each Host's pairing — lives on the controller, so an off-box backup is your only fast path back.

8. Certificate issues

Symptom. Browsers warn on the SimpleVirt-Command or Host web UI, or a signed certificate you installed is rejected.

What it means. The certificates you manage in SimpleVirt are the web UI / agent API certificates; live migration uses its own auto-issued certificates, which you do not manage directly:

• Web UI certificate. Appliances serve a self-signed certificate by default. You can either upload your own signed certificate or generate a CSR on the appliance, have it signed, and install the result. A browser warning with the default cert is expected; a rejection of an uploaded cert usually means a chain/key mismatch or a name that doesn't match the address you browse to.
• Live migration uses its own per-host certificates. By default the data channel (TCP 49152–49215) is mutually authenticated TLS: the controller auto-issues a short-lived migration leaf certificate to each Host (at startup, on host-add, and on renewal). The libvirt control channel on TCP 16509 stays plaintext and is restricted by each Host's migration firewall rather than by a credential. A migration that fails after a clean precheck is usually a reachability problem on those ports, but it can also be a migration leaf that has not been provisioned yet on a freshly booted or freshly added Host — one that generally clears on retry; diagnose it under §3, not here.

Checks.

• For a UI cert rejection: confirm you uploaded the full chain and the matching private key, and that the certificate's subject/SAN includes the exact hostname or IP you use.
• For a Host that reads unreachable with a certificate-pinning failure after a redeploy: the Host presents a new self-signed certificate, which the controller refuses because it no longer matches the fingerprint pinned when the Host was added. Re-add the Host so the new certificate is pinned.

Resolution.

1. UI: re-upload the signed certificate with its full chain and correct key, or regenerate the CSR, get it signed, and install — using a subject/SAN that matches how operators reach the appliance. The default self-signed cert is fine for a lab; use a signed one in production.
2. A re-imaged Host: remove it from Hosts and add it again, so the controller pins its new certificate. (Nothing needs re-provisioning for migration — it holds no certificates.)

9. A Stretched Cluster is not ready

Symptom. The Stretched Cluster readiness report reads "Not ready" and Enable is refused with a message to resolve every required item first. Common unmet items name the topology, an offline site's Hosts, or the Witness.

What it means. A Stretched Cluster spans Site A, Site B, and a Witness at a third location. Because a wrong "ready" verdict on a safety-critical feature could expose one VM disk from two sites, readiness fails closed: any required item that is unmet — or that SimpleVirt cannot verify right now — blocks enablement. Required items include exactly two independent data sites, enough reachable Hosts assigned to each site, and, for the SimpleVirt HCI storage backend, a defined Witness. The Witness holds the tiebreaker only; it never runs VMs or stores VM data.

Configuration records intent; enable/disable activate or exit stretch mode and are gated on this readiness report. (Stretched-cluster orchestration is software-complete and unit-tested; final command execution and live probes are validated against a real two-site cluster before you rely on it in production.)

Checks.

• Open the readiness report and read each required item's Detail and Remedy — it tells you exactly what to fix (assign a Host to a site, bring a site's Hosts online, add a Witness).
• For the SimpleVirt HCI backend, confirm a Witness is defined and its tiebreaker monitor is in quorum. If the storage health rollup says the tiebreaker cannot arbitrate, the Witness is the blocker.
• Confirm both data sites have their assigned Hosts reachable — an item can be unmet simply because a site's Hosts are offline and the topology can't be confirmed.

Resolution.

1. Assign exactly two data sites (one primary, one secondary) and place every participating Host in a site.
2. For the SimpleVirt HCI backend, add a Witness at a third location and confirm its tiebreaker monitor joins quorum.
3. Bring any offline site Hosts back online so their state can be verified — unknown items block readiness the same as failing ones.
4. Re-open the readiness report; when it reads "Ready — every stretch requirement is met.", Enable is accepted.

The NFS_SHARED backend uses an external shared Datastore reachable from both sites; the Witness is optional there, and SimpleVirt owns compute HA and fencing only — it does not claim the external storage array itself survives the loss of its own site.

Quick reference

See also: Operations · Architecture · Installation · Known limitations