SimpleVirt Stretched Cluster

A stretched cluster runs one SimpleVirt cluster across two independent sites (two datacenters, two rooms, two power/network domains) with a third-location witness, so that the loss of an entire site does not take the workload down with it. This document is the operator reference: what it is, the two storage backends, the witness, the prerequisites and fail-closed readiness, how it behaves during failures, and how to take a site down for maintenance.

Read the Status section first. This capability is validated in simulation and in a single-site lab; full two-datacenter hardware validation is still pending. It has graduated out of the experimental preview gate, but it is not yet a hardware-proven production feature. Nothing about it will activate on a cluster that is not a genuine two-site topology with a witness.

1. What a stretched cluster is

An ordinary SimpleVirt cluster assumes its hosts share a fate within one location. A stretched cluster deliberately does not: it is partitioned into failure domains so that one whole site can fail — power, cooling, the inter-site link, the lot — and the surviving site keeps (or quickly restarts) the workload.

A stretched cluster has exactly three roles:

• Site A and Site B — the two data sites. Each holds hosts that run VMs and, for the HCI backend, store a copy of the data. Neither site is a subordinate of the other; they are peers.
• Witness — a single box at a third location that holds no VM workload and no VM data. Its only job is to break ties: to decide, when the two sites can no longer talk to each other, which single site is allowed to keep running. See The witness.

The reason the third location matters is arithmetic. With only two sites, a clean network cut between them is unresolvable — each site sees the other as "gone" and, without an arbiter, both would try to run everything. That is split-brain, and on shared storage it corrupts data. The witness is the odd vote that makes exactly one side win. See Failure behavior.

2. The two storage backends

Stretch is a cluster topology; the storage underneath it is a pluggable backend. SimpleVirt supports two, and they protect data very differently. Choose deliberately.

SimpleVirt HCI — true synchronous cross-site storage

With the SimpleVirt HCI backend, SimpleVirt owns the storage and stretches it itself. It places a synchronous, replicated copy of every write in both sites before acknowledging it, using Ceph's native stretch mode with a tiebreaker monitor at the witness. Concretely, on enable SimpleVirt:

• tags each data site as its own CRUSH datacenter failure domain (the product generates the tags and host/monitor placement — you do not hand-edit CRUSH);
• moves the data-site monitors into their datacenters and the tiebreaker monitor into the witness location;
• creates the stretch CRUSH rule and raises each replicated data pool to the stretch geometry (size 4 / min_size 2 — two copies per site, writes still ack with two copies surviving a site loss); and
• enables stretch mode against the tiebreaker.

Because the data is genuinely present in both sites, HCI survives the loss of an entire storage site: the surviving site already holds a complete, consistent copy and keeps serving reads and writes. This is the backend to pick when you want site-level data protection that SimpleVirt manages end to end.

Erasure-coded pools are excluded from the stretch floor. The size-4/min_size-2 geometry is a replicated geometry; it cannot be applied to an erasure-coded pool. SimpleVirt therefore excludes any erasure-coded data pool from the floor and tells you it did so (it is logged and audited, never silently dropped). The enable still proceeds for your replicated pools.

NFS Shared — compute-stretch over externally-replicated storage

With the NFS Shared backend, SimpleVirt does not own or stretch the storage. You bring a shared datastore that is reachable from both sites — typically an array (or pair of arrays) that your storage team has already replicated between the two locations by some other means. SimpleVirt stretches only the compute layer over it: site-aware placement, HA failover, and fencing, arbitrated by an active witness (see below).

This is the critical honesty point: the NFS Shared backend does not make your storage array survive a storage-site loss. If the site that actually holds the live copy of the data goes down and your external replication has not already made the data available at the surviving site, the VMs have nothing to run from — SimpleVirt cannot conjure a copy it never held. NFS Shared buys you compute failover and split-brain-safe fencing; the data's cross-site survival is entirely your storage layer's responsibility. Use it only when your storage is already stretched by other means.

3. The witness

The witness is one box at a third location — a third room, a third building, a cloud instance; anywhere with an independent path to both sites. It is not a third data site. It runs no VMs and stores no VM data; a readiness check refuses a "witness" that still holds data or runs workload, because such a box is a disguised third data site, not an independent arbiter.

The single witness box wears two hats, one per layer of the stack:

• Ceph tiebreaker monitor (HCI only). For the HCI backend, the witness runs the Ceph tiebreaker monitor. In Ceph stretch mode this monitor is what lets a surviving site keep I/O going after the other site is lost, and it is deliberately marked so it can never itself become the sole surviving monitor that re-forms a quorum on its own.
• Cluster-services quorum witness. For the Command-independent recovery path (simplevirt-clusterd, see §5.5), the witness is the third quorum vote. When the two data sites are partitioned, Raft grants quorum — including the witness's vote — to only one side. The side that cannot form a quorum authors nothing, so it can never start VMs the winning side is also starting.

Both hats are about the same thing from two angles: making sure that when the sites split, exactly one side has the authority to run the workload.

4. Enabling a stretched cluster

Operate a stretched cluster from a cluster's Stretched Cluster tab in SimpleVirt Command. Configuration records your intent (which hosts are in which site, where the witness is); activation only ever happens once SimpleVirt can prove the topology is safe.

Prerequisites

• Exactly two data sites, each with at least the minimum number of reachable, assigned hosts.
• A witness at a third location — for HCI, a reachable host that will carry the tiebreaker monitor and that holds no OSDs and runs no VMs.
• Inter-site reachability between the two data sites, and witness reachability from both sites, with the required MTU.
• For HCI: the data pools you care about are replicated (erasure-coded pools are carried along but excluded from the stretch floor, as above).
• For NFS Shared: a shared datastore mounted and reachable from both sites, already replicated between them by your storage layer.

Fail-closed readiness

The readiness engine answers one question honestly: is this cluster a valid, safe stretched cluster right now, and if not, exactly which requirements are unmet? It is deliberately conservative:

• It blocks; it does not warn-through. The cluster is Ready only when every required item is provably met. A required item SimpleVirt cannot evaluate right now is unknown, and unknown never counts as ready. This is fail-closed on purpose: a wrong "ready" verdict on a safety-critical feature could expose one VM's disk to writes from two sites at once.
• Requirements block; recommendations advise. Reachability, MTU, two-site topology, and the witness are requirements — unmet means not ready. Latency and bandwidth are recommendations — they are surfaced but never block, because SimpleVirt invents no universal latency threshold for your workload.
• A single-site cluster correctly reads Not Ready, naming the missing second site. This is why graduating the feature out of its old preview flag did not weaken anything: a cluster that is not genuinely two-site-plus-witness simply cannot be activated.

enable re-runs this gate and refuses unless the cluster is fully Ready, so a lab that is not a real two-site deployment never activates stretch mode.

5. Failure behavior

This section is what a stretched cluster is for. The guiding rule throughout is: two sites must never both run (and write for) the same VM.

5.1 Same-site-first restart

When a single host fails, SimpleVirt restarts its VMs on another host in the same site whenever a local survivor qualifies. The workload stays next to its storage, and no cross-site traffic is incurred. A VM crosses to the other site only on a genuine full-site loss (next).

5.2 Cross-site failover only on a corroborated full-site loss

SimpleVirt crosses a VM to the other data site only when the VM's own site is genuinely, wholly lost — not merely busy, briefly unreachable, or short on capacity. "Lost" is corroborated, never assumed:

• For HCI, every host in the site being unreachable is confirmed against the live Ceph site-protection verdict: if the site's OSDs and a monitor are still in quorum, the site is not lost — it is a management-plane blip — and no cross-site restart happens.
• For NFS Shared, all of the site's compute hosts being unreachable is the full compute loss; the destination site's ability to actually reach the VM's storage is proven separately before any restart (fail-closed if it cannot be proven).

5.3 Management-plane partition vs. real site loss

A common and dangerous confusion is a site that SimpleVirt Command can no longer reach but that is perfectly alive and still running its VMs. SimpleVirt treats this as what it is — a management-plane partition, not a site loss — and does not fail the VMs over. Restarting a still-running VM elsewhere on shared storage is exactly the double-write a stretched cluster exists to prevent.

5.4 Split-brain prevention (the witness in action)

When the inter-site link itself is cut, both sites can see the other as gone. The witness breaks the tie:

• HCI relies on the Ceph tiebreaker monitor: only the side that retains monitor quorum (with the tiebreaker's help) keeps I/O; the isolated side's storage stops accepting writes.
• NFS Shared uses an active witness arbiter. Before any cross-site failover, the surviving side must prove it is the quorum-winning side — that it still reaches the witness. The decision is three-valued and fail-closed:
  • Proceed only if the surviving side still reaches the witness (it is the quorum winner) and the failed peer is genuinely unreachable.
  • Deny if the surviving side has lost both the peer and the witness (it is the isolated minority — it must stand down, never blindly start the peer's VMs), or if the peer is in fact still directly reachable (a management partition, peer likely still running).
  • Hold if a fresh probe ran but the witness vote is indeterminate — a bounded hold that waits rather than guess, and only releases to proceed once the ceiling elapses (a genuine, total loss), so a real site loss still eventually recovers.

In every partition picture, at most one side is ever authorized to take over the other's VMs.

5.5 Recovery when Command is down

The behaviors above run in SimpleVirt Command. Command itself can be in the site that failed, so SimpleVirt also has a Command-independent recovery path, simplevirt-clusterd, that carries the same site-awareness and the same anti-split-brain discipline:

• Same-site first, and cross-site only on a genuine full-site loss, exactly as above.
• Quorum-arbitrated. The partition that cannot hold a fresh cluster-services authority lease — the isolated minority, because Raft grants quorum (including the witness vote) to only one side — authors nothing. It can never start VMs the quorum-winning side is also starting.
• Single-writer on start. The recovery start path fences the previous owner before a VM is brought up on a new host, so two hosts can never write one disk even across the handover.
• Non-stretch clusters are unaffected — with no site tags, the recovery path is exactly the original first-eligible-host behavior.

6. Site maintenance and evacuation

Taking a whole site down on purpose (planned power work, a network migration) is a first-class operation, not an induced failure. From the Stretched Cluster tab:

1. Precheck (non-mutating). SimpleVirt re-runs the fail-closed gate for the specific site you want to drain: can the surviving site actually take this site's workload and storage right now? It reports the decision and any blockers without changing anything.
2. Enter maintenance. If the precheck passes, SimpleVirt live-evacuates the site's VMs to the surviving site and records the maintenance intent, so the VMs are moved deliberately rather than restarted by HA after the fact. The operation refuses (with precise remediation) unless the site can safely enter maintenance.

Because the drain is gated by the same honesty engine as everything else, you cannot evacuate a site into a surviving site that could not safely hold the result.

7. Status

This feature is validated in simulation and in a single-site lab. Full two-datacenter hardware validation is PENDING.

What that means concretely:

• Validated in simulation. A deterministic two-site-plus-witness failure-simulation harness drives the real engines — site loss, inter-site partition, witness lost or one-sided, monitor failure, both-sites-down, NFS reachable one-side-only, partition restored, and a Command restart mid-incident — and asserts the safety invariants explicitly: no split-brain (two sites are never both authorized to start the same VM), no dual-writer, and correct fail-closed holds. The HCI enable orchestration, the erasure-coded-pool opt-out, the NFS active-witness arbitration, the corroborated site-loss gate, and the Command-independent quorum-arbitrated recovery are all covered against mock backends with no live cluster.
• Validated in a single-site lab. The configuration, readiness, wizard, and the fail-closed gates behave correctly on real hosts in one location — including, correctly, refusing to activate because one location is not a genuine stretched cluster.
• Pending: two-datacenter hardware. The one thing simulation and a single-site lab cannot prove is live command execution and live network/witness probes against two real, independent datacenters. Until that validation is complete, treat a first activation on real two-site hardware as a carefully monitored change, not a routine one.

It has graduated out of the experimental preview gate: the endpoints and the UI are available without setting a flag. Graduation did not relax any safety check — the fail-closed readiness engine still refuses to enable on any cluster that is not a genuine two-site topology with a witness. The honest capability note above is surfaced directly in the Stretched Cluster UI so the pending hardware validation is never out of sight.