valis Operations Runbook

The shipped unit expresses an FHS layout the launcher threads into valis's XDG rooting. Provision these before first start:

StateDirectory holds the irreplaceable durable state. systemd creates and chowns it on first boot; the parent's CAP_CHOWN=/=CAP_FOWNER make the owner-seed write under it succeed. Everything backup-critical lives here, and the valis --backup verb seals the backup-critical subset out of it, see 9.

The binary must be the serving-capable delivery core, not a plain :valis core: the plain core lacks the additive DNS-serving stack and fails closed at boot with c3po's serving codec … not available. Stage it from the versioned delivery tarball that make dist produces (valis-serving-<version>.tar.gz); never rebuild the core on the host. See the contract's serving-capable resident binary section.

The units ship under deploy/systemd/. Install them onto the host:

# The supervisor unit.
install -m 0644 deploy/systemd/valis.service /etc/systemd/system/valis.service

# The bootstrap environment seed. Copy the template, fill it, lock it down.
install -d -m 0750 -o root -g valis /etc/valis
install -m 0640 -o root -g valis deploy/systemd/valis.env.example /etc/valis/valis.env
# ... edit /etc/valis/valis.env (next section) ...

systemctl daemon-reload

The ExecStart is /opt/valis/bin/fulcrum-resident, the fulcrum delivery binary that owns the whole bring-up. Stage that binary and the run account per the contract's provisioning surface before enabling the unit.

/etc/valis/valis.env is the bootstrap seed: the provisioned config that stands the node up before the durable ubiquitous config-of-record (under /var/lib/valis/config) exists. It is re-authored by provisioning and is 0640 root:valis. It is never the writable config-of-record and never enters the tracked repo.

The database DSN names a local UNIX-domain socket, so the resident reaches Postgres by filesystem path under peer authentication rather than over the network: no password travels in the environment and no TCP listener need be open. Thread the socket-path DSN down the launch environment. (Moving the database onto a dedicated address inside the netns is deferred hardening; the host is the root of trust for the resident copy.)

systemctl enable --now valis.service     # enable at boot + start now
systemctl status valis.service           # unit + MainPID (fulcrum) state
journalctl -u valis.service -f           # follow the bring-up + serving log

Readiness is meaningful, and it is reported in two places that are not the same place. The bring-up writes its account to the journal, and the resident separately publishes a one-line summary to the supervisor, which systemctl status shows as Status:. Neither carries the other's text, so a phrase you cannot find in the journal is very likely a status-field phrase, and looking harder will not turn it up.

In the journal (journalctl -u valis.service), the bring-up says what it did:

• site configuration complete: every declared site fact was answered, and the count of facts checked follows on the same line. A boot missing one stops here instead, naming the variable it wanted.
• adopting inherited :53 descriptors: the DNS handoff took; valis is serving authoritative DNS over fulcrum's inherited :53 descriptors, binding nothing privileged itself.
• :443 public edge opening — certificate in custody for <domain>: a cert was found for the configured domain and the edge is coming up.
• no usable certificate in custody for <domain>; :443 public edge cert-gated dark: the domain is named but no cert has been issued for it yet. This is what an operator sees between naming a domain and completing the first obtain, and it is the line that says the gate is the certificate rather than the configuration.
• no VALIS_EDGE_DOMAIN configured; :443 public edge stays dark: no served domain named, so :443 is intentionally dark by configuration.

In the supervisor's status field (systemctl status valis.service, the Status: line), the resident publishes its serving state and refreshes it on every health-loop pass. It reads <serving state>; cert <days>; store gen <n>, where the serving state is one of:

• serving :53 + :443: a usable certificate was in custody at boot; the public HTTPS edge is open.
• serving :53; :443 dark (no cert): a healthy genesis boot: :53 is up (which is exactly what dns-01 needs to obtain the first certificate), and :443 is cert-gated dark until a cert is issued. This is not an error and does not withhold readiness.

A silently dark port would look like a fault; between the two surfaces you get why :443 is or is not open, so you never have to guess.

Nothing in the resident gauges descriptor headroom: there is no count in the journal, no field in the status line, and the liveness gate does not read it. A node that has exhausted its descriptors keeps reporting itself as serving and the supervisor is given no reason to restart it, so headroom is a reading the operator takes from outside the process, on the host:

pid=$(pgrep -x valis)                      # the resident is fulcrum's child, not MainPID
ls /proc/$pid/fd | wc -l                   # descriptors held right now
grep 'Max open files' /proc/$pid/limits    # the ceiling this process actually has

Match the process name to whatever the launcher config's :valis-command stages. Take the reading as root or as the run account: another account's /proc/<pid>/fd is not yours to list. The unit sets no descriptor limit of its own, so the ceiling is the service manager's default on this host: read it rather than assume it.

One reading tells you little. Take two, some hours apart under comparable load: a busy node's count moves in both directions, while a count that only ever climbs is a leak, and the moment to act on it is while the number is still nowhere near the ceiling. Put that pair of readings into whatever monitoring you run against the node. Nothing on the node will raise it for you, and the failure it precedes is one every other instrument reports as healthy. Why that is, and what the liveness gate does and does not ask, is set out in backup-and-observability.org.

The :443 edge is obtained and renewed over the ACME/dns-01 spine (custody and the ACME lifecycle are mercer's; valis is the edge that consumes the cert):

1. Genesis (no cert). Boot with VALIS_EDGE_DOMAIN set but no cert yet: :53 comes up, :443 is cert-gated dark. Serving :53 is the precondition for dns-01: the authoritative answer proves domain control to the CA.
2. Issuance. Once a certificate is issued and lands in the custody store (VALIS_ACME_STORE_PATH), a boot with the cert in hand opens :443. dns-01 is the only challenge a deployed node can answer, and there is no fallback. Nothing in the shipped arrangement serves :80: no HTTP socket is opened for the unit and the firewall renders no accept for that port, so an http-01 order has nothing to validate against. That is why serving :53 is a precondition of issuance rather than a convenience.
3. Renewal (hot-swap, no restart). On a successful renewal the live :443 cert is reloaded in place through the shared credential cell: the edge swaps the new leaf without dropping the listener or restarting the unit.
4. Staging vs production. Default is the ACME staging CA. Set VALIS_ACME_PRODUCTION truthy only when you want a real, publicly-trusted certificate: production carries a strict rate ceiling, so exhaust staging first.

Driving the first obtain by hand. The obtain verb runs the order on the process that serves :53, which is what makes the dns-01 challenge answerable at all:

valis obtain --domain <fqdn> --contact <email> [--profile <name>] [--directory-url <url>]

Both --domain and --contact are required; --domain repeats, or takes a comma-set. ⚠ Take a comma-set only for names that genuinely belong on one credential. Certificate Transparency publishes the name set permanently, so packing several of your domains into one certificate publishes that you hold them all, and nothing takes it back. One certificate per registrable domain.

Like the other verbs it is owner-keyed over the loopback fabric (--keyfile / VALIS_KEYFILE, --endpoint / VALIS_MGMT_ENDPOINT), and it stays on staging unless VALIS_ACME_PRODUCTION is set.

A first-obtain can also be driven remotely by the authenticated owner over the /acme/ctl 9P door: obtain <domain> <contact> starts the dns-01 order on a background thread (single-flight: a second obtain while one runs is refused, since every attempt burns CA rate-limit budget), and status reports idle/running plus the last completed outcome. The door threads no CA selection of its own; the staging-unless-opted-in default above applies unchanged.

valis is authoritative for the SOA zones it serves on :53. The zone data itself is operator state: a node comes up serving whatever zones are in its operator-state store, and comes up with none on a genesis boot. Loading and maintaining that data is the valis zone verb family: a headless, owner-authenticated client the deploy recipe drives non-interactively, and the operator drives by hand.

How it authenticates. Each verb reaches the owner-gated management axis over the resident's loopback fabric, presenting the owner key as its transport identity, the same key custody holds in the StateDirectory keyfile. The key is supplied as a path, resolved --keyfile PATH > VALIS_KEYFILE > the StateDirectory keyfile; the key bytes never cross the command line. The verb runs on the same host as the resident and finds its loopback endpoint automatically (the resident publishes the bound endpoint to a runtime file), or you may point it explicitly with --endpoint HOST:PORT / VALIS_MGMT_ENDPOINT. Because only the holder of the owner key is admitted, an absent or wrong key fails closed: the management axis never opens to an anonymous caller.

The verbs.

valis zone create --origin <fqdn> [--peer <addr>] [--secondary-ns <name>]
valis zone record add --origin <fqdn> --owner <name> --ttl <s> --type <type> --rdata <value>
valis zone record delete --origin <fqdn> --owner <name> --type <type> --rdata <value>
valis zone import --origin <fqdn> --file <master-file>
valis zone export --origin <fqdn> [--out <file>] [--full]
valis zone delete --origin <fqdn>
valis zone delegation --origin <fqdn> [--wait] [--timeout <s>] [--interval <s>]

# Straight to operator state, for a node that is not serving yet:
valis zone load --origin <fqdn> --file <master-file> [--migrate]
valis zone secondary --offline --origin <fqdn> --peer <addr> [--notify <ref>] [--key-name <name>]

• create brings a whole domain up in one move: it mints the apex records from the domain template, enrols the secondary when you name a peer, and prints the block you paste at the registrar. Reach for this before reaching for import on a new domain.
• record add and record delete publish or retract one typed record and leave the rest of the zone alone. This is the editing path. An --owner ending in a dot is absolute; without one it is relative to the zone. The master-file text format is a boundary format for the secondary, never the format you edit in.
• delegation asks the parent zone's nameservers whether the delegation has settled, and exits non-zero until it has. It needs no resident, no keyfile and no database, so you can run it from anywhere while you wait on a registrar. --wait polls.
• import loads an RFC-1035 zone master file as the named zone, committing it atomically; a malformed master is refused with a non-zero exit and changes nothing, so a bad file never half-lands. Re-importing an origin replaces its zone (advance the SOA serial).
• export writes the zone's master text (to --out, or standard output). It defaults to the durable, re-importable master: the published records, the faithful thing to archive or re-load. Pass --full for the whole serving set, which also includes any transient turn-up records (e.g. ACME challenge records) present at that moment.
• delete removes the whole zone (apex-SOA removal is whole-zone removal).
• load and secondary –offline are the two routes that do not need a running resident. They write operator state directly, reading no keyfile and no management endpoint, with the connection coming from VALIS_PG_DSN and never from the command line. Reach for them on a node that is fresh or recovered and not answering yet. ⚠ --offline is what selects that route. secondary without it drives a running resident over the fabric, exactly like the owner-keyed verbs above, so on a node that is not serving yet it refuses and tells you to add the flag. load takes no such flag and is always direct. Either way secondary mints the TSIG key, records the allowlist row, and prints the BIND snippet to paste on the secondary. Both are walked through in first boot and operator moves.

A committed change is picked up by the running resident without a restart: the serving side re-reads the store, so the node begins answering the new data authoritatively.

At go-live. After a fresh node is up and healthy (serving :53, restore-proven per the go-public gate above) the deploy recipe imports the node's own zone before the address is cut over: whichever domain that node serves, with its apex SOA, its apex NS, and the ns1 address record. The recipe runs valis zone import as the account that can read the 0600 keyfile, then confirms the node answers the zone over loopback (dig against :53) and that valis zone export round-trips before declaring the node live. Zone master files are operator data staged onto the host, not part of the delivery binary.

Neither of these takes the node down, and neither wants you editing files on the host.

Publishing a page. publish places a publication into /pub on a running resident, owner-keyed over the same loopback fabric the zone verbs use. It is an ordinary operator act at any time, not a deploy-time one:

valis publish --slug <name> --file <path> --content-type <type>

All three are required. A slug that already exists is revised, not refused.

Updating the binary. apply takes a delivery archive that is already on the host, puts it in place, restarts the serving unit, and confirms the node still answers:

valis apply --artifact <path> --confirm-server <addr> --confirm-zone <fqdn>

It transfers nothing, so move the archive onto the host first. The two confirmation flags also read VALIS_APPLY_CONFIRM_SERVER and VALIS_EDGE_DOMAIN, and there is no loopback default: name them or the apply refuses before anything moves. That refusal is the point. Exit 2 means it refused and nothing moved; exit 3 means it moved and could not confirm, which is the case that wants you looking. --dry-run walks it without touching the host.

By default apply places the serving binary alone. --with-resident and --with-steer are the explicit ask for the wider set, so the privileged parts of the delivery never move because you forgot they were in the archive.

The backup-critical set is irreplaceable: the owner Ed25519 seed (0600), the store head, the store blocks the head's root tree reaches, and the ACME account and leaf keys. Losing the owner seed is losing the node's sovereign identity; there is no re-mint.

The valis binary carries the backup and restore verbs. Each seals or opens a single encrypted, self-contained artifact, not a copy of the whole StateDirectory. The passphrase is read interactively (or piped on standard input for an unattended run); it is never a command-line flag, so it never lands in shell history or /proc.

• Back up. Seal the backup-critical set into one artifact:

  valis --backup --out /path/to/valis-backup.sealed
  # prompts: Backup passphrase:
  

  The artifact is small and host-agnostic (it carries no absolute paths or machine identifiers). A memory-hard key derivation (argon2id) turns the passphrase into the key, and authenticated encryption (ChaChaPoly) seals it: a tampered artifact is rejected rather than silently restored. Treat it as you would a private-key store, and store the artifact and the passphrase separately. A backup can also be driven remotely by the authenticated owner over the /backup/ctl 9P door (backup <passphrase> [out-path]; status reports the last outcome). The passphrase rides one ctl line over the sealed owner session (never cleartext on the wire, never shell history) and the artifact lands on the node's own disk (default: a timestamped bundle under <data-root>/valis/backups/). A restore is deliberately not served over 9P: it requires the unit down and a pristine state directory, so it stays valis --restore on the host.

• Restore. Open the artifact into a pristine state directory:

  valis --restore --in /path/to/valis-backup.sealed --state-dir /var/lib/valis
  # prompts: Restore passphrase:
  

  A non-pristine target (one that already holds valis state) is refused up front, before any write, with exit code 2; a wrong passphrase or corrupt artifact fails cleanly with a single message and exit code 1. The normal resident boot then condenses the durable namespace from the restored head: restore is the first-deploy path run from a saved head instead of from genesis, and a faithful restore reproduces the origin's durable identity byte-for-byte. Deploy, evacuate, and restore are one condense mechanism, see the contract's deploy/evacuate/restore section.

The restore-proven backup is the go-public gate, and nothing else enforces it. Do not cut a node over to a public address until a restore has been proven: restore the artifact into a scratch state directory and confirm the three properties a good restore demonstrates (the owner axis resolves, the store generation matches the source, the TLS keys load). Those properties, and why a silently-wrong restore is treated as worse than an obvious failure, are set out in backup-and-observability.org.

• Restart is just re-running bring-up: it is idempotent. The single-writer instance fence makes a re-claim of the current generation a no-op, so recovery is safe to repeat.

  systemctl restart valis.service
  

• Stop drives an ordered teardown. On SIGTERM the resident unwinds in reverse bring-up order (long-lived active modules drain first, then edge, then fabric, then listener); fulcrum (the MainPID) removes the sk_lookup link and tears the netns down. KillMode=mixed lets fulcrum drive that ordered pair-teardown rather than systemd SIGTERMing both at once.

  systemctl stop valis.service
  

• A superseded writer fails closed. Two instances sharing one operator-state database coordinate only through the single-row write-fence; a superseded writer is fenced out (the fenced-out condition) and writes nothing. This is the split-brain stop that makes a cross-location handover safe. Expect the older instance to fail closed once a successor claims a higher generation.

The default run account is a dedicated static valis:valis. A legacy operator running the resident out of their own account is supported via a drop-in: install legacy-operator.conf.example as /etc/systemd/system/valis.service.d/legacy-operator.conf and set the <operator> / <operator-home> placeholders:

install -d -m 0755 /etc/systemd/system/valis.service.d
install -m 0644 deploy/systemd/valis.service.d/legacy-operator.conf.example \
        /etc/systemd/system/valis.service.d/legacy-operator.conf
# ... edit <operator>/<operator-home>, then ...
systemctl daemon-reload

Overriding User=/=Group and pointing XDG_DATA_HOME (and HOME) at the operator's home keeps valis rooting durable state there; on the first boot after such a move, the resident relocates pre-XDG ~/.valis state cleanly. To fall all the way back to ~/.local/share, clear the inherited StateDirectory (an empty StateDirectory=).

For the architectural why behind each of these, the privilege split, the descriptor handoff, the sandbox directives that ship vs. those deliberately omitted (MemoryDenyWriteExecute, LockPersonality), read the annotated valis.service unit and the host-deployment contract.