valis / Running a node
valis First Boot and Operator Moves
This is the operator's guide to a node's first minutes of life: what happens when a freshly-provisioned host boots for the first time, the one host-specific file the deploy recipe must author, and the ordered moves an operator makes right after genesis to turn a bare node into a live one.
It sits beside the two other deployment documents. The host-deployment contract states what a host must provide and the fail-closed order a node comes up in; the operations runbook is the standing procedure for installing, observing, renewing, and restarting a node. This document is narrower: the genesis event and the operator's first moves on a node that has never run before.
Audience. The operator standing up a deployed unit, not the developer building valis from a REPL. For the inner development loop see DEVELOPMENT.org.
1. Genesis: what first boot does
A first boot has no predecessor to inherit from. valis calls this condense from genesis: the resident reassembles its durable namespace from an empty starting point rather than from a saved head, and mints the one piece of state that cannot be inherited: the owner's sovereign identity.
The sequence, once the Type=notify unit starts the fulcrum=+=valis pair:
- The site facts a boot requires are demanded first. Before anything condenses, the
resident asks the environment seam for every fact it declares a boot cannot proceed
without, and refuses to start when one is missing. Today that is
VALIS_ACME_STORE_PATH, the root of the ACME custody store, and it is required on every boot including a genesis one. I made an unnamed store root refuse the boot rather than reach for a library default, because that root is where the ACME account key is minted: a boot rooted in a directory nobody chose registers a fresh account against the CA and spends registration budget that cannot be given back. A certificate written to the wrong place can be moved; the spent registration cannot be unspent. A boot that came up satisfied every one of them, because a missing fact stops it. The count on the line tells you how many that was. - The namespace is empty. There is no store head, no owner keyfile, no ACME custody;
the
StateDirectory(/var/lib/valis) has just been created and chowned to the run account by systemd. Condense begins from the default manifest at generation 0. The resident mints the owner Ed25519 seed. This is the only genesis-specific step, and valis does it itself: no sibling, no external key service. The resident's own custody store generates a fresh Ed25519 key pair and writes the 32-byte secret scalar to
<state-dir>/valis/keyfilewith mode0600. From the public half it derives the owner's did:key (the node's sovereign identity), which is therefore stable across every restart from the same keyfile. (mercer holds the transport cipher and supplies an X25519 derivation helper for sealed sessions, but the owner seed is minted and custodied by valis.)Because the seed is written once and never re-minted, losing this keyfile is losing the node's identity. Backing it up is the operator's first real move (below).
- The node reaches its healthy genesis state: serving
:53, with:443cert-gated dark. fulcrum's inherited descriptors carry the public DNS sockets, so the resident answers authoritative DNS on:53without binding any privileged port itself. The public HTTPS edge on:443stays deliberately dark at genesis, and two independent facts hold it there: the launcher config names no:edge-port, so fulcrum binds no TLS socket and the firewall accepts none, and the environment seam names noVALIS_EDGE_DOMAIN, so there is nothing to obtain a certificate for. Both have to be supplied before that port can serve, and fail-closed is the right posture for a node that has neither. A dark:443at genesis is healthy, not a fault. Which of the two states the node is in is reported in the supervisor's status field rather than in the journal, and the runbook's status section says which surface carries what.
2. The launcher config: /etc/fulcrum/launcher.conf.lisp
fulcrum (the privileged parent) reads one host-specific configuration file before it provisions the namespace and launches valis. The deploy recipe authors it per host; the delivery tarball deliberately does not carry it, because several of its values (the sklookup object path, the public interface, the address) are specific to the machine.
The file is a ubiquitous store at /etc/fulcrum/launcher.conf.lisp, root-owned and
read at bring-up. Every key below is a value the recipe sets; the required ones fail the
launch closed if absent (a wrong guess for a privileged executable path or a public
address is more dangerous than a clear error).
| Key | Required? | Meaning |
|---|---|---|
:netns-name |
required | the network namespace fulcrum creates for the unit (e.g. valis) |
:unit-ip |
required | the public address the unit answers on, inside the netns |
:unit-cidr |
default 32 |
the prefix length paired with :unit-ip |
:public-interface |
required | the uplink NIC that carries the public address |
:mode |
default :macvlan |
how the unit's link relates to the uplink: :dedicated moves the whole NIC into the netns; :macvlan / :ipvlan give it a child interface on a shared NIC |
:public-gateway |
required | the in-netns default-route gateway, the uplink's own gateway, not the host's default |
:designated-port |
required | the served port fulcrum designates (e.g. 53) |
:catchall-object-path |
required | absolute path to the compiled sk_lookup steer object on this host |
:valis-command |
required | the absolute unit invocation, as a list of strings (e.g. ("/opt/valis/bin/valis")); PATH does not survive the privilege drop, so it must be absolute |
:control-socket-path |
default /run/fulcrum/<netns>.sock |
the AF_LOCAL socket the unit dials for the steered handoff (systemd's RuntimeDirectory wins when set) |
:valis-state-dir |
default /var/lib/fulcrum/<netns>/ |
the unit's durable state home, where valis mints and persists the owner keyfile (systemd's StateDirectory wins when set) |
:owner-port |
default 564 |
the owner's authenticated 9P management port (the registered 9P/styx port) |
:open-owner-port |
default nil (closed) |
flip to true to open the owner management port; shipped closed, so the firewall drops it and no owner socket is bound until an operator deliberately opens it |
:edge-port |
default nil (dark) |
the public TLS port fulcrum binds inside the netns and hands down as an inherited descriptor; unset, no edge socket is bound and the firewall renders no accept for one, so :443 is dark and dropped whatever the environment seam says. Naming a port is itself the act of opening it, which is why there is no companion open flag |
:valis-uid |
optional | numeric run-uid override; without it the valis-user name (default valis) resolves via getpwnam |
:valis-gid |
optional | numeric run-gid override; without it the valis-group name (default valis) resolves via getgrnam |
For the reference deployment (valis on a NIC of its own) the recipe sets :mode
:dedicated, names the dedicated :public-interface and its :public-gateway, and points
:catchall-object-path at the steer object staged from the tarball. An illustrative
authoring, run once at provisioning time, with exemplar values throughout; a real host
carries its own:
(ubiquitous:with-local-storage (#p"/etc/fulcrum/launcher.conf.lisp")
(setf (ubiquitous:value :netns-name) "valis"
(ubiquitous:value :unit-ip) "203.0.113.10"
(ubiquitous:value :public-interface) "ens7"
(ubiquitous:value :mode) :dedicated
(ubiquitous:value :public-gateway) "203.0.113.1"
(ubiquitous:value :designated-port) 53
(ubiquitous:value :catchall-object-path) "/opt/valis/lib/sk-lookup-catchall.bpf.o"
(ubiquitous:value :valis-command) (list "/opt/valis/bin/valis")))
This authoring names no :edge-port, so the node it stands up serves :53 and nothing
on :443: fulcrum binds no TLS socket, and the firewall renders no accept for one.
A node that is to serve the public HTTPS edge carries one more key in the same file:
(ubiquitous:with-local-storage (#p"/etc/fulcrum/launcher.conf.lisp")
(setf ;; ... the keys above, unchanged ...
(ubiquitous:value :edge-port) 443))
That key changes three things at bring-up: fulcrum binds the public TLS socket inside the
netns while it is still privileged, the firewall admits that port, and the resident
receives the socket as --edge-tcp-fd rather than binding anything itself. The served
domain still has to be named in the environment seam, and neither half alone opens the
port.
The run account defaults to a dedicated valis:valis system user; provision that account
(or set :valis-uid=/:valis-gid= to run under an existing id) before first start. The
unit must not run as root: a zero uid or gid is refused fail-closed.
3. The backup CLI: the go-public gate
Before a node is exposed to a public address, its owner-seed backup must be proven. The
valis binary carries the two verbs; each reads its passphrase interactively (never as
a command-line flag, which would land in shell history and /proc):
# Seal the backup-critical set into one encrypted, host-agnostic artifact.
valis --backup --out /path/to/valis-backup.sealed # prompts: Backup passphrase:
# Restore it into a PRISTINE state directory (a scratch dir for the proof).
valis --restore --in /path/to/valis-backup.sealed --state-dir /tmp/restore-proof
--backup seals the backup-critical set (the owner seed, the store head, the store
blocks the head reaches, and the ACME tree) into a single artifact under a memory-hard
(argon2id) key derivation and authenticated (ChaChaPoly) encryption. --restore opens it
into a state directory that must be pristine: a directory already holding valis state is
refused up front, before any write.
Prove the restore before you expose the node. Restore the artifact into a scratch state directory and confirm the three properties a good restore demonstrates: the owner axis resolves, the store generation matches the source, and the TLS keys load. Those properties, and why a silently-wrong restore is treated as worse than an obvious failure, are set out in backup-and-observability.org. Store the artifact and its passphrase separately, and treat both as the master secret they are.
4. Moving a zone onto a not-yet-serving node
A fresh or recovered node holds only the binary: no resident to drive over the loopback
fabric, no owner keyfile to sign a management session. valis has a direct-to-operator-state
path for exactly this: two zone routes that write the operator-state database directly,
reading no keyfile and no management endpoint. The database connection comes from
VALIS_PG_DSN in the environment, never from the command line.
⚠ The two routes are zone load and zone secondary --offline, and the flag is what
selects the route. load is always direct and takes no such flag. secondary is not:
by default it drives a running resident over the fabric, like every other owner-keyed
verb, and --offline is what sends it straight to operator state instead. Run it without
the flag on a node that is not serving yet and it refuses, telling you to add --offline.
(Once a node is serving, the owner-keyed zone import over the loopback fabric is the
standing path, described in the runbook. The routes here are for the node that is not
answering yet.)
4.1. Loading the first zone
valis zone load --origin example.com --file /path/to/example.com.zone
zone load parses a BIND master file and writes it straight into operator state, printing
the origin and the serial it landed at. It is fail-closed: a serial regression, a broken
master file, or an unmigrated schema is refused with a clear message and a non-zero exit,
writing nothing. --migrate is available for the case where the operator-state schema must
be brought forward first.
4.2. Enrolling an AXFR secondary
valis zone secondary --offline --origin example.com --peer 198.51.100.2
zone secondary --offline is the transfer-allowlist sibling of zone load. It mints a TSIG key
into mercer's durable custody, records the (zone, peer, key-name) allowlist row and the
also-notify recipient in operator state, and prints the key name, the algorithm, the
secret, and a ready-to-paste BIND snippet for the secondary. Optional flags: --notify
<ref> (defaults to the peer), --key-name <name> (defaults to a name derived from the
origin), and --master-ip <address> (overrides the primary address the printed
masters{} clause names).
The secret is printed for the operator to install on the secondary and is never logged;
the minted key is persisted to a 0600 file under the resident's state root, where the
resident's read-through key-loader finds it on the secondary's next signed transfer (no
resident restart or reload). For how the transfer exchange is shaped across the repos, and
which repo authorizes and which emits, see the constellation's
zone-transfer section.
5. First moves on your new node
The ordered steps an operator takes right after a genesis boot, to carry a bare node to live:
Confirm the node is serving
:53. Read the boot line (journalctl -u valis.service) foradopting inherited :53 descriptors, then query the node over loopback:dig @127.0.0.1 -p 53 . SOA +shortA dark
:443at this point is healthy: the edge stays cert-gated until you name a served domain inVALIS_EDGE_DOMAIN.Mint and prove the owner-seed backup. The seed at
<state-dir>/valis/keyfileis irreplaceable. Seal it now, and prove the restore into a scratch directory before the node is ever exposed to a public address:valis --backup --out /secure/valis-backup.sealed valis --restore --in /secure/valis-backup.sealed --state-dir /tmp/restore-proofStore the artifact and its passphrase separately. This restore-proven backup is the go-public gate: do not cut the node over to a public address until it passes.
Bring the node's first domain up. On a node that is already serving, one verb mints the apex records, enrols the secondary, and prints the block you paste at the registrar:
valis zone create --origin example.com --peer 198.51.100.2For a node that is not answering yet, or when you are carrying an existing zone across, write it straight to operator state instead:
valis zone load --origin example.com --file /path/to/example.com.zoneEither way, confirm the node answers the zone over loopback (
dig @127.0.0.1 -p 53 example.com SOA) before cutting the public address over.Enroll a secondary, if
zone createdid not. A downstream nameserver that will pull the zone by AXFR needs the printed secret and BIND snippet installed on it:valis zone secondary --origin example.com --peer 198.51.100.2No
--offlinehere: you confirmed at step 1 that the node is serving, so this drives the running resident. Add--offlineonly if you took thezone loadbranch at step 3 because the node is not answering yet.Watch the delegation settle. After the registrar block is in place, ask the parent zone's nameservers whether they have taken it. This needs no resident, no keyfile and no database, so run it from wherever you are:
valis zone delegation --origin example.com --waitIt exits non-zero until the delegation has settled, which makes it usable as a gate in a script rather than something you squint at.
Once the backup is restore-proven and the zone answers over loopback, the node is ready to
take its public address. From here the standing operations runbook governs: observing
the boot status, opening :443 by naming VALIS_EDGE_DOMAIN and driving valis obtain,
the certificate lifecycle, publishing to and updating a running node, and restart and
teardown.