valis / Running a node

valis Operations Runbook

This is the hands-on companion to the host-deployment contract. The contract states what a host must provide and the fail-closed order a node comes up in; this runbook states what the operator does to install, run, observe, renew, back up, and restart a deployed node. Where the contract is the invariant, the runbook is the procedure.

Audience. The operator running a deployed unit, not the developer building valis from a REPL. For the inner development loop see DEVELOPMENT.org; for building your own system on the substrate see BUILDING.org.

The shape of a running node. One Type=notify systemd unit supervises the fulcrum=+=valis process pair. The privileged fulcrum parent creates the network namespace, binds the public sockets its launch configuration names (:53 when :designated-ports includes 53, the TLS edge when the configuration names an :edge-port, and the owner port when it sets :open-owner-port), loads the sk_lookup director, and forks the unprivileged valis resident after a setpriv drop. The director funnels every inbound TCP connection in the node's network namespace, loopback included, into valis. valis adopts the inherited descriptors, brings its fabric and edges up fail-closed, and emits READY=1 once every bring-up step has returned. Readiness attests that the fabric, the edge, and the active modules came up without error; it does not attest that anything has been served, and what keeps the unit alive afterwards is a liveness check on those threads rather than a check that traffic is moving. The unit is the supervisor of last resort: it restarts the whole image if the image dies; the active-module seam inside the image supervises the modules.

1. Host filesystem layout

The shipped unit expresses an FHS layout the launcher threads into valis's XDG rooting. Provision these before first start:

Path Role Ownership
/opt/valis/bin/ the staged delivery binary (fulcrum-resident), world-traversable root, o+x chain
/etc/valis/ the bootstrap EnvironmentFile seed only (read-only to service) root:valis 0750
/var/lib/valis/ StateDirectory: durable irreplaceable state (owner seed, store, ACME custody, the node's configuration under valis/config/) valis:valis 0700
/run/valis/ RuntimeDirectory: the AF_LOCAL control socket + pidfiles auto, ephemeral

StateDirectory holds the irreplaceable durable state. systemd creates and chowns it on first boot; the parent's CAP_CHOWN=/=CAP_FOWNER make the owner-seed write under it succeed. Everything backup-critical lives here, and the valis --backup verb seals the backup-critical subset out of it, see 14.

The binary must be the serving-capable delivery core, not a plain :valis core: the plain core lacks the additive DNS-serving stack and fails closed at boot with c3po's serving codec … not available. Stage it from the versioned delivery tarball that make dist produces (valis-serving-<version>.tar.gz); never rebuild the core on the host. See the contract's serving-capable resident binary section.

2. Installing the unit

The units ship under deploy/systemd/. Install them onto the host:

# The supervisor unit.
install -m 0644 deploy/systemd/valis.service /etc/systemd/system/valis.service

# The bootstrap environment seed. Copy the template, fill it, lock it down.
install -d -m 0750 -o root -g valis /etc/valis
install -m 0640 -o root -g valis deploy/systemd/valis.env.example /etc/valis/valis.env
# ... edit /etc/valis/valis.env (next section) ...

systemctl daemon-reload

The ExecStart is /opt/valis/bin/fulcrum-resident, the fulcrum delivery binary that owns the whole bring-up. Stage that binary and the run account per the contract's provisioning surface before enabling the unit.

3. The environment seam: /etc/valis/valis.env

/etc/valis/valis.env is the node's genesis seed. The node reads each value in it once, at the first boot that considers that value's key, and keeps the result in its configuration store under the data root: <StateDirectory>/valis/config/, which is /var/lib/valis/valis/config/ under the shipped unit. From then on the store is the node's configuration. Change a setting with valis config or through /config over the owner session (see changing a setting). Editing valis.env afterwards has no effect on a key the node has already considered, and the boot log names each such key as superseded by the store, by name and never by value. A key that a newer release adds is seeded from this file once, at that release's first boot.

The file is re-authored by provisioning, is 0640 root:valis, and never enters the tracked repo. Its values travel nowhere: the node's configuration directory is what a backup carries (see backup and restore).

valis apply checks the site facts a boot needs against the node's own configuration first and this file second, the order the node reads them in. Without --confirm-zone it confirms the node's own edge.domain, unless you have set one of your own (see below). When apply cannot read the node's configuration, it checks this file alone and says so; see letting apply read the node's settings.

Each variable below seeds the configuration key beside it:

Variable Configuration key Meaning / effect when unset
VALIS_PG_DSN operator-state.dsn operator-state Postgres URI over the local UNIX-domain socket under peer authentication, e.g. postgresql://valis@/valis_state?host=/var/run/postgresql. No password, no TCP listener: the kernel proves the run account's uid and Postgres maps it to the role. The configuration store refuses an address that carries a password, and says why, so no secret reaches it or a backup; use the socket form. An address with a password stays in the environment, is read from there at every boot, and the boot logs that the key is held in the environment because it carries a secret.
VALIS_EDGE_DOMAIN edge.domain the DNS name the :443 edge serves a certificate for. Unset ⇒ :443 stays dark (fail-closed: no domain named). Naming it is necessary and not sufficient: the launcher config must also name an :edge-port, or no socket is bound for the edge and the firewall accepts none.
VALIS_ACME_PRODUCTION acme.production truthy ⇒ the real Let's Encrypt production CA (strict rate ceiling). Unset/empty ⇒ staging (the fail-safe default; keeps dev/CI off production).
VALIS_ACME_DIRECTORY_URL acme.directory-url override the ACME directory (a Pebble/CI or alternate staging endpoint). Honored only when the production opt-in is not set: production always wins.
VALIS_ACME_STORE_PATH acme.store-path the custody store the :443 edge loads its cert from and the renewal manager writes into. In the shipped unit, under StateDirectory, e.g. /var/lib/valis/acme. Unset ⇒ the resident refuses to boot, on every boot including a genesis one: an unchosen store root would mint a fresh ACME account against the CA and spend registration budget nothing gives back, so it is required rather than defaulted. The shipped valis.env.example already carries a value. A first boot without it settles the key empty, so editing valis.env afterwards has no effect: once a boot has refused for this, set it with the node stopped, as the run account: sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config set acme.store-path /var/lib/valis/acme.
VALIS_ACME_ACCOUNT_CONTACT acme.account-contact the operator's ACME account contact, the address this node's ACME account is registered to. Unset/blank ⇒ automatic renewal refuses rather than ordering, and says so on the boot line: a contact is an identity, and one invented for you would register the account to an address you never chose. A first order can carry a contact typed at valis obtain --contact; a renewal falls due months later with nobody at the keyboard, and this is where it reads one.
VALIS_MODULE_SOURCE_ROOT module.source-root where the bytes of every module this node admits are kept, one directory per module identity. In the shipped unit, under StateDirectory, e.g. /var/lib/valis/modules. Unset means the resident refuses to boot, and with it set the resident creates the directory at boot. The shipped valis.env.example already carries a value. A first boot without it settles the key empty, so editing valis.env afterwards has no effect: once a boot has refused for this, set it with the node stopped, as the run account: sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config set module.source-root /var/lib/valis/modules. See the module source root.
VALIS_EDGE_DOMAINS edge.domains further names declared primary for this node, comma or space separated. They add to edge.domain and never replace it, and the edge still needs that base name to open :443. Unset means the base name alone.
VALIS_MAIL_LOCAL_DOMAINS mail.local-domains the mail domains this node delivers locally, comma or space separated. The value is carried in the configuration, but no part of the running node reads it yet.
VALIS_MAIL_SECONDARY_FOR mail.secondary-for the domains this node is backup MX for, as domain=primary-host tokens. A list with a member naming no primary host is refused whole. Unset means the node is backup MX for none.
VALIS_MAIL_ACCEPT_LOCALPARTS mail.accept-localparts the local parts accepted at RCPT, comma or space separated. Unset leaves the built-in accept list in force; it never widens acceptance to a catch-all.
VALIS_MAIL_RESOLVE_DOT_HOST mail.resolve-dot-host the DNS-over-TLS upstream that outbound mail resolves recipient MX records through. Unset means every relay is deferred, because there is no smarthost to fall back to.
VALIS_MAIL_RESOLVE_DOT_ADN mail.resolve-dot-adn the name that upstream's certificate is verified against, which is also the SNI sent to it.
VALIS_MAIL_RESOLVE_CA_FILE mail.resolve-ca-file the CA bundle that verifies that upstream. Unset is an empty trust store and not a bypass: every peer is rejected, so mail defers until you name one.
VALIS_TRANSFER_MASTER_ADDRESS names.transfer-master-address this node's public authoritative DNS address: the source of outbound NOTIFY, the address a secondary pulls from, and the address a new domain resolves to when you name no other. zone create, and zone secondary without --offline, ask the running node for it over the owner session when no flag names the address. Unset means those verbs refuse rather than pick an address.
VALIS_SECONDARY_NS names.secondary-ns the nameserver name published beside this node's own in every domain's apex NS set. Unset means the apex names this node alone.
VALIS_SECONDARY_TRANSFER_ADDRESS names.secondary-transfer-address the address the secondary pulls zone transfers from. zone create enrols that address when you pass no --peer. Unset means such a create refuses.
VALIS_DEPENDENCY_DIST_URL dist.url the private dist valis dist register registers as the node's dependency source, when --location is not given. Unset means the verb refuses with exit 2.
VALIS_DEPENDENCY_DIST_HOME dist.home where valis dist register records the registration, when --home is not given. Unset means valis/dependency-dist/ under the data home.
VALIS_NET_DIAL_ALLOW net.dial-allow the special-purpose address ranges each client may reach when somebody other than you chose where it connects, as client=range members with one range to a member and a client repeated for several, e.g. delegation-probe=192.168.140.0/22. A range lying wholly inside the floor (the unspecified addresses, multicast and limited broadcast) is refused, because nothing ever reaches the floor. A range with a bit set past its prefix length is refused too. A list with one refused member is refused whole. An allowance stored under looser rules than the node now applies allows nothing, and the node logs config.allowance-refused once, naming net.dial-allow and why. Set it again under the current rules to restore it. Unset means no client is allowed any such range. A running node reads it as it makes each such connection, so a change applies to the next one. valis zone delegation reads it as well, for the client delegation-probe, unless you pass --allow-range, so the example lets that check ask nameservers in a lab's private network. A module you admitted is a client under its ASDF system name: example-client=10.0.0.0/8 lets the module admitted as example-client reach that network, and no other module. The allowance follows the system name, whichever vouched module carries it: a replacement, or another publisher's module admitted under the same name, inherits it. A module whose system name a client name cannot carry (anything but lower-case letters, digits, dots and hyphens, or a first character that is not a letter or a digit) can be allowed nothing, and neither can one named as one of the node's own clients (acme, delegation-probe, dist, secondary-forward, zone-notify).
VALIS_OWNER_PORT_SESSIONS_PER_ORIGIN owner-port.sessions-per-origin how many connections the owner port holds at once from one origin, counted from the moment a connection reaches it, handshake included. Unset means 4. A whole number from 1 to 63. One origin is one IPv4 address or one IPv6 /64, and an IPv6 address carrying an IPv4 one counts as that IPv4 address, so machines behind one public IPv4 address count as one origin. A refused connection is logged at WARN at most once per origin per minute. A running node reads the setting at each connection, so a change applies to the next one.
VALIS_OWNER_PORT_SESSION_CEILING owner-port.session-ceiling how many connections the owner port holds at once from every origin together. Unset means 16. A whole number from 1 to 63, which leaves at least one of the 64 connections every steered port shares to the other ports. A running node reads the setting at each connection, so a change applies to the next one.
VALIS_MODULE_STANDING_GRANT_REVIEW_DAYS module.standing-grant-review-days how many days an active standing grant may go without a review before the owner's grant inventory, /cap/grants, marks it review due, counted from the later of its filing and its latest review. Unset means 90. A whole number from 1 to 3,650. The mark is advisory: nothing stops a module being admitted under a grant that is due. While any active standing grant is due, the node's status line ends by counting them and naming the interval. A running node reads it each time the inventory is read and each time the health loop refreshes the status line, every 15 seconds, so a change applies at the next of those.
VALIS_SLYNK_PORT development.listener-port ⚠ A trap: do not set it on a new node. It names a port for the development listener, which opens only when the node starts: a change takes effect at the next restart. A node has it only if you set it there by hand. Nothing provisions it: the template leaves it out, and a node condensed from genesis, or recondensed onto a new host, comes up without it. On a node whose director is attached, the boot line shows the listener as listening while no connection can reach it, because the director funnels every inbound TCP connection in the node's network namespace, loopback included, into valis's single listening socket. The listener's destination is a unix domain socket; see the listener's destination.

The database DSN names a local UNIX-domain socket, so the resident reaches Postgres by filesystem path under peer authentication rather than over the network: no password travels in the environment or the configuration, and no TCP listener need be open. Put the socket-path DSN in valis.env for the first boot, and change it afterwards with valis config set operator-state.dsn in the form changing a setting gives. (Moving the database onto a dedicated address inside the netns is deferred hardening; the host is the root of trust for the resident copy.)

The tools you run read seven more variables, from the environment you run them in. They are not node configuration and have no configuration key: the node never reads them, and they do not belong in valis.env.

Variable Meaning / effect when unset
VALIS_KEYFILE the owner keyfile the owner-keyed verbs present, when --keyfile is not given. Unset means the keyfile in the data root of whoever runs the verb. A path only: the key bytes never cross the command line. The file must be a regular file, not a link, owned by the account running the verb, private to it, and exactly the 32 octets of a seed; anything else is refused. The key named is the node's own, so the verb accepts only that node as the responder: one that cannot prove it holds the key is refused.
VALIS_MGMT_ENDPOINT the management endpoint, as HOST:PORT, the owner-keyed verbs connect to when --endpoint is not given. Unset means the endpoint file the resident writes. The port changes at every boot, so read it fresh rather than setting this once; see the break-glass route.
VALIS_NODE the pinned node the owner-keyed verbs reach, by its name in the pin store, when --node is not given. Unset means no node is named, and the verbs use the keyfile and the endpoint. Under a node, VALIS_KEYFILE and VALIS_MGMT_ENDPOINT are not read, and a --keyfile or --endpoint flag is refused rather than dropping the pin.
VALIS_KEY the device key presented to a pinned node, when --key is not given. Unset means this machine's device key, the one valis key generate writes under $XDG_CONFIG_HOME/valis/, or under ~/.config/valis/ when XDG_CONFIG_HOME is unset or not an absolute path. A device key is presented only to a pinned node, so VALIS_KEY or --key with no node named is refused.
VALIS_APPLY_CONFIRM_SERVER the server valis apply queries to confirm the node is serving, when --confirm-server is not given. There is no default: with neither, the apply refuses before anything moves. --confirm-zone falls back to edge.domain in the configuration under the invoking account's own data root, or to VALIS_EDGE_DOMAIN in the invoking shell, and then to the node's own edge.domain; when your value and the node's disagree, the apply refuses and names both.
VALIS_APPLY_TARGET the directory valis apply replaces the serving binary in, when --destination is not given. Unset means /opt/valis/bin/.
VALIS_APPLY_UNIT the unit valis apply restarts, when --unit is not given. Unset means valis.service.

One more is read only on the host that builds the binary, while it compiles. No node reads it, and it does not belong in valis.env either.

Variable Meaning / effect when unset
VALIS_BAKE_MANUAL_DIR the staged manual the binary carries, a directory make site site-stage wrote. make dist sets it for you, to the manual it built from the commit it releases, so a fresh node serves the whole manual before anything has been published to it. Unset means the binary carries no manual and a fresh node serves the landing page alone. A directory that is missing, holds no pages, holds a page whose name cannot be a publication slug, or holds no front door stops the build.

4. Changing a setting

valis config reads and changes the node's settings and any module's. Against a running node it goes through the node's owner session, so from your own machine you name the node, as you do for every owner verb (see Reaching a node's owner control plane):

valis --node NODE config list               # every node key: path, origin, type, when it applies, value
valis --node NODE config list SYSTEM        # a module's keys, and what an upgrade kept aside
valis --node NODE config get edge.domain
valis --node NODE config set edge.domain example.org
valis --node NODE config reset edge.domain  # back to the shipped default
valis --node NODE config menu               # walk the node's settings, one at a time
valis --node NODE config interview SYSTEM   # a module's own questions; a blank answer keeps a value
valis --node NODE config set --module SYSTEM KEY VALUE   # a module's key
valis --node NODE config purge SYSTEM       # delete a removed module's kept settings

config get prints one line of five fields separated by tabs: the key, where its value came from (default or operator), its type, when a change to it applies, and the value, which is empty when the key is unset. A value set equal to the default reads back with the origin default. A change answers with when it takes effect, described below, then key KEY and origin ORIGIN. With a key the node has not enrolled, the verb is refused with exit 2 and changes nothing.

From the node's host, the break-glass route. When you cannot use the route from your machine (the owner port closed, the pin or the device key lost, or the node not answering on the owner port), run the verb on the node's host the way you run the other owner-keyed verbs there, inside the node's namespace with its keyfile and endpoint named (see the break-glass route):

v() {
  sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
    /opt/valis/bin/valis config \
    --keyfile /var/lib/valis/valis/keyfile \
    --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)" "$@"
}
v list
v get edge.domain
v set edge.domain example.org

On the host, reading the node's own settings does not need the owner session. config list and config get KEY, run as the run account on the node's data root, read its store directly, running or stopped. The read takes no lock and writes nothing, so it never delays the node:

sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config get edge.domain

I made reads work this way because a running node's owner session lives inside its network namespace, out of reach of the host's, and reading a setting should not need nsenter and root. A module's settings are still read over the owner session, since only a node that has loaded the module can list them.

Against a stopped node, run it as the run account on the node's data root, and it changes the node's own settings directly:

sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config set edge.domain example.org

A module's settings change only through the running node, since only a node that has loaded the module can check a value for it. A secret is only ever shown as set or unset. Run against a stopped node as any account but the one owning its data root, the verb is refused with exit 3, as a configuration that could not be changed. Run without --keyfile and --endpoint, it acts on the data root of the account running it, so under sudo alone it reads and writes root's own store and not the node's.

A change answers with when it takes effect:

  • committed-applied: the value is stored and the running node is using it.
  • committed-on-restart: the value is stored and the node takes it at the next restart. ⚠ This is a success, not a fault: some keys, such as the development listener port and the ACME store, only ever take effect at a restart, and every change to a stopped node answers this way.
  • committed-rebind-failed: the value is stored, the running code refused it, and a reason line names the kind of condition. The next restart applies the value.

A purge answers purged or nothing-to-purge, and a purge of a loaded module also answers when its return to the defaults takes effect, as one of the outcomes above. A refusal names what was wrong and changes nothing. The exit status is 0 when the verb read or committed, 2 when the node or the verb refused and nothing changed, which includes a key file that is missing or one the verb will not present, and 3 when the node could not be reached or its configuration could not be changed.

The configuration directory is part of every backup, so a restored node comes back with the settings it had.

4.1. The served set

The ports a node serves are one setting, edge.served-ports, and you read and change them from your own machine like any other. Read the set first:

valis --node NODE config get edge.served-ports

The value is the ports in ascending order, separated by commas: 53,80 on a node that kept the default core. Change it with config set and give the whole set. The value you set replaces the set, so a port you leave out stops being served. To add 8099 to that set:

valis --node NODE config set edge.served-ports 53,80,8099
valis --node NODE config get edge.served-ports

Keep every member you read unless you mean to stop serving it. A set without 53 stops DNS, and an empty set is a choice the node keeps: it serves nothing, and a restart does not bring the default core back. The change answers committed-applied and takes effect without a restart. To put the set back, set it to exactly the value you read first.

⚠ committed-applied says the node stored the set and took it into the running set. It does not say the host agent's firewall admitted the port. Read that from a machine other than the node's host with scripts/owner-port-reach.sh and both controls, as From another machine describes, here with a port the node admits and a port outside its set:

scripts/owner-port-reach.sh --address 192.0.2.10 --port 8099 \
  --open-control 80 --closed-control 8100 --expect open

Read it with --expect closed before the change, with --expect open after it, and with --expect closed again once you have put the set back. On a node whose director is attached, a port in the set completes a connection even while nothing answers on it, because the director carries every connection the firewall admits to the node's one listener; the reading then shows the firewall admitting the port and nothing more. On a node without the director, a port with nothing listening on it is refused instead, and the probe reads it as refused, not open.

⚠ The gap. The node also takes one port at a time through /edge/served/ctl, which declares or undeclares a single port and reports whether the host agent admits the set, but the valis command line has no verb for that door. A 9P client composes the request itself, and from the command line you change the whole set through config.

5. Enable, start, and check status

systemctl enable --now valis.service     # enable at boot + start now
systemctl status valis.service           # unit + MainPID (fulcrum) state
journalctl -u valis.service -f           # follow the bring-up + serving log

Readiness is meaningful, and it is reported in two places that are not the same place. The bring-up writes its account to the journal, and the resident separately publishes a one-line summary to the supervisor, which systemctl status shows as Status:. Neither carries the other's text, so a phrase you cannot find in the journal is very likely a status-field phrase, and looking harder will not turn it up.

In the journal (journalctl -u valis.service), the bring-up says what it did:

  • site configuration complete: every declared site fact was answered, and the count of facts checked follows on the same line. A boot missing one stops here instead, naming the variable it wanted.
  • adopting the DNS descriptor pair: the DNS handoff took; valis is serving authoritative DNS over fulcrum's inherited :53 descriptors, binding nothing privileged itself.
  • the launch handed down the DNS descriptor pair ... and 53 is not configured: the pair arrived, but this node's served set leaves DNS out, so valis closes it once the fabric is up. DNS does not answer on this node; this is not an adoption.
  • :443 public edge opening — certificate in custody for <domain>: a cert was found for the configured domain and the edge is coming up.
  • no usable certificate in custody for <domain>; :443 public edge cert-gated dark: the domain is named but no cert has been issued for it yet. This is what an operator sees between naming a domain and completing the first obtain, and it is the line that says the gate is the certificate rather than the configuration.
  • no edge.domain configured; :443 public edge stays dark: no served domain named, so :443 is intentionally dark by configuration.

In the supervisor's status field (systemctl status valis.service, the Status: line), the resident publishes its serving state and refreshes it on every health-loop pass. It reads <serving state>; cert <days>; store gen <n>, where the serving state is one of:

  • serving :53 + :443: a usable certificate was in custody at boot; the public HTTPS edge is open.
  • serving :53; :443 dark (no cert): a healthy genesis boot: :53 is up (which is exactly what dns-01 needs to obtain the first certificate), and :443 is cert-gated dark until a cert is issued. This is not an error and does not withhold readiness.

A silently dark port would look like a fault; between the two surfaces you get why :443 is or is not open, so you never have to guess.

6. Descriptor headroom

Nothing in the resident gauges descriptor headroom: there is no count in the journal, no field in the status line, and the liveness gate does not read it. A node that has exhausted its descriptors keeps reporting itself as serving and the supervisor is given no reason to restart it, so headroom is a reading the operator takes from outside the process, on the host:

pid=$(pgrep -o -x valis)                   # the resident is fulcrum's child, not MainPID
ls /proc/$pid/fd | wc -l                   # descriptors held right now
grep 'Max open files' /proc/$pid/limits    # the ceiling this process actually has

Match the process name to whatever the launch configuration's :valis-command stages. Take the reading as root or as the run account: another account's /proc/<pid>/fd is not yours to list. The unit sets no descriptor limit of its own, so the ceiling is the service manager's default on this host: read it rather than assume it.

One reading tells you little. Take two, some hours apart under comparable load: a busy node's count moves in both directions, while a count that only ever climbs is a leak, and the moment to act on it is while the number is still nowhere near the ceiling. Put that pair of readings into whatever monitoring you run against the node. Nothing on the node will raise it for you, and the failure it precedes is one every other instrument reports as healthy. Why that is, and what the liveness gate does and does not ask, is set out in backup-and-observability.org.

7. The certificate lifecycle

The :443 edge is obtained and renewed over the ACME/dns-01 spine (custody and the ACME lifecycle are mercer's; valis is the edge that consumes the cert):

  1. Genesis (no cert). Boot with edge.domain set but no cert yet: :53 comes up, :443 is cert-gated dark. Serving :53 is the precondition for dns-01: the authoritative answer proves domain control to the CA.
  2. Issuance. Once a certificate is issued and lands in the custody store (acme.store-path), a boot with the cert in hand opens :443. dns-01 is the only challenge a node answers today, and there is no fallback. Nothing answers an http-01 challenge on :80 or a tls-alpn-01 challenge on :443, so an order for either has nothing to validate against. That is why serving :53 is a precondition of issuance rather than a convenience.
  3. Renewal (hot-swap, no restart). On a successful renewal the live :443 cert is reloaded in place through the shared credential cell: the edge swaps the new leaf without dropping the listener or restarting the unit. A renewal orders under the same account identity the first order used, and it reads that identity from the node's acme.account-contact setting: set it (valis config set acme.account-contact ADDRESS), or renewal refuses every attempt and backs off while the certificate you are serving walks toward its expiry. The boot line reports whether a contact is configured, so check it there rather than waiting for the backoff to tell you.
  4. Staging vs production. Default is the ACME staging CA. Set acme.production to true only when you want a real, publicly-trusted certificate: production carries a strict rate ceiling, so exhaust staging first.
  5. Wildcards, and the name a certificate is filed under. Custody files an issued certificate under one name, and a wildcard identifier such as *.example.com is a matching rule rather than a name: no filesystem will hold it as a directory. So an order is placed with a host name leading, whatever order you asked for it in, and the certificate covers exactly the names you named. Ask for nothing but wildcards and the order is refused before it reaches the CA, because the alternative is to buy a certificate that has no name to be filed under. Every issuance counts against a rate limit whether or not you get to keep it.
  6. A certificate per name, chosen per connection. Ask the node for a name it holds a certificate for and you are presented that certificate. Ask for anything else and you meet the credential sourced for edge.domain, which is the fallback rather than a failure: an unknown name gets the base certificate, and your client then rejects it on a hostname mismatch as it should. Two names count as one only when they differ in case or in a single trailing root dot, because DNS and custody both file them that way. ⚠ Do not read the boot line holds a certificate for N name(s) as any one certificate's subject names: that is the admission list, the names the edge will answer for. Which certificate each of them is served is decided per connection, from custody.

A domain whose DNS lives elsewhere. A node is meant to prove control of every name it serves. For a domain you bring while its DNS stays with a third party, the proof is tls-alpn-01, answered on the node's live :443 edge inside the TLS stack that already serves it, once you enroll that domain through the control plane. That is not built yet. Today such a domain cannot obtain a certificate, and its site does not serve: valis obtain refuses it as a name the node does not serve, before any order reaches the CA. The same holds for a node whose served set leaves out :53, since a node answers dns-01 only from its own :53. To serve a domain today, delegate its DNS to this node and hold its zone here.

Driving the first obtain by hand. The obtain verb runs the order on the process that serves :53, which is what makes the dns-01 challenge answerable at all. Run it from your own machine and name the node:

valis --node NODE config get acme.production
valis --node NODE obtain --domain example.org --contact hostmaster@example.org

Its whole form is:

valis obtain --domain <fqdn> --contact <email> [--profile <name>] [--directory-url <url>]

⚠ Read acme.production first. The first obtain does not refuse a node whose acme.production is false: it orders from the staging authority, or from acme.directory-url when that is set, and serves the certificate it gets, which no browser trusts, on the node's :443 edge. On a node meant to serve the public, set it before the first obtain with valis --node NODE config set acme.production true. A renewal on a node with edge.domain set is refused while acme.production is false, but the first obtain is not, so that read is yours to make.

The verb answers once the order is done. Its reply begins obtained, then a domain NAME line for each name and a next-renewal line, then a line saying where the new certificate is served: edge serving now when the :443 edge took it at once, or edge not open: serving from the next start on a node whose edge is not open yet. A line for each name follows, saying whether a client asking for that name is handed the certificate. With a key the node has not enrolled, the verb is refused with exit 2 and no order is placed.

To tell the new certificate from the one it replaced, compare the leaf's serial and its notBefore. Its issuer may differ, since the authority can sign the new leaf from another of its intermediates, and a production re-order did. The order also places its dns-01 challenge records in the node's zone and takes them out again, and each change moves the zone's SOA serial forward, so expect a higher serial after an obtain.

Both --domain and --contact are required; --domain repeats, or takes a comma-set. ⚠ Take a comma-set only for names that genuinely belong on one credential. Certificate Transparency publishes the name set permanently, so packing several of your domains into one certificate publishes that you hold them all, and nothing takes it back. One certificate per registrable domain.

Like the other owner verbs it takes a pinned node through --node or VALIS_NODE, and it stays on staging unless the node's acme.production setting is true. When you cannot use the route from your machine (the owner port closed, the pin or the device key lost, or the node not answering on the owner port), run it on the node's host as the break-glass route, inside the node's namespace with --keyfile and --endpoint (see the break-glass route):

sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
  /opt/valis/bin/valis obtain --domain example.org --contact hostmaster@example.org \
  --keyfile /var/lib/valis/valis/keyfile \
  --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"

A first-obtain can also be driven remotely by the authenticated owner over the /acme/ctl 9P door: obtain <domain> <contact> starts the dns-01 order on a background thread (single-flight: a second obtain while one runs is refused, since every attempt burns CA rate-limit budget), and status reports idle/running plus the last completed outcome. The door threads no CA selection of its own; the staging-unless-opted-in default above applies unchanged.

Reading back what a node holds. I added valis cert list so you can see which certificates a node holds, and whether its edge serves each one, without a shell on its host. Name the node as you do for every owner verb:

valis --node NODE cert list

The verb changes nothing on the node. It prints one line per certificate the node holds, sorted by the name each is filed under, with five fields separated by tabs:

  1. the name the certificate is filed under;
  2. its not-after instant;
  3. its next renewal;
  4. whether the node's :443 edge serves it: served when the edge hands it out for every name it carries, partial for only some of them, not-served for none, no-edge when the node has no edge listening, and unconfirmed when the node could not read what its edge serves;
  5. the DNS names it carries, joined by commas.

Both instants are whole numbers in the same encoding as the next-renewal line of an obtain reply, or unknown when the node cannot read them from the certificate. For a node that holds no certificate the verb prints nothing, and no line carries key material. Only the node's owner can read the list: a key the node has not enrolled meets an empty view and is refused before the list is read, as with every owner verb. On the node's host it runs by the break-glass route, as obtain does above.

Exit Meaning
0 the node answered; its list is on standard output
1 a fault on this machine, not the node
2 refused, with the list not read: before dialling, for a malformed command line, no pin, or a key file that is absent, unsafe, or malformed; during the handshake and before your key is sent, for a node that does not match its pin; after the attach, for a key the node gave no owner view
3 the node could not be reached, its name did not resolve, or the session with it failed

ubik's certificate panel reads the same list, and orders a certificate through the same door valis obtain writes. On a node that predates cert list, the panel says that the node does not list certificates and still offers the order.

8. Reaching a node's owner control plane

The owner-keyed verbs (zone, apart from load, secondary --offline and delegation; obtain; publish; publisher; restart; key; and config against a running node) reach the node's owner control plane. You run them from your own machine and name the node:

valis --node NODE key list
valis --node NODE config get edge.served-ports
valis --node NODE zone export --origin example.org

That is the normal route, and it needs three things in place: the node pinned on your machine, a device key on your machine that the node has enrolled, and the node's owner port open. From another machine below shows how to set up all three. The client dials the address you pinned and refuses a responder that cannot prove the pinned DID before your key or any request reaches it. The node's answers print as they do on the host.

A key the node has not enrolled. The node still admits the session, but only to an empty view, in which nothing manages the node. Every owner verb reads that view before it sends a request, refuses with exit 2, and says that the node gave the key no owner view: the key is not enrolled on that node, or was retired, and valis key list on the node's host lists the keys it has enrolled. Nothing on the node changes.

The host route stays for when you cannot use the route from your machine: a shell on the node's host, and the verb run inside the node's network namespace with the node's own key. The break-glass route gives it and says when to reach for it.

8.1. From another machine: a pinned node

From a machine of your own you reach the node by name, with a device key the node has enrolled, and the client checks that the node answering is the one you pinned. The route, once:

  1. On the client, valis key generate writes this machine's device key and prints its DID (see On the client: a device key of its own).
  2. On the node's host, valis key enrol DID --label LABEL admits it (see From the host).
  3. On the client, pin the node: valis node pin NAME HOST:PORT DID, with the node's own Ed25519 DID, the did:key:z6Mk... that valis key list prints first on the host. A node is pinned by that form alone: valis node pin refuses any other DID, an X25519 did:key:z6LS... included, with exit 2 and the store unchanged. A store that holds any other DID, or is written in any other form, is refused whole and never read as empty; move it aside and pin the node again.

Then every owner verb takes the name:

valis --node lab key list
valis zone export --origin example.org --node lab
VALIS_NODE=lab valis restart

--node may stand anywhere before a -- word, and --key FILE presents a device key other than the default. Each is given once and needs a verb, as every owner flag does. When the device key the verb would present is missing, the refusal gives the command that writes it: valis key generate, with --out FILE when --key or VALIS_KEY named a file other than the default. Nothing falls back to another key. --node and VALIS_NODE may not be combined with --keyfile or --endpoint: the verb refuses before reading a key or dialling, rather than letting one replace the other, and --key or VALIS_KEY with no node named is refused for the same reason. Under a node, VALIS_KEYFILE and VALIS_MGMT_ENDPOINT are not read. A node that is not the one pinned is refused before your key or any request reaches it, and the refusal names the command that pins it again. The node's answers print as they do on the host.

The key you present under a node is never the node's own. A --key or VALIS_KEY whose DID is the pinned node's DID, the node's keyfile named on its host for instance, is refused with exit 2 before anything is dialled, and the refusal names the key file. I made the client refuse that key so the node's seed never goes out under a pinned name: under a node the verb presents an enrolled device key, and the node's own key is presented on its host with --keyfile and no node.

Opening the owner port, after the pin. The node's owner port, 564 unless the host agent's configuration names another, ships closed: the unit's firewall, which the host agent installs, drops it, and the host agent binds no socket for it. Open it only once the owner's machine has pinned the node, so the first attach from off the host already checks who answers. In this order, with NODE the name you give the node and ADDRESS its public address:

  1. On the node's host, read the node's DID inside its namespace, as the break-glass route shows:

    sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
      /opt/valis/bin/valis key list \
      --keyfile /var/lib/valis/valis/keyfile \
      --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
    

    The first line names the node's own key, and its did:key:z6Mk... is the DID to pin.

  2. On the owner's machine, pin it: valis node pin NODE ADDRESS:564 DID. Pin the address itself. The client dials an IPv4 or IPv6 address as given, with no reverse lookup first, so an address with no reverse DNS record connects like any other; a host name in a pin is resolved forward to its first address.
  3. Only then, on the node's host, open the port with the host agent's owner-port verb, and restart the unit:

    sudo /opt/valis/bin/fulcrum-resident owner-port open
    sudo systemctl restart valis.service
    
  4. On the owner's machine, attach: valis --node NODE key list.

The owner-port verb sets :open-owner-port in the host agent's configuration store and no other setting, though it rewrites the whole file, so hand comments in it are lost. It restarts nothing and checks no pin, so the order above is yours to keep, and the change takes effect at the next restart of valis.service. To close the port again, run sudo /opt/valis/bin/fulcrum-resident owner-port close and restart the unit; the pin stays on the owner's machine for the next open, and enrolled keys stay enrolled across the restarts. owner-port status changes nothing. Each of the three prints the store it read, the configured port, a configured: line with the store's setting, and a live: line saying whether the unit's firewall carries the owner accept line, so between a change and the restart the two lines disagree. open and close exit 0 when the change was made or the store already held it, 1 when the verb refused or failed, and 2 for an unrecognized argument.

An owner port other than 564. The resident picks the owner terminus's descriptor by its port: 564, unless the launch names another with --owner-port N. N is a whole number from 1 to 65535, written in digits alone, given once, and not a port another service takes (25, 53, or 443); given any other value, the resident refuses the launch and does not start. At boot it writes one line to the journal for each descriptor it was handed that no service it runs takes:

valis: descriptor 9 (stream, port 2564) was handed down and answers for no service this node runs, so it serves nothing

A lone half of the DNS pair gets a line of its own, ending was handed down without the other half of the DNS pair, so it serves nothing. ⚠ The host agent does not yet pass --owner-port when it launches the node. Configure the host agent with an owner port other than 564 and the boot reports that port's descriptor with the line above, and the owner terminus does not come up; keep the owner port at 564 until the host agent passes it.

Checking the port from another machine. scripts/owner-port-reach.sh reads whether one connection to an address and port completes or gets no answer. A firewall that drops a port reads as no answer, so a reading means something only beside two controls on the same address: a port the node admits and a port it drops. The probe reads both first and gives no verdict when either reads otherwise:

scripts/owner-port-reach.sh --address ADDRESS --port 564 \
  --open-control ADMITTED_PORT --closed-control DROPPED_PORT --expect open

It exits 0 when the port read as expected, 1 when it read otherwise, 2 for no verdict, and 3 for a usage error, which includes --expect without both controls. Run --self-test first to see the probe separate the cases on loopback.

To see which view the node gives a key, build owner-view-proof with scripts/build-owner-view-proof.sh, which writes bin/owner-view-proof, and attach with it: owner-view-proof --node NODE --key FILE --expect owner-frame. An enrolled key gets owner-frame, a key the node has not enrolled gets empty, and owner-view-proof --anonymous --endpoint ADDRESS:564 gets published-floor. It exits 0 when the view matched --expect, 1 when the node gave another, 2 when owner-view-proof refused an argument or refused the attach before sending anything, and 3 when the node could not be reached. It prints only a key's public DID, and its --self-test shows it telling the three views apart on a scratch fabric. With --node, --boot also prints boot: HEX, the node's boot identity, which the node draws afresh at every start, so two equal readings bracket a span with no restart in it. --pub SLUG prints pub: SLUG bytes N md5 HEX for the publication as read through the attached view, or pub: SLUG absent. The length and digest are of the bytes as published, so a file that ends in a newline counts that newline.

Owner-port connection caps. The owner port holds at most 4 connections at once from one origin and 16 from all origins together, counting each from the moment it reaches the port, handshake included. Change them with valis config set owner-port.sessions-per-origin N and valis config set owner-port.session-ceiling N, each a whole number from 1 to 63; the node reads the new cap at the next connection. One origin is one IPv4 address or one IPv6 /64, so machines behind one public IPv4 address share one origin and its 4 connections. A connection past either cap is closed before anything is read from it, and the node logs the refusal at WARN at most once per origin per minute, in a line that begins origin cap: refused a connection from. A session the node grants nothing, with no enrolled key and no accepted capability, has 10 seconds before the node closes it.

Where this stands. This route is proven from another machine against a deployed node. The client refused an attach with no pin before dialling anything, the pin was written before the port opened, and the pinned attach listed the keyring. The client refused a pin naming another DID, and a key the node never enrolled got an empty view. The port read closed, open, closed, and open again from off the host as owner-port and a restart moved it, each reading beside both controls. The node's director carries every connection to the owner port to the owner terminus: on that node a silent connection to the port is cut at 10 seconds, and a fifth connection from an origin already holding 4 is refused at accept. Only the terminus's claim on the port, where the director carries those connections, arms that bound and that cap.

Managing a node over this route is proven on a deployed node that serves the public, in one run from the owner's machine with no shell on the node's host. A zone record was added and deleted, and was answered and then gone over UDP and TCP. A page was published and revised, and each text was served at once. With one session from the owner's machine already open, owner-port.sessions-per-origin was set to 2, and of two more connections from that machine one was held and the other refused at accept; the setting was then reset. Port 8099 was added to the served set, read open from off the host beside both controls, and read closed again once the set was put back. The node's boot identity read the same before and after those changes, so none of them restarted it. A certificate for the names the node's certificate already covered was ordered from the production authority, and the :443 edge served the new one at once, with no restart. Last, the node was restarted and answered from a new boot identity, with the device key still enrolled and both settings the run had changed reading as they did before the run. Each change was read back after it took effect. A key the node never enrolled got an empty view and was refused by every one of those verbs with exit 2, and nothing on the node changed. A peer with no key got only the published floor, with no door to the names, the configuration, or the node.

That run showed two gaps. The restart ended the owner's ubik session, and the ubik of that run had to be reconnected by hand; ubik now attaches again on its own, as Managing a node from ubik describes. The page stays published, since no verb retracts a publication.

⚠ The gaps. Opening or closing the port takes a restart of the unit. The first proof on a node that serves the public used the valis command line and owner-view-proof; ubik has since managed that node too, and Managing a node from ubik names what ubik has not shown there yet. Enough origins together can still fill the 16 and keep you out of the port for as long as they keep reconnecting, and the caps are defaults I chose, not measurements; your way in then is the host, by the break-glass route.

One store, two clients. The pin store is $XDG_CONFIG_HOME/ubik/node-pins.sexp (or ~/.config/ubik/node-pins.sexp), shared with ubik, which reads and writes the same file. Before you pin with valis on a machine where ubik also runs, make sure that ubik writes the shared format, or one client may refuse the other's store. Two clients writing the store at the same moment never leave it torn. Both clients change the store under its lock, the file node-pins.sexp.lock beside it: valis node pin and valis node unpin take it, and so does ubik when it records a pin, so neither client loses a change the other makes. valis waits up to ten seconds for that lock. If another writer keeps it longer, the verb exits 3, names the store, and changes nothing; run it again once the other writer finishes.

8.1.1. Managing a node from ubik

ubik reaches a node the way the command line does: by a pin name in the same store, with a device key the node has enrolled. I made ubik present a key of its own, enrolled on each node under its own label, so you can retire the key ubik presents without touching the one your command line presents. Create it on your machine and enrol its DID on each node's host:

valis key generate --out ~/.config/ubik/owner.key --label ubik-laptop

ubik refuses to present the node's own key, by pin name or by any other route, and refuses a key file that grants anything to its group or to others; both refusals come before anything is dialled. A key the node has not enrolled meets an empty view, and ubik closes the connection.

Once ubik is attached, you manage the node from its panels: its configuration, the served set, a restart, its certificates, publication, and the keyring. The panels write the node's ordinary ctl doors, the ones the command line writes, and the served-ports panel writes the per-port door the command line has no verb for. Each panel reads the node again after every act, so what you see is what the node now holds. A disruptive act takes a second click on a confirmation that names its effect: a restart, taking a port out of the served set, retiring a key, or an order from the production authority. ubik never offers to take out the port it is attached through.

When the connection goes, whether ubik sent the restart or the node dropped off the network, ubik attaches again on its own: at once, then after 1, 2, 4, 8, and 16 seconds, and every 30 seconds after that. Every attempt checks the pin again. An attempt that meets a node no longer matching its pin stops there, and ubik shows you the refusal instead of trying again. A node that goes silent without closing the connection is reported lost within 25 seconds of its last answer. Once ubik is attached again, every panel reads the node again. ubik's own manual, docs/OWNER-MANAGEMENT.org in the ubik repository, covers the panels in full.

Where this stands. On a test node, ubik attached with its own enrolled key and, when given the node's own key, refused it before dialling. Each panel changed the node, and each change read back through the valis command line or from off the host. ubik attached again after a restart it sent, after the node reset the connection, and after a silent drop, and it stopped when the pin was changed under it. A key the node never enrolled changed nothing.

Most of the same checks then ran on the deployed node that serves the public, the one the command line was proven on, through ubik-manage, ubik's scripted runner, from the owner's machine with no shell on the node's host. ubik attached with its own enrolled key and read the same keyring listing as the command line, line for line. Given a stand-in key pinned as the node's own (the node's own key never leaves its host), ubik refused it on every route and dialled nothing.

With its own session open, ubik set owner-port.sessions-per-origin to 2. Of two more connections from the owner's machine, the node held one open and refused the other at accept. ubik then reset the setting to its default, as it was before. It added port 8099 to the served set; from off the host the port then read open, and ubik took it out again. It revised a page already published and then restored the page's earlier text, each text read over HTTP. It retired a key enrolled for the checks, which the command line then read as retired. Its certificate list matched the command line's line for line, and its sessions view matched the connections counted on the owner's machine.

ubik refused each disruptive act sent without its confirmation. Even with the confirmation given, it refused to take out the port it was attached through, a request its panels never offer. A restart sent through ubik brought the node back on a new boot identity. ubik attached again on its own about 3 seconds later, and every panel read the node again. A key the node never enrolled got an empty view, every act sent with it was refused, and nothing on the node changed. Every setting, port, and page the checks changed was put back afterwards. Every panel was then opened against that node and showed what the node holds, with the rough edges listed below.

Not yet shown on that node:

  • A ubik session coming back on its own after something else, such as a deploy, restarts the node. The restart in those checks was sent through ubik, so it does not show a restart from elsewhere. The one other ubik session open at the time came from an older build of ubik, without automatic reattachment, and it stayed down.
  • A lost connection without a restart, whether a reset or a silent drop. Causing one on that node needs a shell on its host, so ubik attaching again after one is shown on a test node only.
  • A certificate ordered through ubik. On that node ubik listed the certificates and refused an order without its confirmation, and no confirmed order was sent; certificates ordered through ubik have been issued on a test node only.

ubik's interface has these rough edges today:

  • The publication panel takes the page's name in ubik's input line at the foot of its window. The input line shows no prompt, and no cursor until you type. The editor above it, where you compose the page, shows a blank white field with no cursor.
  • A divider under the publication panel's instructions looks like the box they mention, but is not one.
  • Scrolling a pane redraws the whole pane with a visible flash.
  • The configuration panel works, but its buttons are oversized and its layout is hard to read.

8.2. From the node's host: the break-glass route

Reach for the host when you cannot use the route from your machine:

  • the owner port is closed, or the node does not answer on it;
  • the pin, or the device key, is lost from your machine;
  • the change is one only the node's own key may make, such as enrolling a device key (see Enrolling and retiring an owner device key).

On the host the verbs reach the resident over its loopback fabric. On a deployed node that loopback belongs to the resident's network namespace, not to the host. The fabric binds a fresh port at every boot, and the resident writes the address it bound to mgmt-endpoint in its data root, which is /var/lib/valis/valis/mgmt-endpoint under the shipped unit.

Run the verb on the node's host, inside that namespace, which you enter by the resident's pid:

sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
  /opt/valis/bin/valis zone export --origin example.org \
  --keyfile /var/lib/valis/valis/keyfile \
  --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
  • Read mgmt-endpoint every time. The port is drawn afresh at each boot, so an address you saved is wrong after the next restart.
  • Enter by pid. nsenter -n enters the network namespace and nothing else, so host paths resolve as usual, and the pid is always there to name it by.
  • Run the verb as the account that owns the keyfile. Entering the namespace needs root, so nsenter runs under sudo, and -S and -G then drop the verb to the run account before it starts. Give them numbers, as id -u and id -g print them: nsenter does not accept account names there.
  • The keyfile must be the run account's own. The verb presents the file named by --keyfile or VALIS_KEYFILE, or the one in its data root, only when it is a regular file, not a symbolic link, owned by the account running the verb, and granting nothing to its group or to others. Anything else is refused before the key is read, with a message naming the file and what was wrong; for a file another account owns, it names both account numbers. Run as root, the verb refuses the run account's keyfile for exactly that reason. A keyfile that is missing, or that is not exactly the 32 octets of a seed, is refused the same way, and never reported as a node that could not be reached. Every owner verb (zone, obtain, publish, publisher, restart, key, and config) exits 2 on this refusal, as on any refusal made before anything is sent.
  • The client checks who answered. The keyfile names the node's own key, so the verb presents that key to the node that holds it and to no other. A responder that cannot prove it holds that key is refused before anything else is sent, so a process squatting on the endpoint is refused rather than answered, and it never sees the attach. Every owner verb exits 2 on that refusal too, and a restart waiting for the node to come back stops at the first such refusal.
  • Arguments the verb refuses also exit 2. Each owner flag (--keyfile, --endpoint, --node, and --key) may stand anywhere before a -- word, is given once, and needs a verb. A flag given twice, a command line that has an owner flag and no verb, or an owner flag beside --resident, --daemon, --dev, --backup, or --restore is refused with exit 2 before any key is read or anything is opened, dialled, or started.
  • Pass both --keyfile and --endpoint. The verb keeps root's environment, so its default data root holds neither the node's keyfile nor its endpoint, and the defaults find nothing.
  • An ssh forward reaches nothing. A forward lands on the host's loopback, which is a different 127.0.0.1 from the one the fabric is bound to, and nothing listens there for it. Off the host, use the route from your own machine, in Reaching a node's owner control plane.

9. Enrolling and retiring an owner device key

A node has one owner identity, its did:key, and one seed behind it, which never leaves the host. I wanted the owner's other devices to manage a node without a copy of that seed, so the node's factotum holds a keyring of device keys beside the seed. Each is a device's own Ed25519 did:key, enrolled under the node's fixed owner identity. Enrolling or retiring a device key never changes the node's identity and never moves, rewrites, or copies the seed.

What an enrolled key reaches. A session presenting a live enrolled key is admitted to the owner frame: the node's services through their loaded modules, minting grants, signing with the node's own key, and backups. It is the frame an owner client such as ubik reaches over the node's capability system. The session keeps the device's own DID as its principal, so the session listing names the device rather than the owner.

The node's REPL is not in that frame. The owner's total reach is host authority: ssh to the host, and from there the node's REPL. No device key reaches it, and the ssh-reality proof, which mints a grant over the whole namespace, admits only the pinned operator SSH key. Enrol only your own devices. Give another person a grant, or an instance of their own, never a place on your keyring.

What a key carries. Each key records its DID, a label you choose, the key that enrolled it, and when. Retiring it records who retired it and when, and ends it. It has no expiry, so it stays admitted until you retire it. Read the listing from time to time and retire the key of any device you no longer hold.

Who may change the keyring. Only a session holding the node's own key enrols, so enrolment happens on the host. The node's own key or any live enrolled key may retire a key, so a lost device can be cut off from another device you still hold. The node's own key can never be retired, and a retired key cannot be enrolled again while the keyring records its retirement: a device whose key was retired mints a new one and is enrolled under that. Setting a refused keyring aside discards those records (see Healing a keyring the node refused).

9.1. On the client: a device key of its own

A device you manage the node from holds a key of its own, never a copy of the node's seed. I made the client mint that key itself, so the secret is created on the machine that uses it and never crosses a network or a command line. On the client, run:

valis key generate                    # writes ~/.config/valis/device.key
valis key generate --out FILE --label laptop

The key goes to $XDG_CONFIG_HOME/valis/device.key, or ~/.config/valis/device.key when XDG_CONFIG_HOME is unset, empty, or not an absolute path, unless --out names another file. A relative XDG_CONFIG_HOME is ignored, as the XDG Base Directory specification asks, so the default path never depends on the directory you run the verb from. valis key generate does not read VALIS_KEY: to write the key a later session will look for at another path, name it with --out. The file holds the 32 octets of an Ed25519 seed, the same format as the node's own keyfile, at mode 0600 whatever your umask. When the valis directory does not exist it is created at 0700; a directory that already exists keeps its mode. The verb never replaces a file and never writes through a symbolic link: with anything already at the path it exits 2 and writes nothing. It contacts no node, so --keyfile, --endpoint, --node, and --key are refused.

Standard output carries the device's did:key and nothing else, and standard error carries the file's path and the line to run on the node's host:

valis key enrol did:key:z6Mk... --label laptop

The label is --label when you give one, and otherwise this machine's short host name with any character a label cannot carry replaced by a hyphen. A label is 1 to 64 of the characters A-Z, a-z, 0-9, dot, underscore, and hyphen, the rule the node's keyring applies at enrolment. A --label outside that rule is refused with exit 2 before anything is written, so the line printed is always one the node accepts. Run that line on the node's host, as the next section describes. The seed is never printed.

The client then presents the key to a node you have pinned, with --node, as From another machine describes.

Exit Meaning
0 the key was written; its DID is on standard output
1 the key could not be written
2 refused, nothing written: something is already at the path, the label is one the node's keyring would refuse, or a flag that reaches a node was given

9.2. From the host: valis key

Run valis key the way every owner verb runs on the host, inside the resident's network namespace as the run account, with --keyfile naming the node's keyfile and --endpoint read from mgmt-endpoint (see the break-glass route). The session then presents the node's own key, which is the only session that enrols. The verb has three subcommands, and the wrapper below supplies the rest of each command line:

  • valis key enrol <did> --label <label> enrols a device key;
  • valis key list prints the keyring;
  • valis key retire <did> retires a device key.
k() {
  sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
    /opt/valis/bin/valis key \
    --keyfile /var/lib/valis/valis/keyfile \
    --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)" "$@"
}
k enrol did:key:z6Mk... --label laptop   # the device's Ed25519 did:key
k list                                   # the node's own key, then every device key
k retire did:key:z6Mk...                 # either DID form of the device key

A label is 1 to 64 of the characters A-Z, a-z, 0-9, dot, underscore, and hyphen. The listing prints the node's own key first, then one line per device key with its Ed25519 DID, its X25519 transport DID, its label, when and by whom it was enrolled, and either state=live or, for a retired key, when and by whom it was retired. Retired keys stay in the listing.

The verb exits with one of four statuses:

Exit Meaning
0 the node enrolled or retired the key, or printed the listing; its reply is on standard output
1 a fault on this machine, not the node, such as no management endpoint found, or a malformed --endpoint; standard error names it. The request may have reached the node, so check with valis key list before you repeat an enrol or a retire
2 refused, and nothing changed: the node replied error <reason>, the arguments were refused before anything was sent, or the verb refused the node that answered or the key file it was given: absent, unsafe, not the 32 octets of a seed, or the node's own key presented under a node
3 the node could not be reached, its name did not resolve, or the session with it failed; standard error names it. The request may have reached the node, so check with valis key list before you repeat an enrol or a retire

The reasons a node gives are not-seed-holder (an enrolment from anything but the node's own key), not-owner, malformed, unsupported-proto, invalid-key, invalid-label, seed-key (the node's own key named), already-enrolled, already-retired, not-enrolled, and store (the keyring could not be written). With a keyfile that is not the node's own, such as a device key, the verb exits 2 before anything is sent: the keyfile route accepts only the node that holds the key it presents, and the node answering does not hold that key.

After a successful enrolment the verb prints a notice on standard error, so the consequence is in front of you at the moment you grant it:

valis: key: <did> is now admitted to this node's owner management frame: its services and their modules, minting grants, signing, and backups.
valis: key: it stays admitted, with no expiry, until you run valis key retire <did>
valis: key: enrol only a device you hold.

9.3. From ubik: the keyring panel

ubik's keyring panel lists every key the node has enrolled, with its label and its state, and under each key the sessions presenting it that are live now. It retires a key through the same /id/ctl door valis key retire writes, after a second click on a confirmation that names the effect. Retiring the key ubik itself presents takes a confirmation of its own, saying that the session ends and that this ubik can no longer attach to that node; ubik reports that retirement as applied, and the session as ended. The node's own key is never offered for retirement.

For a new device, generate its key on that device, then paste its public did:key and a label into the panel. The panel checks both and shows the command to run on the node's host, valis key enrol DID --label LABEL, and sends nothing to the node: only a session holding the node's own key can enrol a device key. The device's private key never leaves the device.

The panel shows each key's sessions and does not fence them. Fencing a session from ubik waits for richer session control on the node.

9.4. From a REPL in the running image

The verb and the door call the same two functions, and a REPL attached to the running image can call them directly:

(valis/src/identity/keyring:enrol-owner-key
 (valis/src/namespace/id:factotum-keyring) "did:key:z6Mk..." :label "laptop")
(valis/src/identity/keyring:retire-owner-key
 (valis/src/namespace/id:factotum-keyring) "did:key:z6Mk...")

A refused change signals a condition naming its reason and writes nothing. On a node behind the director the development listener cannot be reached today (see the listener's destination), so there the host verb is the route.

9.5. Over 9P: /id/ctl

The host verb writes one line to the node's /id/ctl over its sealed owner session, and an owner client on the host writes the same line. Each open of /id/ctl is its own conversation: write one command, then read its one-line reply on the same open.

key proto=noise key=<device ed25519 did:key> label=<label>
delkey key=<did:key>

The reply is enrolled <did>, retired <did>, or error <reason> with the reasons above, and a refused command changes nothing. A read before any command returns the listing. Only the owner frame names /id/ctl: no grant, bearer name, or delegated token reaches it.

9.6. Retiring a key cuts off its sessions

Retiring a key takes it from every session presenting it, at once: each such session's view empties and every handle it holds is revoked, so a client in the middle of a read gets an error rather than more data. A session that attaches with the key while the retirement is in progress is cut off as it attaches. A device that retires its own key is cut off the same way, before the reply reaches it; the retirement still stands, and valis key list on the host shows it.

If the node cannot finish cutting off a session, the retirement still stands and the verb still reports it, and the journal names the step that failed. Restart the node to end every session at once.

9.7. Where the keyring lives, and when the node refuses it

The keyring is keyfile.keyring beside the keyfile in the data root, /var/lib/valis/valis/keyfile.keyring under the shipped unit, at mode 0600 and owned by the run account. Every change is one record appended to it and synced before the node acts on it. A change the node cannot write leaves the file exactly as it was, and the verb replies error store. If the node cannot even put the file back, the journal says so and the next boot refuses the file; heal it as the next sections show. Do not edit it by hand.

The node refuses to start on any of these keyrings:

  • one it cannot read whole: a torn last record, a line that does not parse, a version it does not know, or a retirement of a key never enrolled;
  • one reached through a symbolic link, one another account owns, or one granting anything to its group or to others;
  • one written for another owner, such as a keyring copied from another node, or an old keyring left beside a seed minted afresh.

The node refuses to start in the same way on a keyfile it will not load: one reached through a symbolic link, one another account owns, one granting anything to its group or to others, or one that is not exactly the 32 octets of a seed. The keyfile is left as it was.

A refused boot exits 78, and the unit's RestartPreventExitStatus=78 keeps systemd from restarting it into the same refusal. The journal names the file and the problem. A refused node leaves no image and no listener to connect to, so heal it from the host as the next section shows.

9.8. Healing a keyring the node refused

The standard heal is to recondense the node from a backup. Restore the backup into a fresh state directory as Backup and restore shows, then start the unit. The node comes back with the keyring that backup carried and everything else with it. A restore cannot know of a key retired after the backup was taken, so retire any such key again once the node is up.

On a node installed from the delivery archive alone, recondensing from a backup is the only heal today. The manual heal below needs SBCL, Quicklisp, and a valis checkout with its dependencies on the host, and the delivery archive carries neither a REPL entry nor the valis source.

The manual heal repairs the keyring alone and leaves the rest of the node as it stands. Run it on the host, as the run account, with the unit stopped (a refused node already is):

  1. Start SBCL as the run account with the data root the unit uses, and load valis from a checkout the way the Makefile's test-one recipe does:

    sudo -u valis env XDG_DATA_HOME=/var/lib/valis sbcl
    
    (require :asdf)
    (load "<quicklisp>/setup.lisp")
    (push "<workspace>/" asdf:*central-registry*)
    (asdf:load-asd #p"<checkout>/valis.asd")
    (asdf:load-system :valis)
    
  2. Load Slynk, start the loopback development listener, and reach it from your own machine through an ssh tunnel:

    (asdf:load-system :slynk)
    (valis/src/dev/slynk:start-development-listener 4005)
    
    ssh -L 4005:127.0.0.1:4005 <host>   # then attach your editor to localhost:4005
    

    The plain SBCL debugger in the ssh terminal offers the same restarts if you prefer it, and then you need no listener at all.

  3. Evaluate the heal:

    (valis/src/keyring-heal:heal-owner-keyring :state-dir #p"/var/lib/valis/")
    

    It takes the node configuration's lock for the whole heal. It refuses while the node runs, and when no seed is at the keyfile path. It then reads the seed and the keyring exactly as the boot does. Either it reports that the boot's loader accepts the keyring, or it stops on the boot's own refusal with the restarts that apply to it.

  4. Choose one restart:

    • keep-keyring, offered when the file was refused only for its mode or for the account owning it. For the mode, it sets 0600 on the same file, and it is offered only when the file grants no write permission to its group or to others: a keyring others could write may hold content the node never wrote, so recondense it from a backup or set it aside. For another account's file, it installs a copy the run account owns once the content parses whole under the node's owner, and sets the original aside.
    • salvage-keyring, offered for a damaged file. It keeps the whole valid records before the damage, drops the rest, and retires every key a dropped line names.
    • recondense-from-backup, offered for every refusal. Restore a backup into a scratch state directory first, then give the restart that directory; it takes only the keyring from there:

      sudo -u valis mkdir -m 0700 /tmp/keyring-restore
      sudo -u valis valis --restore --in /path/to/valis-backup.sealed \
        --state-dir /tmp/keyring-restore --acme-store-dir /tmp/keyring-restore/acme
      

      A key the damaged keyring enrolled and the backup's keyring lacks is not on the healed keyring. The heal report lists each such key; enrol it again. Recondensing applies the retirements the damaged keyring holds, even when the node refused that file for its mode or its owner. When the damaged keyring could not be read at all, the heal report says so and its retirements are not applied, so retire any such key again once the node is up.

    • set-keyring-aside, offered for every refusal. The node then starts with no enrolled keys, and you enrol your devices again.

    After a restart acts, the heal checks the result with the boot's loader again and returns a report: every key live and retired, every key the heal retired and the line that named it, every line a salvage dropped, and every key the damaged file enrolled that a backup did not carry.

  5. Start the unit and read the boot line, then remove the scratch restore, which holds a copy of the seed:

    sudo systemctl start valis.service
    journalctl -u valis.service -n 20
    sudo rm -rf /tmp/keyring-restore
    

What the manual heal guarantees:

  • the damaged file is renamed aside as keyfile.keyring.refused-<UTC time> and never removed;
  • the seed is never moved, rewritten, or created;
  • only the node's own seed binds the result, so another owner's keyring, or a backup of another node, is refused;
  • after keep-keyring, salvage-keyring, or recondense-from-backup, a key the damaged file retires, or that a damaged or torn line names, stays retired, so a device retired that way is enrolled again with a new key. A torn retirement that names no key retires every live key, because nothing on the line says which key it retired. This holds whenever the heal can read the damaged file; when it cannot, the heal report says so.

set-keyring-aside discards the retirement records. After a set-aside the node no longer knows which keys were retired, so re-enrol only keys you hold. Enrolment still needs a session proven with the node's own key.

9.9. The development listener

The owner frame includes the node's configuration and its restart door, so an enrolled device can set development.listener-port and restart the node. An enrolled device can turn the development listener on but cannot reach it: the listener binds loopback only, and on a node behind the director nothing reaches it today. The node's REPL stays with host access.

9.10. Not built yet

  • Opening or closing the owner port takes effect only at a restart of the unit; nothing changes it on a running node.
  • The owner port's caps hold one origin to 4 connections and all origins together to 16, but enough origins together can still fill the 16 and keep the owner out of the port while they keep reconnecting. The figures are defaults, not measurements.
  • ubik has managed a deployed node that serves the public and attached again after a restart it sent, but three things are not yet shown there: a ubik session coming back on its own after something else restarts the node, a lost connection without a restart, and a certificate ordered through ubik. The last two are shown on a test node. The publication panel, scrolling, and the configuration panel have the rough edges Managing a node from ubik lists.
  • ubik shows each key's live sessions but does not fence one; fencing a session from ubik waits for richer session control on the node.
  • A device key is enrolled only on the node's host, by design, since only a session holding the node's own key can enrol one. ubik's keyring panel and valis key generate give you the command to run there.
  • An enrolled key reaches the owner frame over the owner port, not over the :443 edge: the edge's owner proof accepts only a signature by the node's own key.
  • An enrolled key has no expiry. Until it does, the listing and valis key retire are how a key you no longer hold is ended.
  • On a node installed from the delivery archive alone the manual heal is unavailable: the archive carries no REPL entry and no valis source.
  • No verb retracts a publication; a page stays served until you revise it.
  • The served set's per-port door, /edge/served/ctl, has no command-line verb; ubik's served-ports panel composes its request one port at a time, and the command line changes the whole set through config.
  • A transfer key for a zone's secondary is minted only on the node's host, never over a pinned node.
  • The first obtain is not refused on a node whose acme.production is false, although a renewal there is, so read it before you order a certificate for a node that serves the public.
  • Mail is not managed from your own machine: the command line has no mail verb, and the node's mail over the owner port has not been through the checks the other verbs have.

10. Managing standing grants

A module loads only with two signatures: its publisher's vouch and your grant. You give a grant for one module, or once for a publisher as a standing grant, which covers everything that publisher vouches. You file publishers and grants by writing commands to /cap/admit in your owner session; grant-publisher PUBLISHER-DID mints a standing grant that lasts a century and files it in one step. A grant you mint yourself and file with file-standing-grant ends at its own expiry, which admission checks. Because a standing grant can stand for a century, I made the node keep a record of every change to your trust and ask you to look at each standing grant again from time to time.

Two read-only files sit beside /cap/admit in the owner's /cap, and nowhere else: no grant, bearer name, or delegated token reaches them. Each is rendered when you open it, so everything you read from one open is one state.

/cap/grants
the inventory: one stanza for each grant you hold, then one for each publisher you have filed, each a run of KEY VALUE lines, stanzas separated by a blank line. A standing grant's stanza carries grant (its hash), kind, state (active, withdrawn, backed-out, or revoked), publisher, filed-at, reviewed-at (or never), review (due or ok), last-admission, modules, and one admitted line for each module it admitted. Times are Unix times. No stanza carries the text of a grant or a vouch.
/cap/audit
the grant audit trail, one entry a line, oldest first: UNIX-TIME ACT SUBJECT ACTOR. ACT is one of filed-publisher, filed-vouch, filed-list, filed-owner-grant, cleared-owner-grant, filed-standing-grant, granted-publisher, withdrew, backed-out, revoked, reviewed, and restored. SUBJECT is a grant hash, a designation, or a did:key, and ACTOR is the DID of the session that acted, or local.

On a node in service the trail only grows, and a restore whose backup carries a trail replaces it whole. It records changes to trust, so a refused act, or one that changes nothing, leaves no entry. When an entry cannot be written the act still stands, and its reply ends with a line saying the trail did not record it. A torn final line, left by a crash during an append, is skipped. Any other line that is not an entry makes /cap/audit an error beginning cap audit unreadable: line N. An unreadable or unsafe trail gives cap audit unreadable: and the reason, with no line number. Either way /cap/grants, which takes its filing and review times from the trail, is an error beginning cap grants unreadable. Backups carry the trail (see Backup and restore).

Reviewing a standing grant. An active standing grant is due a review once more than module.standing-grant-review-days days have passed since the later of its filing and its latest review: 90 unless you set it, a whole number from 1 to 3,650 (see the environment seam). An active grant with neither on record is due at once. Write reviewed GRANT-HASH to /cap/admit to stamp it, and read back:

reviewed standing grant <hash> at <unix-time>; the review is advisory and admits nothing

The stamp is the trail's reviewed entry and nothing else, so a review whose entry cannot be written is refused and nothing is stamped. Only an active grant you hold takes one: a withdrawn, backed-out, or revoked grant is refused. The review is advisory: nothing refuses a module under a grant that is due. While any active standing grant is due, the node's status line (systemctl status valis.service) ends by counting them:

; 2 standing grants due a review (90-day interval)

Previewing before you withdraw or back out. Write preview withdraw GRANT-HASH or preview back-out GRANT-HASH and read what the act would do, with nothing changed. The first line is preview ACT HASH: nothing was changed, the second state and the grant's state, and then what the act would do: for a withdrawal, the modules that stay and what a fresh install from that publisher would then need; for a back-out, the modules admitted under the grant, every one refused from then on. When the act would change nothing, one line says so. A back-out preview is refused wherever the back-out itself would be.

Withdrawing and backing out. withdraw-standing-grant GRANT-HASH ends the grant for anything new: it admits nothing more, and the modules it admitted stay, across restarts too. back-out-standing-grant GRANT-HASH withdraws it and revokes its hash, so every module it admitted is refused from then on. Withdrawal is one way, and there is no reinstate: to trust that publisher again, file a new standing grant, which carries a new hash and its own filing time. A publisher holds one active standing grant at a time, so filing a different one while one is active is refused, naming the active grant's hash.

⚠ Where this stands. The command line has no verb for /cap/admit, /cap/grants, or /cap/audit, and ubik does not yet compose any of them; you reach them with a 9P client in your owner session. A back-out refuses the modules it names from then on, but nothing removes them from a running node: they run until a restart, when the boot refuses each by name and comes up serving the rest.

11. DNS zone management

valis is authoritative for the SOA zones it serves on :53. The zone data itself is operator state: a node comes up serving whatever zones are in its operator-state store, and comes up with none on a genesis boot. Loading and maintaining that data is the valis zone verb family: a headless, owner-authenticated client the deploy recipe drives non-interactively, and the operator drives by hand.

How it authenticates. From your own machine you name a pinned node with --node NAME or VALIS_NODE, and each verb presents your enrolled device key and reaches the node's owner-gated management axis through its owner port, as From another machine describes:

valis --node NODE zone record add --origin example.org --owner www --ttl 300 --type A --rdata 192.0.2.10
valis --node NODE zone export --origin example.org
valis --node NODE zone record delete --origin example.org --owner www --type A --rdata 192.0.2.10

An added record appears in the export and is answered over UDP and TCP; a deleted one is gone from both. Only a key the node holds in its keyring is given the management axis, so an absent or wrong key fails closed: the management axis never opens to an anonymous caller, and a key the node has not enrolled is refused with exit 2 and changes nothing.

From the node's host, the break-glass route. When you cannot use the route from your machine (the owner port closed, the pin or the device key lost, or the node not answering on the owner port), or to mint a transfer key for a secondary (see secondary below), run the verb on the node's host. There it reaches the management axis over the resident's loopback fabric, inside the resident's network namespace, at the endpoint the resident wrote at boot, and presents the owner key custody holds in the StateDirectory keyfile. The key is supplied as a path, resolved --keyfile PATH > VALIS_KEYFILE > the StateDirectory keyfile; the key bytes never cross the command line. The break-glass route gives the command that gets it there:

sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
  /opt/valis/bin/valis zone export --origin example.org \
  --keyfile /var/lib/valis/valis/keyfile \
  --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"

A transfer key is kept under the data root of the process that mints it, and the recipe above keeps root's environment, whose data root is not the node's. To mint one the node can use, name the node's data root as well, so the key lands in the node's custody under /var/lib/valis/valis/tsig/:

sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
  /usr/bin/env XDG_DATA_HOME=/var/lib/valis \
  /opt/valis/bin/valis zone secondary --origin example.org --peer 198.51.100.2 \
  --keyfile /var/lib/valis/valis/keyfile \
  --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"

Either way, a verb that refuses before anything is sent exits 2: for a key file it will not present, a responder that is not the expected node, or arguments it refuses.

The verbs.

valis zone create --origin <fqdn> [--peer <addr>] [--secondary-ns <name>]
valis zone record add --origin <fqdn> --owner <name> --ttl <s> --type <type> --rdata <value>
valis zone record delete --origin <fqdn> --owner <name> --type <type> --rdata <value>
valis zone import --origin <fqdn> --file <master-file>
valis zone export --origin <fqdn> [--out <file>] [--full]
valis zone delete --origin <fqdn>
valis zone delegation --origin <fqdn> [--wait] [--timeout <s>] [--interval <s>] [--allow-range <client=range,...>]
valis zone secondary --origin <fqdn> --peer <addr> [--notify <ref>] [--key-name <name>] [--master-ip <addr>]
valis zone secondary --origin <fqdn> --peer <addr> --key-name <name> --existing-key

# Straight to operator state, for a node that is not serving yet:
valis zone load --origin <fqdn> --file <master-file> [--migrate]
valis zone secondary --offline --origin <fqdn> --peer <addr> [--notify <ref>] [--key-name <name>]
  • create brings a whole domain up in one move: it mints the apex records from the domain template, enrols the secondary when you name a peer, and prints the block you paste at the registrar. Reach for this before reaching for import on a new domain.
  • record add and record delete publish or retract one typed record and leave the rest of the zone alone. This is the editing path. An --owner ending in a dot is absolute; without one it is relative to the zone. The master-file text format is a boundary format for the secondary, never the format you edit in.
  • delegation asks the parent zone's nameservers whether the delegation has settled, and exits non-zero until it has. It needs no resident, no keyfile and no database, so you can run it from anywhere while you wait on a registrar. --wait polls. It asks nameservers on port 53 and nowhere else. The zones it walks are somebody else's, so a nameserver whose address lies in a private or other special-purpose range is skipped unless net.dial-allow allows that range for the client delegation-probe. --allow-range names the allowance for one run instead, in the same client=range form: a lab whose simulated internet sits in 192.168.140.0/22 passes --allow-range delegation-probe=192.168.140.0/22. The check holds that value to the net.dial-allow rules before it asks any nameserver: a member they refuse stops the run with exit status 2 and an error naming --allow-range, the member and why.
  • import loads an RFC-1035 zone master file as the named zone, committing it atomically; a malformed master is refused with a non-zero exit and changes nothing, so a bad file never half-lands. Re-importing an origin replaces its zone (advance the SOA serial).
  • export writes the zone's master text (to --out, or standard output). It defaults to the durable, re-importable master: the published records, the faithful thing to archive or re-load. Pass --full for the whole serving set, which also includes any transient turn-up records (e.g. ACME challenge records) present at that moment.
  • delete removes the whole zone (apex-SOA removal is whole-zone removal).
  • secondary enrols a downstream secondary for a zone: it lands the allowlist row for the address the secondary pulls from, mints a TSIG key, and prints the secret and a BIND snippet to paste on the secondary. Over the owner session the key is minted only after the node has accepted the enrolment, so a refused, unserved, or unreachable node leaves no secret behind. With --offline it is minted only after operator state has recorded the row, so a write that fails leaves the key already in custody under that name as it was. --existing-key mints nothing: it lands the allowlist row for a key already installed on both sides, and prints no secret. Over a pinned node (--node or VALIS_NODE) only --existing-key applies. The minting form and --offline are both refused with exit 2, before any setting is read, anything is minted, or anything is dialled. A key minted on your machine stays in your machine's custody, where the node cannot use it to authenticate a transfer, and --offline writes your machine's operator state, never the node's. To mint a new key, run the verb on the node's host. ⚠ The gap. No door on the node mints a transfer key for an existing zone yet, so a new key for a pinned node is minted only on its host.
  • load and secondary –offline are the two routes that do not need a running resident. They write operator state directly, reading no keyfile and no management endpoint, with the connection coming from the node's operator-state.dsn setting (or, when the address carries a password, from VALIS_PG_DSN) and never from the command line. Reach for them on a node that is fresh or recovered and not answering yet. ⚠ --offline is what selects that route. secondary without it drives a running resident over the fabric, exactly like the owner-keyed verbs above, so on a node that is not serving yet it refuses and tells you to add the flag. load takes no such flag and is always direct. Either way secondary records the allowlist row; without --existing-key it also mints the TSIG key and prints the BIND snippet to paste on the secondary. Both are walked through in first boot and operator moves.

A committed change is picked up by the running resident without a restart: the serving side re-reads the store, so the node begins answering the new data authoritatively.

At go-live. After a fresh node is up and healthy (serving :53, restore-proven per the go-public gate above) the deploy recipe imports the node's own zone before the address is cut over: whichever domain that node serves, with its apex SOA, its apex NS, and the ns1 address record. The recipe runs valis zone import as the account that can read the 0600 keyfile, then confirms the node answers the zone over loopback (dig against :53) and that valis zone export round-trips before declaring the node live. Zone master files are operator data staged onto the host, not part of the delivery binary.

12. Publishing to a node, and updating one

Neither of these takes the node down, and neither wants you editing files on the host.

Publishing a page. publish places a publication into /pub on a running node, owner-keyed like the zone verbs. It is an ordinary operator act at any time, not a deploy-time one, and you run it from your own machine and name the node:

valis --node NODE publish --slug notes --file notes.html --content-type text/html

Its whole form is:

valis publish --slug <name> --file <path> --content-type <type>

All three are required. The slug is lower-case letters, digits, and hyphens, and --file names a file on the machine you run the verb on, placed on the node byte for byte. A slug that already exists is revised, not refused, and the node serves the new text at once. With a key the node has not enrolled, the verb is refused with exit 2 and nothing is placed.

⚠ The gap. No verb retracts a publication. A page you publish stays served until you revise it, so publish only what you mean to leave up.

From the node's host, the break-glass route. When you cannot use the route from your machine (the owner port closed, the pin or the device key lost, or the node not answering on the owner port), run publish on the node's host, inside the node's namespace with its keyfile and endpoint (see the break-glass route). The file must then be on the host, readable by the run account:

sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
  /opt/valis/bin/valis publish --slug notes --file /var/lib/valis/notes.html \
  --content-type text/html \
  --keyfile /var/lib/valis/valis/keyfile \
  --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"

Updating the binary. apply takes a delivery archive that is already on the host, puts it in place, restarts the serving unit, and confirms the node still answers:

valis apply --artifact <path> --confirm-server <addr> --confirm-zone <fqdn>

It transfers nothing, so move the archive onto the host first. Without --confirm-server it reads VALIS_APPLY_CONFIRM_SERVER; without --confirm-zone it reads edge.domain in the configuration under your own account's data root, or VALIS_EDGE_DOMAIN in your shell, and when neither is set, the node's own edge.domain. When yours and the node's name different domains, the apply refuses and names both: that disagreement is exactly the wrong-node case confirmation is there to catch. Pass --confirm-zone to settle it; the flag always wins, since a node may confirm for a zone other than its edge domain. The report names the zone it confirmed and where that zone came from. There is no loopback default: name them or the apply refuses before anything moves. That refusal is the point. Exit 2 means it refused and nothing moved; exit 3 means it moved and could not confirm, which is the case that wants you looking. --dry-run walks it without touching the host.

By default apply places the serving binary alone. --with-resident and --with-steer are the explicit ask for the wider set, so the privileged parts of the delivery never move because you forgot they were in the archive.

An apply leaves the host agent's configuration store as it is, so a node whose owner port was open answers on it again once the unit is back, under the same DID, and you go on managing it from your own machine with the pin you already hold.

12.1. Letting apply read the node's settings

Before it moves anything, apply reads the node's settings so it can judge what the node will actually boot on. It does this by running its own binary as config list under the node's run account, the owner of the unit's StateDirectory, through sudo -n. It runs its own binary, never the one standing at the destination, because an older binary there may not know the config verb and would start a node instead. The read takes no lock and writes nothing.

Grant the deploy account that command in a file under /etc/sudoers.d/ (edit it with visudo -f):

<deploy-account> ALL=(valis) NOPASSWD: /usr/bin/env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config list

sudo matches the command word for word, so the grant fits only when:

  • you run apply as /opt/valis/bin/valis (a symbolic link to it is fine, since the binary names itself by its resolved path);
  • the unit's StateDirectory is valis, which apply turns into /var/lib/valis with no trailing slash;
  • the run account is valis.

Understand what this grant is before you add it. It is not a narrow read permission. apply needs /opt/valis/bin writable by the deploy account so it can replace the binary, so the deploy account can put any binary it likes at /opt/valis/bin/valis and run it as valis through this grant at once: no restart, no unit and no apply log stand in the way. In effect the grant lets the deploy account run code it chose as the node's run account, with everything that account can read and write, the node's store included. That is the same trust the deploy account already holds through sudo -n systemctl restart valis.service, which apply runs to restart the node, since a restart runs whatever stands in /opt/valis/bin, but this one takes effect immediately. Give it only to an account you already trust to deploy the node.

Through the grant the deploy account also sees every setting config list shows. A secret shows only as set or unset, but a personal setting shows its value: today that is acme.account-contact, the address the node's ACME account is registered to.

Without the grant, or when any of the three conditions above does not hold, the read is refused, apply checks the flags and the unit's environment alone, as it did before it could read the node, and its report says why the node's settings were not read. A refusal for a missing site fact says so too.

One gap remains: config list does not show which keys the node's first boot already settled from valis.env. A required key reset in the store after that boot reads as unset to the node, but passes apply's check when valis.env still names it. The check is never weaker than the environment-only check it replaced.

13. The module source root

module.source-root (seeded from VALIS_MODULE_SOURCE_ROOT) names the directory holding the bytes of every module the node admits. Each module lives in a directory named for the identity of its bytes, written once and never changed, and a new version of a module is a new directory beside the old one. Never edit files under it. Keep it apart from the directory that holds the owner keyfile, and keep it, and every directory above it, unwritable by other users: an install refuses a root that is not.

I made the root a fact you state rather than a place valis picks, because it is where the code a node compiles into itself comes from. It is a boot-site requirement, so it is checked at three points:

  • A resident whose configuration names no root refuses to boot, and the refusal names the key, and the variable that seeds it at a first boot. With a root named, the resident creates the directory at boot.
  • valis apply reads the node's settings, and the unit's environment, before it moves anything. When the root, or any other fact the resident needs to boot, is missing from both, the apply refuses with exit 2 and the node keeps serving what it served. Set it as the run account with valis config set module.source-root ... with the node stopped, or, before the node's first boot, add the line to /etc/valis/valis.env, then apply.
  • Provisioning renders the env file from the same requirements, so a provisioned node carries the root.

What a node does with its modules. A module recorded as installed is admitted again from its identity directory at every boot, and its publisher's and owner's signatures are checked again then. A credential that expired after the module was admitted still admits it; a revocation recorded since stops it from loading. While a node serves real names, because it declares an edge domain or is authoritative for zones held on someone's behalf, it refuses to record a module install: a recorded module is stood up at every restart, and nothing yet falls back from a module that fails to stand up. A module that cannot be admitted again at boot does not run: the node logs it by name with the reason, comes up serving everything else, and names it on its status line. A backup carries no module bytes, so a node restored with modules recorded comes up without each one whose identity directory is not back under its module source root. Put those directories back and restart the unit to bring them up.

To install a module into a running node, or to supersede the version it carries, use valis module install (Installing a module as the owner below). Two verbs let you check a module first, on the node's host. Neither binds a port or writes the node's stores, so either may run beside the running node:

  • valis module try-install /module/HEX installs the module into a process of its own, through the same gate a running node applies, reading the node's keys and stores, and records nothing. Run it as the run account on the node's data root, so it reads the node's configuration:

    sudo -u valis env XDG_DATA_HOME=/var/lib/valis \
      /opt/valis/bin/valis module try-install /module/<hex>
    

    Exit 0 means the module was admitted and loaded. Exit 2 means you gave the verb no designation or more than one; 3, the argument is not a module designation; 4, no identity directory holds that module; 5, the module was refused before any of its code ran; 6, the module source root or the node state the verb needs is missing or unusable, or something else stopped the verb, such as a scratch directory it could not make; 7, both signatures admitted the module and its load then failed, so some of its code may have run. The verb's last line on standard output begins valis-dry-run: and carries the same JSON verdict try-supersede prints, with phase set to new. That line is what a node reads when it tries an install first: before installing a module, a running node runs this verb in a child process of its own binary and goes on only when the verdict says the module loaded.

  • valis module try-supersede /module/OLD /module/NEW loads the old version as a restart would and then the new one as an install would, again in a process of its own. It prints one line beginning valis-dry-run:, carrying a JSON verdict that names what the new version redefined, and exits with the try-install status for the version named in the verdict's phase field. The node's log can write to standard output as well, so take the line with that prefix. A node runs this same check in a child process before it supersedes a running module.

Where a node compiles modules. A node compiles each module beneath module-fasls/ in its data root (/var/lib/valis/module-fasls/ on a packaged node), in a directory named for the module's identity, and compiles a library a module needs that the binary does not carry beneath module-fasls/deps/. Nothing is compiled into the run account's home. A node upgraded from an earlier release keeps that release's compiled output under module-fasls/, filed by each source's full path (.../var/lib/...); nothing reads it any more. To reclaim the space, stop the unit, remove module-fasls/, and start it: the next boot compiles its modules again.

A supersede is always tentative. A change which a running image cannot make in place is refused before any of it is compiled: a changed structure, macro, constant, type, or inline or ftype declaration, a removed file, a changed grouping form, or a change inside a top-level form the comparison compares only whole. The new version must then pass the child dry run and the node's own gate. Even then the old version's code can stay live: a closure that captured it keeps running it, and the scan for leftover code can miss some. So the node records its image as diverged from what a restart would build. Restart the unit when you need the running image to match its record.

13.1. Installing a module as the owner

valis module install asks a running node, as its owner, to install a module, or to supersede in place the version of it the node already runs. You name the module by its designation and say where its bytes are:

valis --node NODE module install /module/<hex>
valis --node NODE module install /module/<hex> --dist /path/to/release
valis --node NODE module install /module/<hex> --git REPOSITORY TAG

With no source, the node installs the bytes already in the module's identity directory under its module source root. --dist names a release directory on the node by its absolute path, and it must lie apart from the node's keys and data. --git names a third party's tagged release in a repository the node reaches over https or ssh. Bytes from either source land only when they are the module the designation names. The request carries no system name, vouch, or signature: you file publishers and grants through /cap/admit (Managing standing grants), and the module loads only when the node's two-signature gate admits it. I kept trust out of the request, so nothing a request carries can admit the bytes it names.

Run the verb from your own machine with --node, like every owner verb, or on the node's host by the break-glass route.

What the node does, in order. Each step that refuses has loaded nothing.

  1. When the node already carries the identity you named (loaded from its identity directory, recorded in its module manifest, and admitted in its vouch store), it answers already-current and does nothing else.
  2. It refuses the request when a newer instance of the node holds the instance fence (fenced), and when the node serves real names (serving-real-names).
  3. It fetches or copies the release you named, giving a fetch 300 seconds, and lands it only when its bytes are the identity you named.
  4. It takes the module change lock, so one change runs at a time; a request that waits more than 5 seconds for it is refused (change-in-progress).
  5. It tries the change in a child process of its own binary within 300 seconds: for a new module the try-install check above; for a module it already runs, the try-supersede check, once a comparison of the two versions finds no change a running image cannot make in place. It goes on only when the child loaded the module.
  6. It checks the fence again, then its gate checks the publisher's vouch and your grant and loads the module into the running image. It records the module in the module manifest, so a restart stands it up again, and puts any edge adapter a new module registered in front of the running edge, with no restart.

The reply. valis module install prints the node's reply as it came: a token on the first line, then KEY VALUE lines, system and designation first.

Token What it means What to do next
installed-serving The module was admitted, loaded, and recorded. Each adapter line names an edge adapter answering now, then its ports. Nothing. valis module show SYSTEM reads it back.
installed-unserved Admitted, loaded, and recorded, with nothing answering on the edge. The reason line says why: no-edge-adapter (the module registered none), edge-not-running (the edge binds it when it next starts), or attach-refused (the edge would not attach it). For attach-refused, read the journal line naming the adapter, below. Otherwise nothing.
installed-unrecorded Admitted and loaded, so it runs now, but its record could not be published. The reason line says why, and runs-now yes and after-restart absent say that a restart does not stand it up again. Put right what reason names, then restart the unit before you ask again. Until a restart the node refuses another request for that module as not-carried.
superseded The version the node ran, previous, was replaced in place, tentative until-restart. recorded says whether the module manifest names the new version. residue counts the old version's code the scan found still alive, or reads unscanned. Restart the unit when you need the image to match its record. With recorded no, a restart returns to the version the manifest names.
already-current The node already carries, records, and admitted that identity. Nothing changed. Nothing.

After installed-unrecorded the verb also writes valis: the module runs now and will not return after a restart on standard error.

Exit statuses.

  • 0: the node answered with one of the five tokens.
  • 1: a fault on this machine, or a reply this client cannot read.
  • 2: refused with nothing installed: a command line the verb does not take (printed with its usage), a refusal before your key was sent, a key the node gave no owner view, a node with no /modules door, or a node refusal, printed as valis: the node refused: and the node's text.
  • 3: the node could not be reached, or the session with it failed.
  • 7: the install failed and code of the module may have run: the gate admitted it and its load failed, or the node's text does not say how far it got.

When the session failed or a fault on this machine stopped the verb after the request was sent, the reply was lost, and a second line says valis: module: the request may have reached the node; run valis module list to see what it holds before you ask again. A reply this client cannot read prints only its own line.

Refusals. A refusal is an error beginning module install refused, and nothing of the module was loaded. Apart from a malformed request or a caller who is not the owner, neither of which the verb sends, the reason follows in parentheses:

  • not-a-designation: the request names no module.
  • not-a-source: the request names no source an owner may name.
  • no-source-root: the node has no module source root.
  • fenced: a newer instance of the node has taken over.
  • serving-real-names: the node declares an edge domain or is authoritative for zones held on someone's behalf. The text reads this node serves real names, by its declared names, and a module recorded here would be stood up at every restart with nothing to fall back to, and names which: by its declared names or by its owned zones.
  • fetch-timed-out: the fetch ran out of time.
  • not-the-requested-identity: the release holds other bytes.
  • staging-refused: the release could not be fetched or staged.
  • conflict: its identity directory already holds other bytes.
  • vouch-refused: the node refused the vouch the release ships.
  • bytes-not-found: no identity directory holds the module.
  • not-one-system: the identity directory holds no system definition, or more than one.
  • change-in-progress: another module change held the lock for more than 5 seconds.
  • dry-run-unavailable: the child could not be asked, or did not answer within 300 seconds.
  • dry-run-refused: the child answered that the module did not load.
  • install-refused: the two-signature gate refused it, and the text after the reason says why.
  • For a module the node already runs: not-carried (the running version was never admitted as a module, or an unrecorded change awaits a restart), same-identity, publisher-changed (the new version comes from another publisher), restart-required (a change a running image cannot make in place), and dry-run-failed.

A load that failed after the gate admitted the module is not a refusal: its error begins module install failed after loading, the image is recorded as diverged, and a later request for that module is refused as restart-required until you restart the unit.

The journal. The node logs each outcome, so journalctl -u valis.service shows what it did with a request:

module /module/<hex> landed at <module-source-root>/<hex>
module <system> installed as /module/<hex>
module <system> (/module/<hex>) installed for the owner: installed-serving
module <system> (/module/<hex>) is already current for the owner
module <system> superseded, tentatively until a restart, from /module/<old> to /module/<new>
module <system> superseded for the owner, from /module/<old> to /module/<new>

and at WARN, for what did not go as asked:

module <system> not installed from <directory>: <reason>
module <system> installed from <directory> and running, not recorded: <reason>
module <system> (/module/<hex>) installed for the owner and not recorded: <reason>
the request for /module/<hex> was refused (<reason>): <cause>
the module change for /module/<hex> was refused (<reason>): <detail>; nothing was loaded
the module change for /module/<hex> failed after the gate admitted it (load-failed): <detail>; code of the module may have run in this image
module adapter <name> was admitted and does not answer: <reason>

A supersede whose record could not be published logs its line with , not recorded at the end.

How long an install holds its session. The verb waits for the node's reply, and the node answers only once the change is done: up to 5 seconds for the module change lock, up to 300 seconds for a fetch, up to 300 seconds for the dry run, and then the compile and load, which have no bound. Keep the session open until the reply arrives.

From a 9P client. The door is /modules/install in the owner's session. Write the request as newline-framed lines, install, the designation, and then optionally the source (here, dist and the absolute directory on the next line, or git and the repository and then the tag), then read the reply, then clunk. The first read runs the request and the fid keeps its outcome. A clunk before any read still runs it, but the reply is lost. A session that is not the owner's finds no /modules.

⚠ Where this stands. A node that serves real names refuses every install (serving-real-names, above) until a node can fall back from a module that fails to stand up. Nothing retires an installed module or rolls one back on a running node. ubik does not yet compose this door, and an install from a second machine against a deployed node is not yet proven.

13.2. Reading the module set

valis module list prints one tab-separated line per module the node lists: system, recorded, loaded, divergence, and boot. valis module show SYSTEM prints the five files the node holds for that module, identity, generation, files, explain, and state, each name on its own line and its lines indented beneath it. Both read and change nothing, take --node like every owner verb, and exit as install does, without

The node serves these as read-only files under /modules/manifest/SYSTEM/, for the owner alone. There is one directory for each module the module manifest records, resident modules included, and one for each module the image loaded with no record naming it, as installed-unrecorded leaves one.

identity
the recorded designation, resident, or none.
generation
the generation the manifest pinned when it was published. It describes; nothing checks it.
files
the record's relative paths, one a line.
explain
which two signatures admitted the module, and when: publisher (the did:key its vouch is rooted at), owner-credential (per-module-grant or standing-grant), grant (the grant's hash), and admitted-at (a Unix time), each none when unknown, and publisher unreadable for a vouch that does not decode; or resident yes.
state
the running image beside the record: recorded; loaded, the designation the module was loaded from, resident, unknown, or no; divergence, the reasons the image's divergence record holds, or none; boot, ok, refused, or manifest-unreadable; and diverged, yes or no.

diverged yes means the running image is not what a restart would build, or the boot left the module out; divergence and boot say which. A boot refusal shows its kind alone, and its text stays in the journal and on the status line, since it can name a host, an address, a key, or a path. No file carries vouch text, grant text, or key material beyond the publisher's did:key. A manifest the node cannot read is an error beginning module manifest unreadable and naming the kind of failure, never an empty listing.

/modules is the module set: which modules run, from which bytes, and why. One module's settings are somewhere else, under /config/modules/SYSTEM/, which valis config reads and changes with --module SYSTEM (see Changing a setting). /modules sits in the owner's own session and never under the node root, so no grant, bearer name, or delegated token reaches it. It is one letter from /module/, the prefix designations are written under, and the two never meet: no directory is ever made at /module/.

14. Backup and restore

The backup-critical set is irreplaceable: the owner Ed25519 seed (0600), the store head, the store blocks the head's root tree reaches, the vouch and revocation stores, the grant audit trail beside them (see Managing standing grants), the owner keyring of enrolled device keys (see Enrolling and retiring an owner device key), the node's configuration (its own settings and every module's, under <data-root>/config/), and the ACME account and leaf keys. Losing the owner seed is losing the node's sovereign identity; there is no re-mint.

Run both verbs as the run account, never as root. The configuration directory is private to the account that owns it, and a backup reads the grant audit trail only when the account running it owns the file, so a root-run backup is refused at whichever of the two it reaches first, before anything is written. A restore run as any account other than the one owning --state-dir (or, for a directory not made yet, the nearest directory above it) is refused before any write, with exit code 2, because the node could not open the files it would land. A restore run as root into a directory that does not exist yet is refused the same way, since whatever it made would belong to root. The same holds for --acme-store-dir.

The valis binary carries the backup and restore verbs. Each seals or opens a single encrypted, self-contained artifact, not a copy of the whole StateDirectory. The passphrase is read interactively (or piped on standard input for an unattended run); it is never a command-line flag, so it never lands in shell history or /proc.

  • Back up. Seal the backup-critical set into one artifact:

    sudo -u valis env XDG_DATA_HOME=/var/lib/valis valis --backup --out /path/to/valis-backup.sealed
    # prompts: Backup passphrase:
    

    The artifact is small. It carries the node's configuration as written, so it carries what that configuration names: absolute paths such as the ACME store (acme.store-path, /var/lib/valis/acme/ on a standard install) and the module source root, and the node's own domains and addresses. It carries no database password: the configuration never holds one. A memory-hard key derivation (argon2id) turns the passphrase into the key, and authenticated encryption (ChaChaPoly) seals it: a tampered artifact is rejected rather than silently restored. Treat it as you would a private-key store, and store the artifact and the passphrase separately. An owner seed that is absent, or that is not exactly 32 octets, stops the backup with exit 1 before anything is sealed. A backup can also be driven remotely by the authenticated owner over the /backup/ctl 9P door (backup <passphrase> [name]; status reports the last outcome). The passphrase rides one ctl line over the sealed owner session (never cleartext on the wire, never shell history) and the artifact always lands in the node's backup directory, <data-root>/backups/ (/var/lib/valis/valis/backups/ under the shipped unit), under NAME or, with no name, a timestamped bundle. NAME is a bare file name: a path, a name starting with a dot, or a name already present there is refused with an error destination reply, so no session can aim a sealed copy of the seed elsewhere. To write the artifact to a path of your choosing, run valis --backup --out <path> on the host. A restore is deliberately not served over 9P: it requires the unit down and a pristine state directory, so it stays valis --restore on the host.

  • Restore. Open the artifact into a pristine state directory. When the state directory does not exist yet, make it as the run account first; starting the unit once also has systemd make it:

    sudo install -d -o valis -g valis -m 0700 /var/lib/valis
    sudo -u valis valis --restore --in /path/to/valis-backup.sealed --state-dir /var/lib/valis
    # prompts: Restore passphrase:
    

    The restore lands the configuration first, then places the ACME store where that configuration says the node keeps it. The ACME store must lie inside --state-dir: one the configuration places elsewhere is refused before any write, with exit code 2, so a restore never overwrites certificates outside its target. To restore somewhere else, such as a scratch directory for a proof, name where the ACME store lands with --acme-store-dir:

    sudo -u valis mkdir -m 0700 /tmp/restore-proof
    sudo -u valis valis --restore --in /path/to/valis-backup.sealed \
      --state-dir /tmp/restore-proof --acme-store-dir /tmp/restore-proof/acme
    

    The restored configuration still names the original ACME store, so a node you boot from a proof directory looks for its certificates there; the proof checks the keys under --acme-store-dir. A non-pristine target (one that already holds valis state, configuration, or an ACME store where this restore lands one) is refused up front, before any write, with exit code 2; a wrong passphrase or corrupt artifact fails cleanly with a single message and exit code 1. The normal resident boot then condenses the durable namespace from the restored head: restore is the first-deploy path run from a saved head instead of from genesis, and a faithful restore reproduces the origin's durable identity byte-for-byte. Deploy, evacuate, and restore are one condense mechanism, see the contract's deploy/evacuate/restore section.

  • The keyring on restore. A restore lands the owner keyring with its retirements, so a key retired before the backup was taken stays retired, and it prints every key it restores, live or retired, for you to review:

    valis: restored owner key did:key:z6Mk... label=laptop live
    

    A restore cannot know of a key retired after its backup was taken. Retire any such key again once the restored node is up. A backup taken before the node kept a keyring restores the seed alone, with a warning saying the node admits only its own key until you enrol your devices again. A binary older than the keyring refuses a backup that carries one, before writing anything. A keyring the backup cannot read whole, or one written for another owner, stops the backup before anything is written, and one a restore cannot land rolls the whole restore back.

  • The grant audit trail on restore. Backups carry the trail from container format 6. A restore lands it right after the revocation store and appends one restored entry, so the trail shows when the node came back. A backup in an older format restores with this warning, and the restored trail starts at the restore:

    valis: WARNING: this backup is in container format 5, which cannot carry the grant audit trail. The restored node's trail starts at the restore: when each grant was filed and reviewed before it is not known.
    

    Each active standing grant then has no known filing and no review, so each is due a review at once: review them again (see Managing standing grants). A target that already holds a trail is refused as not pristine unless the backup carries one, which then replaces it whole. When the restored entry cannot be written, the restore stands and a warning beginning the restore stands, but its restored entry was not written to the grant audit trail says so. A backup that cannot read the trail is refused before anything is written.

The restore-proven backup is the go-public gate, and nothing else enforces it. Do not cut a node over to a public address until a restore has been proven: restore the artifact into a scratch state directory and confirm the three properties a good restore demonstrates (the owner axis resolves, the store generation matches the source, the TLS keys load). Those properties, and why a silently-wrong restore is treated as worse than an obvious failure, are set out in backup-and-observability.org.

15. Restart, stop, and teardown

  • Restart is just re-running bring-up: it is idempotent. The single-writer instance fence makes a re-claim of the current generation a no-op, so recovery is safe to repeat.

    systemctl restart valis.service
    
  • Restart as the owner with valis restart, when you hold the owner key, or when the node has told you a restart clears threads a retired module left running. The node asks fulcrum to have systemd restart valis.service, so it is the same stop and start as systemctl restart, with the same few seconds without service. Run it from your own machine and name the node:

    valis --node NODE restart
    
    • It waits for the new boot. After it asks, the verb waits, 120 seconds by default (--wait SECONDS sets the bound), until the node answers again over the same pin from a boot identity other than the one it had when asked. It then prints valis: the node is back, boot with the new identity, and exits 0. The owner port keeps its address across the restart, so the wait needs no shell on the host.
    • Every owner session ends. The restart ends every session the node held, including ubik's. Each valis verb opens a session of its own, so the next one you run attaches to the new boot. ubik attaches again on its own once the node is back, checking the pin as it does, and every panel reads the node again. When you restart the node from ubik's restart panel, the panel shows the node's boot identity before and after; Managing a node from ubik covers the panels.
    • Exit statuses. 0: the node came back from a new boot, or you passed --wait 0 and the restart was accepted. 2: the node refused, or the client refused before asking (for example, for a key the node has not enrolled), so nothing was asked. 3: the node could not be reached to ask, or did not come back within the wait; the message says which.
    • A refusal says why. A node running without a supervisor that can restart it, such as a development image or --run, says so and tells you to restart it the way it was started. When fulcrum is too old to serve the restart request, the refusal says it cannot restart the node: restart the service on the host with systemctl restart valis.service.
  • Restart as the owner from the node's host, the break-glass route, when you cannot use the route from your machine (the owner port closed, the pin or the device key lost, or the node not answering on the owner port). Run the verb inside the resident's namespace as the run account (see the break-glass route), and confirm from the host:

    systemctl show -p MainPID -p ExecMainStartTimestamp valis.service
    sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
      /opt/valis/bin/valis restart --wait 0 \
      --keyfile /var/lib/valis/valis/keyfile \
      --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
    systemctl show -p MainPID -p ExecMainStartTimestamp valis.service
    
    • Pass --wait 0 from inside the namespace. A restart replaces the namespace and draws a new fabric port, so a verb entered into the old namespace, or pinned to the old port with --endpoint, can never see the new boot and would wait out the whole bound. With --wait 0 the verb asks, prints the boot the node answered from, and exits.
    • Confirm on the host. MainPID is fulcrum, which every restart replaces, so a different MainPID and a later ExecMainStartTimestamp after the restart than before it say the node was restarted. Run any owner verb again through a fresh nsenter to see it answering. Where the node's owner port is open and you name it with --endpoint, the address survives the restart and the verb's own wait works, with no root shell at all.
  • Stop drives an ordered teardown. On SIGTERM the resident unwinds in reverse bring-up order (long-lived active modules drain first, then edge, then fabric, then listener); fulcrum (the MainPID) removes the sk_lookup link and tears the netns down. KillMode=mixed lets fulcrum drive that ordered pair-teardown rather than systemd SIGTERMing both at once.

    systemctl stop valis.service
    
    • This stop does not depend on the stop notice. systemctl stop signals fulcrum first, and fulcrum ends the run on that request and reaps the resident. The resident gives its stop notice as it does on any orderly stop, but the outcome does not depend on it, so a warning about the notice during this stop needs no action. The same holds for an owner restart, which goes through fulcrum and systemd.
  • A signal sent to the resident's own process is a stop the resident begins itself: SIGTERM or SIGINT delivered to the resident directly, for example kill with its PID. The bullets below cover this stop only.
    • The resident tells fulcrum the stop is intended. Before it stops anything, the resident sends fulcrum a stop notice over its control session, and it closes that session last. Once the session ends, fulcrum waits up to 70 seconds for the resident to exit and exits with the resident's status; a resident still running at 70 seconds is killed and fulcrum exits 1. The journal shows the host agent was told this stop is intended at info level.
    • A warning about the notice means the stop can read as a failure. An older fulcrum has no file to take the notice, and the resident warns the host agent serves no stopping file; update fulcrum to clear it. Any other warning about the notice (a request still held the session, the session had ended, or fulcrum refused or did not acknowledge it) means the notice did not land this once. Either way the stop itself carries on, and fulcrum gives the resident only 2 seconds from the session's end: a resident that exits in that time has its status passed on, and one still running makes fulcrum exit 1.
  • A superseded writer fails closed. Two instances sharing one operator-state database coordinate only through the single-row write-fence; a superseded writer is fenced out (the fenced-out condition) and writes nothing. This is the split-brain stop that makes a cross-location handover safe. Expect the older instance to fail closed once a successor claims a higher generation.

16. Legacy from-$HOME operator mode

The default run account is a dedicated static valis:valis. A legacy operator running the resident out of their own account is supported via a drop-in: install legacy-operator.conf.example as /etc/systemd/system/valis.service.d/legacy-operator.conf and set the <operator> / <operator-home> placeholders:

install -d -m 0755 /etc/systemd/system/valis.service.d
install -m 0644 deploy/systemd/valis.service.d/legacy-operator.conf.example \
        /etc/systemd/system/valis.service.d/legacy-operator.conf
# ... edit <operator>/<operator-home>, then ...
systemctl daemon-reload

Overriding User=/=Group and pointing XDG_DATA_HOME (and HOME) at the operator's home keeps valis rooting durable state there; on the first boot after such a move, the resident relocates pre-XDG ~/.valis state cleanly. To fall all the way back to ~/.local/share, clear the inherited StateDirectory (an empty StateDirectory=).

17. Troubleshooting

Symptom Cause / fix
Error opening shared object … libfixposix.so iolib dlopen=s the *unversioned* SONAME. Install the runtime package *and* create the =libfixposix.so symlink (or install -dev).
c3po's serving codec … not available at boot a plain :valis core was staged. Stage the serving-capable core from the make dist delivery tarball instead.
:443 dark unexpectedly check edge.domain is set (v get edge.domain, with the v wrapper from changing a setting) and a cert is in the custody store; read the boot line for which of the two. A genesis dark :443 is healthy.
setpriv: … Operation not permitted / no privilege drop Under NoNewPrivileges the execed setpriv helper keeps only the ambient caps, so emptying AmbientCapabilities breaks the privilege drop outright. Do not narrow the ambient set.
DNS boot cannot build its zone index Postgres must be reachable over its local UNIX-domain socket under peer auth: verify the socket path in operator-state.dsn (valis config get operator-state.dsn, in the running-node or stopped-node form changing a setting gives) exists and the run account's uid maps to the database role.
Type=notify times out despite the resident appearing to run NotifyAccess=all is required (valis is fulcrum's fork-child, not MainPID); and the launch chain must not clear the environment, or $NOTIFY_SOCKET never reaches valis.
unit restarts but never opens :443 after a cert renewal renewal hot-swaps in place through the shared cell: a restart is not required and not the renewal path. Check the renewal manager's custody-store writes.
new connections are refused while the unit stays active (running) and healthy descriptor exhaustion, which no instrument on the node reports: an accept loop that cannot accept is still an alive thread, so the liveness gate holds and the status line still says serving. Count /proc/<pid>/fd against Max open files in /proc/<pid>/limits for the resident, per descriptor headroom above.

For the architectural why behind each of these, the privilege split, the descriptor handoff, the sandbox directives that ship vs. those deliberately omitted (MemoryDenyWriteExecute, LockPersonality), read the annotated valis.service unit and the host-deployment contract.