valis / Running a node
valis Operations Runbook
This is the hands-on companion to the host-deployment contract. The contract states what a host must provide and the fail-closed order a node comes up in; this runbook states what the operator does to install, run, observe, renew, back up, and restart a deployed node. Where the contract is the invariant, the runbook is the procedure.
Audience. The operator running a deployed unit, not the developer building valis from a REPL. For the inner development loop see DEVELOPMENT.org; for building your own system on the substrate see BUILDING.org.
The shape of a running node. One Type=notify systemd unit supervises the
fulcrum=+=valis process pair. The privileged fulcrum parent creates the network
namespace, binds the public sockets its launch configuration names (:53 when
:designated-ports includes 53, the TLS edge when the configuration names an
:edge-port, and the owner port when it sets :open-owner-port), loads the sk_lookup
director, and forks the unprivileged valis resident after a setpriv drop. The
director funnels every inbound TCP connection in the node's network namespace, loopback
included, into valis. valis adopts the inherited descriptors, brings its fabric and
edges up fail-closed, and emits READY=1 once every bring-up step has returned.
Readiness attests that the fabric, the edge, and the active modules came up without
error; it does not attest that anything has been served, and what keeps the unit alive
afterwards is a liveness check on those threads rather than a check that traffic is
moving. The unit is the supervisor of last resort: it restarts the whole image if the
image dies; the active-module seam inside the image supervises the modules.
1. Host filesystem layout
The shipped unit expresses an FHS layout the launcher threads into valis's XDG rooting. Provision these before first start:
| Path | Role | Ownership |
|---|---|---|
/opt/valis/bin/ |
the staged delivery binary (fulcrum-resident), world-traversable |
root, o+x chain |
/etc/valis/ |
the bootstrap EnvironmentFile seed only (read-only to service) |
root:valis 0750 |
/var/lib/valis/ |
StateDirectory: durable irreplaceable state (owner seed, store, ACME custody, the node's configuration under valis/config/) |
valis:valis 0700 |
/run/valis/ |
RuntimeDirectory: the AF_LOCAL control socket + pidfiles |
auto, ephemeral |
StateDirectory holds the irreplaceable durable state. systemd creates and chowns it
on first boot; the parent's CAP_CHOWN=/=CAP_FOWNER make the owner-seed write under
it succeed. Everything backup-critical lives here, and the valis --backup verb seals
the backup-critical subset out of it, see 14.
The binary must be the serving-capable delivery core, not a plain :valis core: the
plain core lacks the additive DNS-serving stack and fails closed at boot with c3po's
serving codec … not available. Stage it from the versioned delivery tarball that
make dist produces (valis-serving-<version>.tar.gz); never rebuild the core on the
host. See the contract's
serving-capable resident binary section.
2. Installing the unit
The units ship under deploy/systemd/. Install them onto the host:
# The supervisor unit.
install -m 0644 deploy/systemd/valis.service /etc/systemd/system/valis.service
# The bootstrap environment seed. Copy the template, fill it, lock it down.
install -d -m 0750 -o root -g valis /etc/valis
install -m 0640 -o root -g valis deploy/systemd/valis.env.example /etc/valis/valis.env
# ... edit /etc/valis/valis.env (next section) ...
systemctl daemon-reload
The ExecStart is /opt/valis/bin/fulcrum-resident, the fulcrum delivery binary
that owns the whole bring-up. Stage that binary and the run account per the contract's
provisioning surface before enabling the unit.
3. The environment seam: /etc/valis/valis.env
/etc/valis/valis.env is the node's genesis seed. The node reads each value in it
once, at the first boot that considers that value's key, and keeps the result in its
configuration store under the data root: <StateDirectory>/valis/config/, which is
/var/lib/valis/valis/config/ under the shipped unit. From then on the store is the
node's configuration. Change a setting with valis config or through /config over the
owner session (see changing a setting). Editing valis.env
afterwards has no effect on a key the node has already considered, and the boot log
names each such key as superseded by the store, by name and never by value. A key that
a newer release adds is seeded from this file once, at that release's first boot.
The file is re-authored by provisioning, is 0640 root:valis, and never enters the
tracked repo. Its values travel nowhere: the node's configuration directory is what a
backup carries (see backup and restore).
valis apply checks the site facts a boot needs against the node's own configuration
first and this file second, the order the node reads them in. Without
--confirm-zone it confirms the node's own edge.domain, unless you have set one of
your own (see below). When apply cannot read the node's configuration,
it checks this file alone and says so; see letting apply read the node's settings.
Each variable below seeds the configuration key beside it:
| Variable | Configuration key | Meaning / effect when unset |
|---|---|---|
VALIS_PG_DSN |
operator-state.dsn |
operator-state Postgres URI over the local UNIX-domain socket under peer authentication, e.g. postgresql://valis@/valis_state?host=/var/run/postgresql. No password, no TCP listener: the kernel proves the run account's uid and Postgres maps it to the role. The configuration store refuses an address that carries a password, and says why, so no secret reaches it or a backup; use the socket form. An address with a password stays in the environment, is read from there at every boot, and the boot logs that the key is held in the environment because it carries a secret. |
VALIS_EDGE_DOMAIN |
edge.domain |
the DNS name the :443 edge serves a certificate for. Unset ⇒ :443 stays dark (fail-closed: no domain named). Naming it is necessary and not sufficient: the launcher config must also name an :edge-port, or no socket is bound for the edge and the firewall accepts none. |
VALIS_ACME_PRODUCTION |
acme.production |
truthy ⇒ the real Let's Encrypt production CA (strict rate ceiling). Unset/empty ⇒ staging (the fail-safe default; keeps dev/CI off production). |
VALIS_ACME_DIRECTORY_URL |
acme.directory-url |
override the ACME directory (a Pebble/CI or alternate staging endpoint). Honored only when the production opt-in is not set: production always wins. |
VALIS_ACME_STORE_PATH |
acme.store-path |
the custody store the :443 edge loads its cert from and the renewal manager writes into. In the shipped unit, under StateDirectory, e.g. /var/lib/valis/acme. Unset ⇒ the resident refuses to boot, on every boot including a genesis one: an unchosen store root would mint a fresh ACME account against the CA and spend registration budget nothing gives back, so it is required rather than defaulted. The shipped valis.env.example already carries a value. A first boot without it settles the key empty, so editing valis.env afterwards has no effect: once a boot has refused for this, set it with the node stopped, as the run account: sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config set acme.store-path /var/lib/valis/acme. |
VALIS_ACME_ACCOUNT_CONTACT |
acme.account-contact |
the operator's ACME account contact, the address this node's ACME account is registered to. Unset/blank ⇒ automatic renewal refuses rather than ordering, and says so on the boot line: a contact is an identity, and one invented for you would register the account to an address you never chose. A first order can carry a contact typed at valis obtain --contact; a renewal falls due months later with nobody at the keyboard, and this is where it reads one. |
VALIS_MODULE_SOURCE_ROOT |
module.source-root |
where the bytes of every module this node admits are kept, one directory per module identity. In the shipped unit, under StateDirectory, e.g. /var/lib/valis/modules. Unset means the resident refuses to boot, and with it set the resident creates the directory at boot. The shipped valis.env.example already carries a value. A first boot without it settles the key empty, so editing valis.env afterwards has no effect: once a boot has refused for this, set it with the node stopped, as the run account: sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config set module.source-root /var/lib/valis/modules. See the module source root. |
VALIS_EDGE_DOMAINS |
edge.domains |
further names declared primary for this node, comma or space separated. They add to edge.domain and never replace it, and the edge still needs that base name to open :443. Unset means the base name alone. |
VALIS_MAIL_LOCAL_DOMAINS |
mail.local-domains |
the mail domains this node delivers locally, comma or space separated. The value is carried in the configuration, but no part of the running node reads it yet. |
VALIS_MAIL_SECONDARY_FOR |
mail.secondary-for |
the domains this node is backup MX for, as domain=primary-host tokens. A list with a member naming no primary host is refused whole. Unset means the node is backup MX for none. |
VALIS_MAIL_ACCEPT_LOCALPARTS |
mail.accept-localparts |
the local parts accepted at RCPT, comma or space separated. Unset leaves the built-in accept list in force; it never widens acceptance to a catch-all. |
VALIS_MAIL_RESOLVE_DOT_HOST |
mail.resolve-dot-host |
the DNS-over-TLS upstream that outbound mail resolves recipient MX records through. Unset means every relay is deferred, because there is no smarthost to fall back to. |
VALIS_MAIL_RESOLVE_DOT_ADN |
mail.resolve-dot-adn |
the name that upstream's certificate is verified against, which is also the SNI sent to it. |
VALIS_MAIL_RESOLVE_CA_FILE |
mail.resolve-ca-file |
the CA bundle that verifies that upstream. Unset is an empty trust store and not a bypass: every peer is rejected, so mail defers until you name one. |
VALIS_TRANSFER_MASTER_ADDRESS |
names.transfer-master-address |
this node's public authoritative DNS address: the source of outbound NOTIFY, the address a secondary pulls from, and the address a new domain resolves to when you name no other. zone create, and zone secondary without --offline, ask the running node for it over the owner session when no flag names the address. Unset means those verbs refuse rather than pick an address. |
VALIS_SECONDARY_NS |
names.secondary-ns |
the nameserver name published beside this node's own in every domain's apex NS set. Unset means the apex names this node alone. |
VALIS_SECONDARY_TRANSFER_ADDRESS |
names.secondary-transfer-address |
the address the secondary pulls zone transfers from. zone create enrols that address when you pass no --peer. Unset means such a create refuses. |
VALIS_DEPENDENCY_DIST_URL |
dist.url |
the private dist valis dist register registers as the node's dependency source, when --location is not given. Unset means the verb refuses with exit 2. |
VALIS_DEPENDENCY_DIST_HOME |
dist.home |
where valis dist register records the registration, when --home is not given. Unset means valis/dependency-dist/ under the data home. |
VALIS_NET_DIAL_ALLOW |
net.dial-allow |
the special-purpose address ranges each client may reach when somebody other than you chose where it connects, as client=range members with one range to a member and a client repeated for several, e.g. delegation-probe=192.168.140.0/22. A range lying wholly inside the floor (the unspecified addresses, multicast and limited broadcast) is refused, because nothing ever reaches the floor. A range with a bit set past its prefix length is refused too. A list with one refused member is refused whole. An allowance stored under looser rules than the node now applies allows nothing, and the node logs config.allowance-refused once, naming net.dial-allow and why. Set it again under the current rules to restore it. Unset means no client is allowed any such range. A running node reads it as it makes each such connection, so a change applies to the next one. valis zone delegation reads it as well, for the client delegation-probe, unless you pass --allow-range, so the example lets that check ask nameservers in a lab's private network. A module you admitted is a client under its ASDF system name: example-client=10.0.0.0/8 lets the module admitted as example-client reach that network, and no other module. The allowance follows the system name, whichever vouched module carries it: a replacement, or another publisher's module admitted under the same name, inherits it. A module whose system name a client name cannot carry (anything but lower-case letters, digits, dots and hyphens, or a first character that is not a letter or a digit) can be allowed nothing, and neither can one named as one of the node's own clients (acme, delegation-probe, dist, secondary-forward, zone-notify). |
VALIS_OWNER_PORT_SESSIONS_PER_ORIGIN |
owner-port.sessions-per-origin |
how many connections the owner port holds at once from one origin, counted from the moment a connection reaches it, handshake included. Unset means 4. A whole number from 1 to 63. One origin is one IPv4 address or one IPv6 /64, and an IPv6 address carrying an IPv4 one counts as that IPv4 address, so machines behind one public IPv4 address count as one origin. A refused connection is logged at WARN at most once per origin per minute. A running node reads the setting at each connection, so a change applies to the next one. |
VALIS_OWNER_PORT_SESSION_CEILING |
owner-port.session-ceiling |
how many connections the owner port holds at once from every origin together. Unset means 16. A whole number from 1 to 63, which leaves at least one of the 64 connections every steered port shares to the other ports. A running node reads the setting at each connection, so a change applies to the next one. |
VALIS_MODULE_STANDING_GRANT_REVIEW_DAYS |
module.standing-grant-review-days |
how many days an active standing grant may go without a review before the owner's grant inventory, /cap/grants, marks it review due, counted from the later of its filing and its latest review. Unset means 90. A whole number from 1 to 3,650. The mark is advisory: nothing stops a module being admitted under a grant that is due. While any active standing grant is due, the node's status line ends by counting them and naming the interval. A running node reads it each time the inventory is read and each time the health loop refreshes the status line, every 15 seconds, so a change applies at the next of those. |
VALIS_SLYNK_PORT |
development.listener-port |
⚠ A trap: do not set it on a new node. It names a port for the development listener, which opens only when the node starts: a change takes effect at the next restart. A node has it only if you set it there by hand. Nothing provisions it: the template leaves it out, and a node condensed from genesis, or recondensed onto a new host, comes up without it. On a node whose director is attached, the boot line shows the listener as listening while no connection can reach it, because the director funnels every inbound TCP connection in the node's network namespace, loopback included, into valis's single listening socket. The listener's destination is a unix domain socket; see the listener's destination. |
The database DSN names a local UNIX-domain socket, so the resident reaches Postgres
by filesystem path under peer authentication rather than over the network: no password
travels in the environment or the configuration, and no TCP listener need be open.
Put the socket-path DSN in valis.env for the first boot, and change it afterwards
with valis config set operator-state.dsn in the form
changing a setting gives. (Moving the database onto a
dedicated address inside the netns is deferred hardening; the host is the root of trust
for the resident copy.)
The tools you run read seven more variables, from the environment you run them in. They are not node configuration and have no configuration key: the node never
reads them, and they do not belong in valis.env.
| Variable | Meaning / effect when unset |
|---|---|
VALIS_KEYFILE |
the owner keyfile the owner-keyed verbs present, when --keyfile is not given. Unset means the keyfile in the data root of whoever runs the verb. A path only: the key bytes never cross the command line. The file must be a regular file, not a link, owned by the account running the verb, private to it, and exactly the 32 octets of a seed; anything else is refused. The key named is the node's own, so the verb accepts only that node as the responder: one that cannot prove it holds the key is refused. |
VALIS_MGMT_ENDPOINT |
the management endpoint, as HOST:PORT, the owner-keyed verbs connect to when --endpoint is not given. Unset means the endpoint file the resident writes. The port changes at every boot, so read it fresh rather than setting this once; see the break-glass route. |
VALIS_NODE |
the pinned node the owner-keyed verbs reach, by its name in the pin store, when --node is not given. Unset means no node is named, and the verbs use the keyfile and the endpoint. Under a node, VALIS_KEYFILE and VALIS_MGMT_ENDPOINT are not read, and a --keyfile or --endpoint flag is refused rather than dropping the pin. |
VALIS_KEY |
the device key presented to a pinned node, when --key is not given. Unset means this machine's device key, the one valis key generate writes under $XDG_CONFIG_HOME/valis/, or under ~/.config/valis/ when XDG_CONFIG_HOME is unset or not an absolute path. A device key is presented only to a pinned node, so VALIS_KEY or --key with no node named is refused. |
VALIS_APPLY_CONFIRM_SERVER |
the server valis apply queries to confirm the node is serving, when --confirm-server is not given. There is no default: with neither, the apply refuses before anything moves. --confirm-zone falls back to edge.domain in the configuration under the invoking account's own data root, or to VALIS_EDGE_DOMAIN in the invoking shell, and then to the node's own edge.domain; when your value and the node's disagree, the apply refuses and names both. |
VALIS_APPLY_TARGET |
the directory valis apply replaces the serving binary in, when --destination is not given. Unset means /opt/valis/bin/. |
VALIS_APPLY_UNIT |
the unit valis apply restarts, when --unit is not given. Unset means valis.service. |
One more is read only on the host that builds the binary, while it compiles.
No node reads it, and it does not belong in valis.env either.
| Variable | Meaning / effect when unset |
|---|---|
VALIS_BAKE_MANUAL_DIR |
the staged manual the binary carries, a directory make site site-stage wrote. make dist sets it for you, to the manual it built from the commit it releases, so a fresh node serves the whole manual before anything has been published to it. Unset means the binary carries no manual and a fresh node serves the landing page alone. A directory that is missing, holds no pages, holds a page whose name cannot be a publication slug, or holds no front door stops the build. |
4. Changing a setting
valis config reads and changes the node's settings and any module's. Against a
running node it goes through the node's owner session, so from your own machine you
name the node, as you do for every owner verb (see
Reaching a node's owner control plane):
valis --node NODE config list # every node key: path, origin, type, when it applies, value
valis --node NODE config list SYSTEM # a module's keys, and what an upgrade kept aside
valis --node NODE config get edge.domain
valis --node NODE config set edge.domain example.org
valis --node NODE config reset edge.domain # back to the shipped default
valis --node NODE config menu # walk the node's settings, one at a time
valis --node NODE config interview SYSTEM # a module's own questions; a blank answer keeps a value
valis --node NODE config set --module SYSTEM KEY VALUE # a module's key
valis --node NODE config purge SYSTEM # delete a removed module's kept settings
config get prints one line of five fields separated by tabs: the key, where its value
came from (default or operator), its type, when a change to it applies, and the
value, which is empty when the key is unset. A value set equal to the default reads
back with the origin default. A change answers with when it takes effect, described
below, then key KEY and origin ORIGIN. With a key the node has not enrolled, the
verb is refused with exit 2 and changes nothing.
From the node's host, the break-glass route. When you cannot use the route from your machine (the owner port closed, the pin or the device key lost, or the node not answering on the owner port), run the verb on the node's host the way you run the other owner-keyed verbs there, inside the node's namespace with its keyfile and endpoint named (see the break-glass route):
v() {
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis config \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)" "$@"
}
v list
v get edge.domain
v set edge.domain example.org
On the host, reading the node's own settings does not need the owner session.
config list and config get KEY, run as the run account on the node's data root, read
its store directly, running or stopped. The read takes no lock and writes nothing, so it
never delays the node:
sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config get edge.domain
I made reads work this way because a running node's owner session lives inside its
network namespace, out of reach of the host's, and reading a setting should not need
nsenter and root. A module's settings are still read over the owner session, since
only a node that has loaded the module can list them.
Against a stopped node, run it as the run account on the node's data root, and it changes the node's own settings directly:
sudo -u valis env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config set edge.domain example.org
A module's settings change only through the running node, since only a node that has
loaded the module can check a value for it. A secret is only ever shown as set or
unset. Run against a stopped node as any account but the one owning its data root,
the verb is refused with exit 3, as a configuration that could not be changed. Run
without --keyfile and --endpoint, it acts on the data root of the account running
it, so under sudo alone it reads and writes root's own store and not the node's.
A change answers with when it takes effect:
committed-applied: the value is stored and the running node is using it.committed-on-restart: the value is stored and the node takes it at the next restart. ⚠ This is a success, not a fault: some keys, such as the development listener port and the ACME store, only ever take effect at a restart, and every change to a stopped node answers this way.committed-rebind-failed: the value is stored, the running code refused it, and areasonline names the kind of condition. The next restart applies the value.
A purge answers purged or nothing-to-purge, and a purge of a loaded module also
answers when its return to the defaults takes effect, as one of the outcomes above. A refusal names what was wrong and
changes nothing. The exit status is 0 when the verb read or committed, 2 when the node
or the verb refused and nothing changed, which includes a key file that is missing or
one the verb will not present, and 3 when the node could not be reached or its configuration could not be
changed.
The configuration directory is part of every backup, so a restored node comes back with the settings it had.
4.1. The served set
The ports a node serves are one setting, edge.served-ports, and you read and change
them from your own machine like any other. Read the set first:
valis --node NODE config get edge.served-ports
The value is the ports in ascending order, separated by commas: 53,80 on a node that
kept the default core. Change it with config set and give the whole set. The value
you set replaces the set, so a port you leave out stops being served. To add 8099 to
that set:
valis --node NODE config set edge.served-ports 53,80,8099
valis --node NODE config get edge.served-ports
Keep every member you read unless you mean to stop serving it. A set without 53 stops
DNS, and an empty set is a choice the node keeps: it serves nothing, and a restart does
not bring the default core back. The change answers committed-applied and takes
effect without a restart. To put the set back, set it to exactly the value you read
first.
⚠ committed-applied says the node stored the set and took it into the running set.
It does not say the host agent's firewall admitted the port. Read that from a machine
other than the node's host with scripts/owner-port-reach.sh and both controls, as
From another machine describes, here with a
port the node admits and a port outside its set:
scripts/owner-port-reach.sh --address 192.0.2.10 --port 8099 \
--open-control 80 --closed-control 8100 --expect open
Read it with --expect closed before the change, with --expect open after it, and
with --expect closed again once you have put the set back. On a node whose director
is attached, a port in the set completes a connection even while nothing answers on
it, because the director carries every connection the firewall admits to the node's
one listener; the reading then shows the firewall admitting the port and nothing more.
On a node without the director, a port with nothing listening on it is refused
instead, and the probe reads it as refused, not open.
⚠ The gap. The node also takes one port at a time through /edge/served/ctl, which
declares or undeclares a single port and reports whether the host agent admits the
set, but the valis command line has no verb for that door. A 9P client composes the
request itself, and from the command line you change the whole set through config.
5. Enable, start, and check status
systemctl enable --now valis.service # enable at boot + start now
systemctl status valis.service # unit + MainPID (fulcrum) state
journalctl -u valis.service -f # follow the bring-up + serving log
Readiness is meaningful, and it is reported in two places that are not the same
place. The bring-up writes its account to the journal, and the resident separately
publishes a one-line summary to the supervisor, which systemctl status shows as
Status:. Neither carries the other's text, so a phrase you cannot find in the journal
is very likely a status-field phrase, and looking harder will not turn it up.
In the journal (journalctl -u valis.service), the bring-up says what it did:
site configuration complete: every declared site fact was answered, and the count of facts checked follows on the same line. A boot missing one stops here instead, naming the variable it wanted.adopting the DNS descriptor pair: the DNS handoff took; valis is serving authoritative DNS over fulcrum's inherited:53descriptors, binding nothing privileged itself.the launch handed down the DNS descriptor pair ... and 53 is not configured: the pair arrived, but this node's served set leaves DNS out, so valis closes it once the fabric is up. DNS does not answer on this node; this is not an adoption.:443 public edge opening — certificate in custody for <domain>: a cert was found for the configured domain and the edge is coming up.no usable certificate in custody for <domain>; :443 public edge cert-gated dark: the domain is named but no cert has been issued for it yet. This is what an operator sees between naming a domain and completing the first obtain, and it is the line that says the gate is the certificate rather than the configuration.no edge.domain configured; :443 public edge stays dark: no served domain named, so:443is intentionally dark by configuration.
In the supervisor's status field (systemctl status valis.service, the Status:
line), the resident publishes its serving state and refreshes it on every health-loop
pass. It reads <serving state>; cert <days>; store gen <n>, where the serving state is
one of:
serving :53 + :443: a usable certificate was in custody at boot; the public HTTPS edge is open.serving :53; :443 dark (no cert): a healthy genesis boot::53is up (which is exactly what dns-01 needs to obtain the first certificate), and:443is cert-gated dark until a cert is issued. This is not an error and does not withhold readiness.
A silently dark port would look like a fault; between the two surfaces you get why
:443 is or is not open, so you never have to guess.
6. Descriptor headroom
Nothing in the resident gauges descriptor headroom: there is no count in the journal, no field in the status line, and the liveness gate does not read it. A node that has exhausted its descriptors keeps reporting itself as serving and the supervisor is given no reason to restart it, so headroom is a reading the operator takes from outside the process, on the host:
pid=$(pgrep -o -x valis) # the resident is fulcrum's child, not MainPID
ls /proc/$pid/fd | wc -l # descriptors held right now
grep 'Max open files' /proc/$pid/limits # the ceiling this process actually has
Match the process name to whatever the launch configuration's :valis-command stages. Take
the reading as root or as the run account: another account's /proc/<pid>/fd is not
yours to list. The unit sets no descriptor limit of its own, so the ceiling is the
service manager's default on this host: read it rather than assume it.
One reading tells you little. Take two, some hours apart under comparable load: a busy node's count moves in both directions, while a count that only ever climbs is a leak, and the moment to act on it is while the number is still nowhere near the ceiling. Put that pair of readings into whatever monitoring you run against the node. Nothing on the node will raise it for you, and the failure it precedes is one every other instrument reports as healthy. Why that is, and what the liveness gate does and does not ask, is set out in backup-and-observability.org.
7. The certificate lifecycle
The :443 edge is obtained and renewed over the ACME/dns-01 spine (custody and the
ACME lifecycle are mercer's; valis is the edge that consumes the cert):
- Genesis (no cert). Boot with
edge.domainset but no cert yet::53comes up,:443is cert-gated dark. Serving:53is the precondition for dns-01: the authoritative answer proves domain control to the CA. - Issuance. Once a certificate is issued and lands in the custody store
(
acme.store-path), a boot with the cert in hand opens:443. dns-01 is the only challenge a node answers today, and there is no fallback. Nothing answers an http-01 challenge on:80or a tls-alpn-01 challenge on:443, so an order for either has nothing to validate against. That is why serving:53is a precondition of issuance rather than a convenience. - Renewal (hot-swap, no restart). On a successful renewal the live
:443cert is reloaded in place through the shared credential cell: the edge swaps the new leaf without dropping the listener or restarting the unit. A renewal orders under the same account identity the first order used, and it reads that identity from the node'sacme.account-contactsetting: set it (valis config set acme.account-contact ADDRESS), or renewal refuses every attempt and backs off while the certificate you are serving walks toward its expiry. The boot line reports whether a contact is configured, so check it there rather than waiting for the backoff to tell you. - Staging vs production. Default is the ACME staging CA. Set
acme.productiontotrueonly when you want a real, publicly-trusted certificate: production carries a strict rate ceiling, so exhaust staging first. - Wildcards, and the name a certificate is filed under. Custody files an issued
certificate under one name, and a wildcard identifier such as
*.example.comis a matching rule rather than a name: no filesystem will hold it as a directory. So an order is placed with a host name leading, whatever order you asked for it in, and the certificate covers exactly the names you named. Ask for nothing but wildcards and the order is refused before it reaches the CA, because the alternative is to buy a certificate that has no name to be filed under. Every issuance counts against a rate limit whether or not you get to keep it. - A certificate per name, chosen per connection. Ask the node for a name it holds a
certificate for and you are presented that certificate. Ask for anything else and you
meet the credential sourced for
edge.domain, which is the fallback rather than a failure: an unknown name gets the base certificate, and your client then rejects it on a hostname mismatch as it should. Two names count as one only when they differ in case or in a single trailing root dot, because DNS and custody both file them that way. ⚠ Do not read the boot lineholds a certificate for N name(s)as any one certificate's subject names: that is the admission list, the names the edge will answer for. Which certificate each of them is served is decided per connection, from custody.
A domain whose DNS lives elsewhere. A node is meant to prove control of every name it
serves. For a domain you bring while its DNS stays with a third party, the proof is
tls-alpn-01, answered on the node's live :443 edge inside the TLS stack that already
serves it, once you enroll that domain through the control plane. That is not built
yet. Today such a domain cannot obtain a certificate, and its site does not serve:
valis obtain refuses it as a name the node does not serve, before any order reaches
the CA. The same holds for a node whose served set leaves out :53, since a node
answers dns-01 only from its own :53. To serve a domain today, delegate its DNS to
this node and hold its zone here.
Driving the first obtain by hand. The obtain verb runs the order on the process
that serves :53, which is what makes the dns-01 challenge answerable at all. Run it
from your own machine and name the node:
valis --node NODE config get acme.production
valis --node NODE obtain --domain example.org --contact hostmaster@example.org
Its whole form is:
valis obtain --domain <fqdn> --contact <email> [--profile <name>] [--directory-url <url>]
⚠ Read acme.production first. The first obtain does not refuse a node whose
acme.production is false: it orders from the staging authority, or from
acme.directory-url when that is set, and serves the certificate it gets, which no
browser trusts, on the node's :443 edge. On a node meant to serve the public, set it
before the first obtain with valis --node NODE config set acme.production true. A
renewal on a node with edge.domain set is refused while acme.production is
false, but the first obtain is not, so that read is yours to make.
The verb answers once the order is done. Its reply begins obtained, then a
domain NAME line for each name and a next-renewal line, then a line saying where
the new certificate is served: edge serving now when the :443 edge took it at once,
or edge not open: serving from the next start on a node whose edge is not open yet.
A line for each name follows, saying whether a client asking for that name is handed
the certificate. With a key the node has not enrolled, the verb is refused with exit 2
and no order is placed.
To tell the new certificate from the one it replaced, compare the leaf's serial and its
notBefore. Its issuer may differ, since the authority can sign the new leaf from
another of its intermediates, and a production re-order did. The order also places its
dns-01 challenge records in the node's zone and takes them out again, and each change
moves the zone's SOA serial forward, so expect a higher serial after an obtain.
Both --domain and --contact are required; --domain repeats, or takes a comma-set.
⚠ Take a comma-set only for names that genuinely belong on one credential. Certificate
Transparency publishes the name set permanently, so packing several of your domains
into one certificate publishes that you hold them all, and nothing takes it back. One
certificate per registrable domain.
Like the other owner verbs it takes a pinned node through --node or VALIS_NODE,
and it stays on staging unless the node's acme.production setting is true. When
you cannot use the route from your machine (the owner port closed, the pin or the device
key lost, or the node not answering on the owner port), run it on the node's host as the
break-glass route, inside the node's namespace with --keyfile and --endpoint (see
the break-glass route):
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis obtain --domain example.org --contact hostmaster@example.org \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
A first-obtain can also be driven remotely by the authenticated owner over the
/acme/ctl 9P door: obtain <domain> <contact> starts the dns-01 order on a
background thread (single-flight: a second obtain while one runs is refused,
since every attempt burns CA rate-limit budget), and status reports
idle/running plus the last completed outcome. The door threads no CA selection
of its own; the staging-unless-opted-in default above applies unchanged.
Reading back what a node holds. I added valis cert list so you can see which
certificates a node holds, and whether its edge serves each one, without a shell on its
host. Name the node as you do for every owner verb:
valis --node NODE cert list
The verb changes nothing on the node. It prints one line per certificate the node holds, sorted by the name each is filed under, with five fields separated by tabs:
- the name the certificate is filed under;
- its not-after instant;
- its next renewal;
- whether the node's
:443edge serves it:servedwhen the edge hands it out for every name it carries,partialfor only some of them,not-servedfor none,no-edgewhen the node has no edge listening, andunconfirmedwhen the node could not read what its edge serves; - the DNS names it carries, joined by commas.
Both instants are whole numbers in the same encoding as the next-renewal line of an
obtain reply, or unknown when the node cannot read them from the certificate. For a
node that holds no certificate the verb prints nothing, and no line carries key
material. Only the node's owner can read the list: a key the node has not enrolled meets
an empty view and is refused before the list is read, as with every owner verb. On the
node's host it runs by the break-glass route, as obtain does above.
| Exit | Meaning |
|---|---|
| 0 | the node answered; its list is on standard output |
| 1 | a fault on this machine, not the node |
| 2 | refused, with the list not read: before dialling, for a malformed command line, no pin, or a key file that is absent, unsafe, or malformed; during the handshake and before your key is sent, for a node that does not match its pin; after the attach, for a key the node gave no owner view |
| 3 | the node could not be reached, its name did not resolve, or the session with it failed |
ubik's certificate panel reads the same list, and orders a certificate through the same
door valis obtain writes. On a node that predates cert list, the panel says that the
node does not list certificates and still offers the order.
8. Reaching a node's owner control plane
The owner-keyed verbs (zone, apart from load, secondary --offline and
delegation; obtain; publish; publisher; restart; key; and config against a
running node) reach the node's owner control plane. You run them from your own
machine and name the node:
valis --node NODE key list
valis --node NODE config get edge.served-ports
valis --node NODE zone export --origin example.org
That is the normal route, and it needs three things in place: the node pinned on your machine, a device key on your machine that the node has enrolled, and the node's owner port open. From another machine below shows how to set up all three. The client dials the address you pinned and refuses a responder that cannot prove the pinned DID before your key or any request reaches it. The node's answers print as they do on the host.
A key the node has not enrolled. The node still admits the session, but only to an
empty view, in which nothing manages the node. Every owner verb reads that view before
it sends a request, refuses with exit 2, and says that the node gave the key no owner
view: the key is not enrolled on that node, or was retired, and valis key list on
the node's host lists the keys it has enrolled. Nothing on the node changes.
The host route stays for when you cannot use the route from your machine: a shell on the node's host, and the verb run inside the node's network namespace with the node's own key. The break-glass route gives it and says when to reach for it.
8.1. From another machine: a pinned node
From a machine of your own you reach the node by name, with a device key the node has enrolled, and the client checks that the node answering is the one you pinned. The route, once:
- On the client,
valis key generatewrites this machine's device key and prints its DID (see On the client: a device key of its own). - On the node's host,
valis key enrol DID --label LABELadmits it (see From the host). - On the client, pin the node:
valis node pin NAME HOST:PORT DID, with the node's own Ed25519 DID, thedid:key:z6Mk...thatvalis key listprints first on the host. A node is pinned by that form alone:valis node pinrefuses any other DID, an X25519did:key:z6LS...included, with exit 2 and the store unchanged. A store that holds any other DID, or is written in any other form, is refused whole and never read as empty; move it aside and pin the node again.
Then every owner verb takes the name:
valis --node lab key list
valis zone export --origin example.org --node lab
VALIS_NODE=lab valis restart
--node may stand anywhere before a -- word, and --key FILE presents a device key
other than the default. Each is given once and needs a verb, as every owner flag does.
When the device key the verb would present is missing, the refusal gives the command
that writes it: valis key generate, with --out FILE when --key or VALIS_KEY named
a file other than the default. Nothing falls back to another key. --node and
VALIS_NODE may not be combined with --keyfile or
--endpoint: the verb refuses before reading a key or dialling, rather than letting one
replace the other, and --key or VALIS_KEY with no node named is refused for
the same reason. Under a node, VALIS_KEYFILE and VALIS_MGMT_ENDPOINT are not read. A
node that is not the one pinned is refused before your key or any request reaches it,
and the refusal names the command that pins it again. The node's answers print as they
do on the host.
The key you present under a node is never the node's own. A --key or VALIS_KEY whose
DID is the pinned node's DID, the node's keyfile named on its host for instance, is
refused with exit 2 before anything is dialled, and the refusal names the key file. I
made the client refuse that key so the node's seed never goes
out under a pinned name: under a node the verb presents an enrolled device key, and the
node's own key is presented on its host with --keyfile and no node.
Opening the owner port, after the pin. The node's owner port, 564 unless the host
agent's configuration names another, ships closed: the unit's firewall, which the host
agent installs, drops it, and the host agent binds no socket for it. Open it only once the owner's machine has pinned the node,
so the first attach from off the host already checks who answers. In this order, with
NODE the name you give the node and ADDRESS its public address:
On the node's host, read the node's DID inside its namespace, as the break-glass route shows:
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \ /opt/valis/bin/valis key list \ --keyfile /var/lib/valis/valis/keyfile \ --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"The first line names the node's own key, and its
did:key:z6Mk...is the DID to pin.- On the owner's machine, pin it:
valis node pin NODE ADDRESS:564 DID. Pin the address itself. The client dials an IPv4 or IPv6 address as given, with no reverse lookup first, so an address with no reverse DNS record connects like any other; a host name in a pin is resolved forward to its first address. Only then, on the node's host, open the port with the host agent's
owner-portverb, and restart the unit:sudo /opt/valis/bin/fulcrum-resident owner-port open sudo systemctl restart valis.service- On the owner's machine, attach:
valis --node NODE key list.
The owner-port verb sets :open-owner-port in the host agent's configuration store
and no other setting, though it rewrites the whole file, so hand comments in it are
lost. It restarts nothing and checks no pin, so the order above is yours to keep, and
the change takes effect at the next restart of valis.service. To close the
port again, run sudo /opt/valis/bin/fulcrum-resident owner-port close and restart the
unit; the pin stays on the owner's machine for the next open, and enrolled keys stay
enrolled across the restarts. owner-port status changes nothing. Each of the three
prints the store it read, the configured port, a configured: line with the store's
setting, and a live: line saying whether the unit's firewall carries the owner accept
line, so between a change and the restart the two lines disagree. open and close
exit 0 when the change was made or the store already held it, 1 when the verb refused
or failed, and 2 for an unrecognized argument.
An owner port other than 564. The resident picks the owner terminus's descriptor by
its port: 564, unless the launch names another with --owner-port N. N is a whole
number from 1 to 65535, written in digits alone, given once, and not a port another
service takes (25, 53, or 443); given any other value, the resident refuses the launch
and does not start. At boot it writes one line to the journal for each descriptor it was handed that
no service it runs takes:
valis: descriptor 9 (stream, port 2564) was handed down and answers for no service this node runs, so it serves nothing
A lone half of the DNS pair gets a line of its own, ending was handed down without the
other half of the DNS pair, so it serves nothing. ⚠ The host agent does not yet pass
--owner-port when it launches the node. Configure the host agent with an owner port
other than 564 and the boot reports that port's descriptor with the line above, and the owner terminus
does not come up; keep the owner port at 564 until the host agent passes it.
Checking the port from another machine. scripts/owner-port-reach.sh reads whether
one connection to an address and port completes or gets no answer. A firewall that drops
a port reads as no answer, so a reading means something only beside two controls on the
same address: a port the node admits and a port it drops. The probe reads both first and
gives no verdict when either reads otherwise:
scripts/owner-port-reach.sh --address ADDRESS --port 564 \
--open-control ADMITTED_PORT --closed-control DROPPED_PORT --expect open
It exits 0 when the port read as expected, 1 when it read otherwise, 2 for no verdict,
and 3 for a usage error, which includes --expect without both controls. Run
--self-test first to see the probe separate the cases on loopback.
To see which view the node gives a key, build owner-view-proof with
scripts/build-owner-view-proof.sh, which writes bin/owner-view-proof, and attach
with it: owner-view-proof --node NODE --key FILE --expect owner-frame. An enrolled
key gets owner-frame, a key the node has not enrolled gets empty, and
owner-view-proof --anonymous --endpoint ADDRESS:564 gets published-floor. It exits 0
when the view matched --expect, 1 when the node gave another, 2 when
owner-view-proof refused an argument or refused the attach before sending anything,
and 3 when the node could not be reached. It prints only a key's
public DID, and its --self-test shows it telling the three views apart on a scratch
fabric. With --node, --boot also prints boot: HEX, the node's boot identity, which
the node draws afresh at every start, so two equal readings bracket a span with no
restart in it. --pub SLUG prints pub: SLUG bytes N md5 HEX for the publication as
read through the attached view, or pub: SLUG absent. The length and digest are of the
bytes as published, so a file that ends in a newline counts that newline.
Owner-port connection caps. The owner port holds at most 4 connections at once from
one origin and 16 from all origins together, counting each from the moment it reaches
the port, handshake included. Change them with
valis config set owner-port.sessions-per-origin N and
valis config set owner-port.session-ceiling N, each a whole number from 1 to 63; the
node reads the new cap at the next connection. One origin is one IPv4 address or one
IPv6 /64, so machines behind one public IPv4 address share one origin and its 4
connections. A connection past either cap is closed before anything is read from it,
and the node logs the refusal at WARN at most once per origin per minute, in a line
that begins origin cap: refused a connection from. A session the node grants
nothing, with no enrolled key and no accepted capability, has 10 seconds before the
node closes it.
Where this stands. This route is proven from another machine against a deployed
node. The client refused an attach with no pin before dialling anything, the pin was
written before the port opened, and the pinned attach listed the keyring. The client
refused a pin naming another DID, and a key the node never enrolled got an empty view.
The port read closed, open, closed, and open again from off the host as owner-port
and a restart moved it, each reading beside both controls. The node's director carries
every connection to the owner port to the owner terminus: on that node a silent
connection to the port is cut at 10 seconds, and a fifth connection from an origin
already holding 4 is refused at accept. Only the terminus's claim on the port, where
the director carries those connections, arms that bound and that cap.
Managing a node over this route is proven on a deployed node that serves the public,
in one run from the owner's machine with no shell on the node's host. A zone record
was added and deleted, and was answered and then gone over UDP and TCP. A page was
published and revised, and each text was served at once. With one session from the
owner's machine already open, owner-port.sessions-per-origin was set to 2, and of two
more connections from that machine one was held and the other refused at accept; the
setting was then reset. Port 8099 was added to the served set, read open from off the host
beside both controls, and read closed again once the set was put back. The node's
boot identity read the same before and after those changes, so none of them restarted
it. A certificate for the names the node's certificate already covered was ordered
from the production authority, and the :443 edge served the new one at once, with no
restart. Last, the node was restarted and answered from a new boot identity, with the
device key still enrolled and both settings the run had changed reading as they did
before the run. Each change was read back after it took effect. A key the node never
enrolled got an empty view and was refused by every one of those verbs with exit 2,
and nothing on the node changed. A peer with no key got only the published floor, with
no door to the names, the configuration, or the node.
That run showed two gaps. The restart ended the owner's ubik session, and the ubik of that run had to be reconnected by hand; ubik now attaches again on its own, as Managing a node from ubik describes. The page stays published, since no verb retracts a publication.
⚠ The gaps. Opening or closing the port takes a restart of the unit. The first proof
on a node that serves the public used the valis command line and owner-view-proof;
ubik has since managed that node too, and
Managing a node from ubik names what ubik has not shown
there yet. Enough origins together can still fill the 16 and keep you out
of the port for as long as they keep reconnecting, and the caps are defaults I chose,
not measurements; your way in then is the host, by
the break-glass route.
One store, two clients. The pin store is $XDG_CONFIG_HOME/ubik/node-pins.sexp (or
~/.config/ubik/node-pins.sexp), shared with ubik, which reads and writes the same file.
Before you pin with valis on a machine where ubik also runs, make sure that ubik writes
the shared format, or one client may refuse the other's store. Two clients writing the
store at the same moment never leave it torn. Both clients change the store under its
lock, the file node-pins.sexp.lock beside it: valis node pin and valis node unpin
take it, and so does ubik when it records a pin, so neither client loses a change the
other makes. valis waits up to ten seconds for that lock. If another writer keeps it
longer, the verb exits 3, names the store, and changes nothing; run it again once the
other writer finishes.
8.1.1. Managing a node from ubik
ubik reaches a node the way the command line does: by a pin name in the same store, with a device key the node has enrolled. I made ubik present a key of its own, enrolled on each node under its own label, so you can retire the key ubik presents without touching the one your command line presents. Create it on your machine and enrol its DID on each node's host:
valis key generate --out ~/.config/ubik/owner.key --label ubik-laptop
ubik refuses to present the node's own key, by pin name or by any other route, and refuses a key file that grants anything to its group or to others; both refusals come before anything is dialled. A key the node has not enrolled meets an empty view, and ubik closes the connection.
Once ubik is attached, you manage the node from its panels: its configuration, the served set, a restart, its certificates, publication, and the keyring. The panels write the node's ordinary ctl doors, the ones the command line writes, and the served-ports panel writes the per-port door the command line has no verb for. Each panel reads the node again after every act, so what you see is what the node now holds. A disruptive act takes a second click on a confirmation that names its effect: a restart, taking a port out of the served set, retiring a key, or an order from the production authority. ubik never offers to take out the port it is attached through.
When the connection goes, whether ubik sent the restart or the node dropped off the
network, ubik attaches again on its own: at once, then after 1, 2, 4, 8, and 16 seconds,
and every 30 seconds after that. Every attempt checks the pin again. An attempt that
meets a node no longer matching its pin stops there, and ubik shows you the refusal
instead of trying again. A node that goes silent without closing the connection is
reported lost within 25 seconds of its last answer. Once ubik is attached again, every
panel reads the node again. ubik's own manual, docs/OWNER-MANAGEMENT.org in the ubik
repository, covers the panels in full.
Where this stands. On a test node, ubik attached with its own enrolled key and, when given the node's own key, refused it before dialling. Each panel changed the node, and each change read back through the valis command line or from off the host. ubik attached again after a restart it sent, after the node reset the connection, and after a silent drop, and it stopped when the pin was changed under it. A key the node never enrolled changed nothing.
Most of the same checks then ran on the deployed node that serves the public, the one
the command line was proven on, through ubik-manage, ubik's scripted runner, from the
owner's machine with no shell on the node's host. ubik attached with its own enrolled
key and read the same keyring listing as the command line, line for line. Given a
stand-in key pinned as the node's own (the node's own key never leaves its host), ubik
refused it on every route and dialled nothing.
With its own session open, ubik set owner-port.sessions-per-origin to 2. Of two more
connections from the owner's machine, the node held one open and refused the other at
accept. ubik then reset the setting to its default, as it was before. It added port 8099
to the served set; from off the host the port then read open, and ubik took it out
again. It revised a page already published and then restored the page's earlier text,
each text read over HTTP. It retired a key enrolled for the checks, which the command line
then read as retired. Its certificate list matched the command line's line for line, and
its sessions view matched the connections counted on the owner's machine.
ubik refused each disruptive act sent without its confirmation. Even with the confirmation given, it refused to take out the port it was attached through, a request its panels never offer. A restart sent through ubik brought the node back on a new boot identity. ubik attached again on its own about 3 seconds later, and every panel read the node again. A key the node never enrolled got an empty view, every act sent with it was refused, and nothing on the node changed. Every setting, port, and page the checks changed was put back afterwards. Every panel was then opened against that node and showed what the node holds, with the rough edges listed below.
Not yet shown on that node:
- A ubik session coming back on its own after something else, such as a deploy, restarts the node. The restart in those checks was sent through ubik, so it does not show a restart from elsewhere. The one other ubik session open at the time came from an older build of ubik, without automatic reattachment, and it stayed down.
- A lost connection without a restart, whether a reset or a silent drop. Causing one on that node needs a shell on its host, so ubik attaching again after one is shown on a test node only.
- A certificate ordered through ubik. On that node ubik listed the certificates and refused an order without its confirmation, and no confirmed order was sent; certificates ordered through ubik have been issued on a test node only.
ubik's interface has these rough edges today:
- The publication panel takes the page's name in ubik's input line at the foot of its window. The input line shows no prompt, and no cursor until you type. The editor above it, where you compose the page, shows a blank white field with no cursor.
- A divider under the publication panel's instructions looks like the box they mention, but is not one.
- Scrolling a pane redraws the whole pane with a visible flash.
- The configuration panel works, but its buttons are oversized and its layout is hard to read.
8.2. From the node's host: the break-glass route
Reach for the host when you cannot use the route from your machine:
- the owner port is closed, or the node does not answer on it;
- the pin, or the device key, is lost from your machine;
- the change is one only the node's own key may make, such as enrolling a device key (see Enrolling and retiring an owner device key).
On the host the verbs reach the resident over its loopback fabric. On a deployed node
that loopback belongs to the resident's network namespace, not to the host. The
fabric binds a fresh port at every boot, and the resident writes the address it bound
to mgmt-endpoint in its data root, which is /var/lib/valis/valis/mgmt-endpoint
under the shipped unit.
Run the verb on the node's host, inside that namespace, which you enter by the resident's pid:
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis zone export --origin example.org \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
- Read
mgmt-endpointevery time. The port is drawn afresh at each boot, so an address you saved is wrong after the next restart. - Enter by pid.
nsenter -nenters the network namespace and nothing else, so host paths resolve as usual, and the pid is always there to name it by. - Run the verb as the account that owns the keyfile. Entering the namespace needs
root, so
nsenterruns undersudo, and-Sand-Gthen drop the verb to the run account before it starts. Give them numbers, asid -uandid -gprint them:nsenterdoes not accept account names there. - The keyfile must be the run account's own. The verb presents the file named by
--keyfileorVALIS_KEYFILE, or the one in its data root, only when it is a regular file, not a symbolic link, owned by the account running the verb, and granting nothing to its group or to others. Anything else is refused before the key is read, with a message naming the file and what was wrong; for a file another account owns, it names both account numbers. Run as root, the verb refuses the run account's keyfile for exactly that reason. A keyfile that is missing, or that is not exactly the 32 octets of a seed, is refused the same way, and never reported as a node that could not be reached. Every owner verb (zone,obtain,publish,publisher,restart,key, andconfig) exits 2 on this refusal, as on any refusal made before anything is sent. - The client checks who answered. The keyfile names the node's own key, so the verb
presents that key to the node that holds it and to no other. A responder that cannot prove it
holds that key is refused before anything else is sent, so a process squatting on the
endpoint is refused rather than answered, and it never sees the attach. Every owner
verb exits 2 on that refusal too, and a
restartwaiting for the node to come back stops at the first such refusal. - Arguments the verb refuses also exit 2. Each owner flag (
--keyfile,--endpoint,--node, and--key) may stand anywhere before a--word, is given once, and needs a verb. A flag given twice, a command line that has an owner flag and no verb, or an owner flag beside--resident,--daemon,--dev,--backup, or--restoreis refused with exit 2 before any key is read or anything is opened, dialled, or started. - Pass both
--keyfileand--endpoint. The verb keeps root's environment, so its default data root holds neither the node's keyfile nor its endpoint, and the defaults find nothing. - An ssh forward reaches nothing. A forward lands on the host's loopback, which is a
different
127.0.0.1from the one the fabric is bound to, and nothing listens there for it. Off the host, use the route from your own machine, in Reaching a node's owner control plane.
9. Enrolling and retiring an owner device key
A node has one owner identity, its did:key, and one seed behind it, which never
leaves the host. I wanted the owner's other devices to manage a node without a copy of
that seed, so the node's factotum holds a keyring of device keys beside the seed. Each
is a device's own Ed25519 did:key, enrolled under the node's fixed owner identity.
Enrolling or retiring a device key never changes the node's identity and never moves,
rewrites, or copies the seed.
What an enrolled key reaches. A session presenting a live enrolled key is admitted to the owner frame: the node's services through their loaded modules, minting grants, signing with the node's own key, and backups. It is the frame an owner client such as ubik reaches over the node's capability system. The session keeps the device's own DID as its principal, so the session listing names the device rather than the owner.
The node's REPL is not in that frame. The owner's total reach is host authority: ssh to
the host, and from there the node's REPL. No device key reaches it, and the
ssh-reality proof, which mints a grant over the whole namespace, admits only the
pinned operator SSH key. Enrol only your own devices. Give another person a grant, or an instance of
their own, never a place on your keyring.
What a key carries. Each key records its DID, a label you choose, the key that enrolled it, and when. Retiring it records who retired it and when, and ends it. It has no expiry, so it stays admitted until you retire it. Read the listing from time to time and retire the key of any device you no longer hold.
Who may change the keyring. Only a session holding the node's own key enrols, so enrolment happens on the host. The node's own key or any live enrolled key may retire a key, so a lost device can be cut off from another device you still hold. The node's own key can never be retired, and a retired key cannot be enrolled again while the keyring records its retirement: a device whose key was retired mints a new one and is enrolled under that. Setting a refused keyring aside discards those records (see Healing a keyring the node refused).
9.1. On the client: a device key of its own
A device you manage the node from holds a key of its own, never a copy of the node's seed. I made the client mint that key itself, so the secret is created on the machine that uses it and never crosses a network or a command line. On the client, run:
valis key generate # writes ~/.config/valis/device.key
valis key generate --out FILE --label laptop
The key goes to $XDG_CONFIG_HOME/valis/device.key, or ~/.config/valis/device.key
when XDG_CONFIG_HOME is unset, empty, or not an absolute path, unless --out names
another file. A relative XDG_CONFIG_HOME is ignored, as the XDG Base Directory
specification asks, so the default path never depends on the directory you run the
verb from. valis key generate does not read VALIS_KEY: to write the key a later
session will look for at another path, name it with --out. The file holds the
32 octets of an Ed25519 seed, the same format as the node's own keyfile, at mode 0600
whatever your umask. When the valis directory does not exist it is created at 0700; a
directory that already exists keeps its mode. The verb never replaces a file and never
writes through a symbolic link: with anything already at the path it exits 2 and writes
nothing. It contacts no node, so --keyfile, --endpoint, --node, and --key are
refused.
Standard output carries the device's did:key and nothing else, and standard error
carries the file's path and the line to run on the node's host:
valis key enrol did:key:z6Mk... --label laptop
The label is --label when you give one, and otherwise this machine's short host name
with any character a label cannot carry replaced by a hyphen. A label is 1 to 64 of the
characters A-Z, a-z, 0-9, dot, underscore, and hyphen, the rule the node's keyring
applies at enrolment. A --label outside that rule is refused with exit 2 before
anything is written, so the line printed is always one the node accepts. Run that line
on the node's host, as the next section describes. The seed is never printed.
The client then presents the key to a node you have pinned, with --node, as
From another machine describes.
| Exit | Meaning |
|---|---|
| 0 | the key was written; its DID is on standard output |
| 1 | the key could not be written |
| 2 | refused, nothing written: something is already at the path, the label is one the node's keyring would refuse, or a flag that reaches a node was given |
9.2. From the host: valis key
Run valis key the way every owner verb runs on the host, inside the resident's network
namespace as the run account, with --keyfile naming the node's keyfile and
--endpoint read from mgmt-endpoint (see
the break-glass route). The session then presents the node's own
key, which is the only session that enrols. The verb has three subcommands, and the
wrapper below supplies the rest of each command line:
valis key enrol <did> --label <label>enrols a device key;valis key listprints the keyring;valis key retire <did>retires a device key.
k() {
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis key \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)" "$@"
}
k enrol did:key:z6Mk... --label laptop # the device's Ed25519 did:key
k list # the node's own key, then every device key
k retire did:key:z6Mk... # either DID form of the device key
A label is 1 to 64 of the characters A-Z, a-z, 0-9, dot, underscore, and hyphen.
The listing prints the node's own key first, then one line per device key with its
Ed25519 DID, its X25519 transport DID, its label, when and by whom it was enrolled, and
either state=live or, for a retired key, when and by whom it was retired. Retired
keys stay in the listing.
The verb exits with one of four statuses:
| Exit | Meaning |
|---|---|
| 0 | the node enrolled or retired the key, or printed the listing; its reply is on standard output |
| 1 | a fault on this machine, not the node, such as no management endpoint found, or a malformed --endpoint; standard error names it. The request may have reached the node, so check with valis key list before you repeat an enrol or a retire |
| 2 | refused, and nothing changed: the node replied error <reason>, the arguments were refused before anything was sent, or the verb refused the node that answered or the key file it was given: absent, unsafe, not the 32 octets of a seed, or the node's own key presented under a node |
| 3 | the node could not be reached, its name did not resolve, or the session with it failed; standard error names it. The request may have reached the node, so check with valis key list before you repeat an enrol or a retire |
The reasons a node gives are not-seed-holder (an enrolment from anything but the
node's own key), not-owner, malformed, unsupported-proto, invalid-key,
invalid-label, seed-key (the node's own key named), already-enrolled,
already-retired, not-enrolled, and store (the keyring could not be written). With
a keyfile that is not the node's own, such as a device key, the verb exits 2 before
anything is sent: the keyfile route accepts only the node that holds the key it presents,
and the node answering does not hold that key.
After a successful enrolment the verb prints a notice on standard error, so the consequence is in front of you at the moment you grant it:
valis: key: <did> is now admitted to this node's owner management frame: its services and their modules, minting grants, signing, and backups. valis: key: it stays admitted, with no expiry, until you run valis key retire <did> valis: key: enrol only a device you hold.
9.3. From ubik: the keyring panel
ubik's keyring panel lists every key the node has enrolled, with its label and its state,
and under each key the sessions presenting it that are live now. It retires a key through
the same /id/ctl door valis key retire writes, after a second click on a
confirmation that names the effect. Retiring the key ubik itself presents takes a
confirmation of its own, saying that the session ends and that this ubik can no longer
attach to that node; ubik reports that retirement as applied, and the session as ended.
The node's own key is never offered for retirement.
For a new device, generate its key on that device, then paste its public did:key and a
label into the panel. The panel checks both and shows the command to run on the node's
host, valis key enrol DID --label LABEL, and sends nothing to the node: only a session
holding the node's own key can enrol a device key. The device's private key never leaves
the device.
The panel shows each key's sessions and does not fence them. Fencing a session from ubik waits for richer session control on the node.
9.4. From a REPL in the running image
The verb and the door call the same two functions, and a REPL attached to the running image can call them directly:
(valis/src/identity/keyring:enrol-owner-key
(valis/src/namespace/id:factotum-keyring) "did:key:z6Mk..." :label "laptop")
(valis/src/identity/keyring:retire-owner-key
(valis/src/namespace/id:factotum-keyring) "did:key:z6Mk...")
A refused change signals a condition naming its reason and writes nothing. On a node behind the director the development listener cannot be reached today (see the listener's destination), so there the host verb is the route.
9.5. Over 9P: /id/ctl
The host verb writes one line to the node's /id/ctl over its sealed owner session, and
an owner client on the host writes the same line. Each open of /id/ctl is its own conversation: write one
command, then read its one-line reply on the same open.
key proto=noise key=<device ed25519 did:key> label=<label> delkey key=<did:key>
The reply is enrolled <did>, retired <did>, or error <reason> with the reasons
above, and a refused command changes nothing. A read before any command returns the
listing. Only the owner frame names /id/ctl: no grant, bearer name, or delegated
token reaches it.
9.6. Retiring a key cuts off its sessions
Retiring a key takes it from every session presenting it, at once: each such session's
view empties and every handle it holds is revoked, so a client in the middle of a read
gets an error rather than more data. A session that attaches with the key while the
retirement is in progress is cut off as it attaches. A device that retires its own
key is cut off the same way, before the reply reaches it; the retirement still stands,
and valis key list on the host shows it.
If the node cannot finish cutting off a session, the retirement still stands and the verb still reports it, and the journal names the step that failed. Restart the node to end every session at once.
9.7. Where the keyring lives, and when the node refuses it
The keyring is keyfile.keyring beside the keyfile in the data root,
/var/lib/valis/valis/keyfile.keyring under the shipped unit, at mode 0600 and owned by
the run account. Every change is one record appended to it and synced before the node
acts on it. A change the node cannot write leaves the file exactly as it was, and the
verb replies error store. If the node cannot even put the file back, the journal says
so and the next boot refuses the file; heal it as the next sections show. Do not edit
it by hand.
The node refuses to start on any of these keyrings:
- one it cannot read whole: a torn last record, a line that does not parse, a version it does not know, or a retirement of a key never enrolled;
- one reached through a symbolic link, one another account owns, or one granting anything to its group or to others;
- one written for another owner, such as a keyring copied from another node, or an old keyring left beside a seed minted afresh.
The node refuses to start in the same way on a keyfile it will not load: one reached through a symbolic link, one another account owns, one granting anything to its group or to others, or one that is not exactly the 32 octets of a seed. The keyfile is left as it was.
A refused boot exits 78, and the unit's RestartPreventExitStatus=78 keeps systemd from
restarting it into the same refusal. The journal names the file and the problem. A
refused node leaves no image and no listener to connect to, so heal it from the host as
the next section shows.
9.8. Healing a keyring the node refused
The standard heal is to recondense the node from a backup. Restore the backup into a fresh state directory as Backup and restore shows, then start the unit. The node comes back with the keyring that backup carried and everything else with it. A restore cannot know of a key retired after the backup was taken, so retire any such key again once the node is up.
On a node installed from the delivery archive alone, recondensing from a backup is the only heal today. The manual heal below needs SBCL, Quicklisp, and a valis checkout with its dependencies on the host, and the delivery archive carries neither a REPL entry nor the valis source.
The manual heal repairs the keyring alone and leaves the rest of the node as it stands. Run it on the host, as the run account, with the unit stopped (a refused node already is):
Start SBCL as the run account with the data root the unit uses, and load valis from a checkout the way the Makefile's
test-onerecipe does:sudo -u valis env XDG_DATA_HOME=/var/lib/valis sbcl(require :asdf) (load "<quicklisp>/setup.lisp") (push "<workspace>/" asdf:*central-registry*) (asdf:load-asd #p"<checkout>/valis.asd") (asdf:load-system :valis)Load Slynk, start the loopback development listener, and reach it from your own machine through an ssh tunnel:
(asdf:load-system :slynk) (valis/src/dev/slynk:start-development-listener 4005)ssh -L 4005:127.0.0.1:4005 <host> # then attach your editor to localhost:4005The plain SBCL debugger in the ssh terminal offers the same restarts if you prefer it, and then you need no listener at all.
Evaluate the heal:
(valis/src/keyring-heal:heal-owner-keyring :state-dir #p"/var/lib/valis/")It takes the node configuration's lock for the whole heal. It refuses while the node runs, and when no seed is at the keyfile path. It then reads the seed and the keyring exactly as the boot does. Either it reports that the boot's loader accepts the keyring, or it stops on the boot's own refusal with the restarts that apply to it.
Choose one restart:
keep-keyring, offered when the file was refused only for its mode or for the account owning it. For the mode, it sets0600on the same file, and it is offered only when the file grants no write permission to its group or to others: a keyring others could write may hold content the node never wrote, so recondense it from a backup or set it aside. For another account's file, it installs a copy the run account owns once the content parses whole under the node's owner, and sets the original aside.salvage-keyring, offered for a damaged file. It keeps the whole valid records before the damage, drops the rest, and retires every key a dropped line names.recondense-from-backup, offered for every refusal. Restore a backup into a scratch state directory first, then give the restart that directory; it takes only the keyring from there:sudo -u valis mkdir -m 0700 /tmp/keyring-restore sudo -u valis valis --restore --in /path/to/valis-backup.sealed \ --state-dir /tmp/keyring-restore --acme-store-dir /tmp/keyring-restore/acmeA key the damaged keyring enrolled and the backup's keyring lacks is not on the healed keyring. The heal report lists each such key; enrol it again. Recondensing applies the retirements the damaged keyring holds, even when the node refused that file for its mode or its owner. When the damaged keyring could not be read at all, the heal report says so and its retirements are not applied, so retire any such key again once the node is up.
set-keyring-aside, offered for every refusal. The node then starts with no enrolled keys, and you enrol your devices again.
After a restart acts, the heal checks the result with the boot's loader again and returns a report: every key live and retired, every key the heal retired and the line that named it, every line a salvage dropped, and every key the damaged file enrolled that a backup did not carry.
Start the unit and read the boot line, then remove the scratch restore, which holds a copy of the seed:
sudo systemctl start valis.service journalctl -u valis.service -n 20 sudo rm -rf /tmp/keyring-restore
What the manual heal guarantees:
- the damaged file is renamed aside as
keyfile.keyring.refused-<UTC time>and never removed; - the seed is never moved, rewritten, or created;
- only the node's own seed binds the result, so another owner's keyring, or a backup of another node, is refused;
- after
keep-keyring,salvage-keyring, orrecondense-from-backup, a key the damaged file retires, or that a damaged or torn line names, stays retired, so a device retired that way is enrolled again with a new key. A torn retirement that names no key retires every live key, because nothing on the line says which key it retired. This holds whenever the heal can read the damaged file; when it cannot, the heal report says so.
set-keyring-aside discards the retirement records. After a set-aside the node no longer knows
which keys were retired, so re-enrol only keys you hold. Enrolment still needs a session
proven with the node's own key.
9.9. The development listener
The owner frame includes the node's configuration and its restart door, so an enrolled
device can set development.listener-port and restart the node. An enrolled device can
turn the development listener on but cannot reach it: the listener binds loopback only,
and on a node behind the director nothing reaches it today. The node's REPL stays with
host access.
9.10. Not built yet
- Opening or closing the owner port takes effect only at a restart of the unit; nothing changes it on a running node.
- The owner port's caps hold one origin to 4 connections and all origins together to 16, but enough origins together can still fill the 16 and keep the owner out of the port while they keep reconnecting. The figures are defaults, not measurements.
- ubik has managed a deployed node that serves the public and attached again after a restart it sent, but three things are not yet shown there: a ubik session coming back on its own after something else restarts the node, a lost connection without a restart, and a certificate ordered through ubik. The last two are shown on a test node. The publication panel, scrolling, and the configuration panel have the rough edges Managing a node from ubik lists.
- ubik shows each key's live sessions but does not fence one; fencing a session from ubik waits for richer session control on the node.
- A device key is enrolled only on the node's host, by design, since only a session
holding the node's own key can enrol one. ubik's keyring panel and
valis key generategive you the command to run there. - An enrolled key reaches the owner frame over the owner port, not over the
:443edge: the edge's owner proof accepts only a signature by the node's own key. - An enrolled key has no expiry. Until it does, the listing and
valis key retireare how a key you no longer hold is ended. - On a node installed from the delivery archive alone the manual heal is unavailable: the archive carries no REPL entry and no valis source.
- No verb retracts a publication; a page stays served until you revise it.
- The served set's per-port door,
/edge/served/ctl, has no command-line verb; ubik's served-ports panel composes its request one port at a time, and the command line changes the whole set throughconfig. - A transfer key for a zone's secondary is minted only on the node's host, never over a pinned node.
- The first obtain is not refused on a node whose
acme.productionisfalse, although a renewal there is, so read it before you order a certificate for a node that serves the public. - Mail is not managed from your own machine: the command line has no mail verb, and the node's mail over the owner port has not been through the checks the other verbs have.
10. Managing standing grants
A module loads only with two signatures: its publisher's vouch and your grant. You give
a grant for one module, or once for a publisher as a standing grant, which covers
everything that publisher vouches. You file publishers and grants by writing commands to
/cap/admit in your owner session; grant-publisher PUBLISHER-DID mints a standing
grant that lasts a century and files it in one step. A grant you mint yourself and file
with file-standing-grant ends at its own expiry, which admission checks. Because a
standing grant can stand for a century, I made the node keep a record of every change to your trust and ask you to look
at each standing grant again from time to time.
Two read-only files sit beside /cap/admit in the owner's /cap, and nowhere else: no
grant, bearer name, or delegated token reaches them. Each is rendered when you open it,
so everything you read from one open is one state.
/cap/grants- the inventory: one stanza for each grant you hold, then one for each
publisher you have filed, each a run of
KEY VALUElines, stanzas separated by a blank line. A standing grant's stanza carriesgrant(its hash),kind,state(active,withdrawn,backed-out, orrevoked),publisher,filed-at,reviewed-at(ornever),review(dueorok),last-admission,modules, and oneadmittedline for each module it admitted. Times are Unix times. No stanza carries the text of a grant or a vouch. /cap/audit- the grant audit trail, one entry a line, oldest first:
UNIX-TIME ACT SUBJECT ACTOR. ACT is one offiled-publisher,filed-vouch,filed-list,filed-owner-grant,cleared-owner-grant,filed-standing-grant,granted-publisher,withdrew,backed-out,revoked,reviewed, andrestored. SUBJECT is a grant hash, a designation, or a did:key, and ACTOR is the DID of the session that acted, orlocal.
On a node in service the trail only grows, and a restore whose backup carries a trail
replaces it whole. It records changes to trust, so a refused act, or one that changes
nothing, leaves no entry. When an entry cannot be written the act still stands, and its
reply ends with a line saying the trail did not record it. A torn final line, left by a
crash during an append, is skipped. Any other line that is not an entry makes
/cap/audit an error beginning cap audit unreadable: line N. An unreadable or unsafe
trail gives cap audit unreadable: and the reason, with no line number. Either way
/cap/grants, which takes its filing and review times from the trail, is an error
beginning cap grants unreadable. Backups carry the trail (see
Backup and restore).
Reviewing a standing grant. An active standing grant is due a review once more than
module.standing-grant-review-days days have passed since the later of its filing and
its latest review: 90 unless you set it, a whole number from 1 to 3,650 (see
the environment seam). An active grant with neither on record is
due at once. Write reviewed GRANT-HASH to /cap/admit to stamp it, and read back:
reviewed standing grant <hash> at <unix-time>; the review is advisory and admits nothing
The stamp is the trail's reviewed entry and nothing else, so a review whose entry cannot
be written is refused and nothing is stamped. Only an active grant you hold takes one: a
withdrawn, backed-out, or revoked grant is refused. The review is advisory: nothing
refuses a module under a grant that is due. While any active standing grant is due, the
node's status line (systemctl status valis.service) ends by counting them:
; 2 standing grants due a review (90-day interval)
Previewing before you withdraw or back out. Write preview withdraw GRANT-HASH or
preview back-out GRANT-HASH and read what the act would do, with nothing changed. The
first line is preview ACT HASH: nothing was changed, the second state and the grant's
state, and then what the act would do: for a withdrawal, the modules that stay and what a
fresh install from that publisher would then need; for a back-out, the modules admitted
under the grant, every one refused from then on. When the act would change nothing, one
line says so. A back-out preview is refused wherever the back-out itself would be.
Withdrawing and backing out. withdraw-standing-grant GRANT-HASH ends the grant for
anything new: it admits nothing more, and the modules it admitted stay, across restarts
too. back-out-standing-grant GRANT-HASH withdraws it and revokes its hash, so every
module it admitted is refused from then on. Withdrawal is one way, and there is no
reinstate: to trust that publisher again, file a new standing grant, which carries a new
hash and its own filing time. A publisher holds one active standing grant at a time, so
filing a different one while one is active is refused, naming the active grant's hash.
⚠ Where this stands. The command line has no verb for /cap/admit, /cap/grants, or
/cap/audit, and ubik does not yet compose any of them; you reach them with a 9P client
in your owner session. A back-out refuses the modules it names from then on, but nothing
removes them from a running node: they run until a restart, when the boot refuses each
by name and comes up serving the rest.
11. DNS zone management
valis is authoritative for the SOA zones it serves on :53. The zone data itself is
operator state: a node comes up serving whatever zones are in its operator-state
store, and comes up with none on a genesis boot. Loading and maintaining that data
is the valis zone verb family: a headless, owner-authenticated client the deploy
recipe drives non-interactively, and the operator drives by hand.
How it authenticates. From your own machine you name a pinned node with
--node NAME or VALIS_NODE, and each verb presents your enrolled device key and
reaches the node's owner-gated management axis through its owner port, as
From another machine describes:
valis --node NODE zone record add --origin example.org --owner www --ttl 300 --type A --rdata 192.0.2.10
valis --node NODE zone export --origin example.org
valis --node NODE zone record delete --origin example.org --owner www --type A --rdata 192.0.2.10
An added record appears in the export and is answered over UDP and TCP; a deleted one is gone from both. Only a key the node holds in its keyring is given the management axis, so an absent or wrong key fails closed: the management axis never opens to an anonymous caller, and a key the node has not enrolled is refused with exit 2 and changes nothing.
From the node's host, the break-glass route. When you cannot use the route from your
machine (the owner port closed, the pin or the device key lost, or the node not answering
on the owner port), or to mint a transfer key for a secondary (see secondary below),
run the verb on the node's host. There it reaches the management axis over the
resident's loopback fabric, inside the resident's network namespace, at the endpoint
the resident wrote at boot, and presents the owner key custody holds in the
StateDirectory keyfile. The key is supplied as a path, resolved
--keyfile PATH > VALIS_KEYFILE > the StateDirectory keyfile; the key bytes never
cross the command line. The break-glass route
gives the command that gets it there:
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis zone export --origin example.org \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
A transfer key is kept under the data root of the process that mints it, and the
recipe above keeps root's environment, whose data root is not the node's. To mint one
the node can use, name the node's data root as well, so the key lands in the node's
custody under /var/lib/valis/valis/tsig/:
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/usr/bin/env XDG_DATA_HOME=/var/lib/valis \
/opt/valis/bin/valis zone secondary --origin example.org --peer 198.51.100.2 \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
Either way, a verb that refuses before anything is sent exits 2: for a key file it will not present, a responder that is not the expected node, or arguments it refuses.
The verbs.
valis zone create --origin <fqdn> [--peer <addr>] [--secondary-ns <name>] valis zone record add --origin <fqdn> --owner <name> --ttl <s> --type <type> --rdata <value> valis zone record delete --origin <fqdn> --owner <name> --type <type> --rdata <value> valis zone import --origin <fqdn> --file <master-file> valis zone export --origin <fqdn> [--out <file>] [--full] valis zone delete --origin <fqdn> valis zone delegation --origin <fqdn> [--wait] [--timeout <s>] [--interval <s>] [--allow-range <client=range,...>] valis zone secondary --origin <fqdn> --peer <addr> [--notify <ref>] [--key-name <name>] [--master-ip <addr>] valis zone secondary --origin <fqdn> --peer <addr> --key-name <name> --existing-key # Straight to operator state, for a node that is not serving yet: valis zone load --origin <fqdn> --file <master-file> [--migrate] valis zone secondary --offline --origin <fqdn> --peer <addr> [--notify <ref>] [--key-name <name>]
- create brings a whole domain up in one move: it mints the apex records from the domain template, enrols the secondary when you name a peer, and prints the block you paste at the registrar. Reach for this before reaching for import on a new domain.
- record add and record delete publish or retract one typed record and leave the
rest of the zone alone. This is the editing path. An
--ownerending in a dot is absolute; without one it is relative to the zone. The master-file text format is a boundary format for the secondary, never the format you edit in. - delegation asks the parent zone's nameservers whether the delegation has settled,
and exits non-zero until it has. It needs no resident, no keyfile and no database,
so you can run it from anywhere while you wait on a registrar.
--waitpolls. It asks nameservers on port 53 and nowhere else. The zones it walks are somebody else's, so a nameserver whose address lies in a private or other special-purpose range is skipped unlessnet.dial-allowallows that range for the clientdelegation-probe.--allow-rangenames the allowance for one run instead, in the sameclient=rangeform: a lab whose simulated internet sits in 192.168.140.0/22 passes--allow-range delegation-probe=192.168.140.0/22. The check holds that value to thenet.dial-allowrules before it asks any nameserver: a member they refuse stops the run with exit status 2 and an error naming--allow-range, the member and why. - import loads an RFC-1035 zone master file as the named zone, committing it atomically; a malformed master is refused with a non-zero exit and changes nothing, so a bad file never half-lands. Re-importing an origin replaces its zone (advance the SOA serial).
- export writes the zone's master text (to
--out, or standard output). It defaults to the durable, re-importable master: the published records, the faithful thing to archive or re-load. Pass--fullfor the whole serving set, which also includes any transient turn-up records (e.g. ACME challenge records) present at that moment. - delete removes the whole zone (apex-SOA removal is whole-zone removal).
- secondary enrols a downstream secondary for a zone: it lands the allowlist row for
the address the secondary pulls from, mints a TSIG key, and prints the secret and a
BIND snippet to paste on the secondary. Over the owner session the key is minted only
after the node has accepted the enrolment, so a refused, unserved, or unreachable
node leaves no secret behind. With
--offlineit is minted only after operator state has recorded the row, so a write that fails leaves the key already in custody under that name as it was.--existing-keymints nothing: it lands the allowlist row for a key already installed on both sides, and prints no secret. Over a pinned node (--nodeorVALIS_NODE) only--existing-keyapplies. The minting form and--offlineare both refused with exit 2, before any setting is read, anything is minted, or anything is dialled. A key minted on your machine stays in your machine's custody, where the node cannot use it to authenticate a transfer, and--offlinewrites your machine's operator state, never the node's. To mint a new key, run the verb on the node's host. ⚠ The gap. No door on the node mints a transfer key for an existing zone yet, so a new key for a pinned node is minted only on its host. - load and secondary –offline are the two routes that do not need a running
resident. They write operator state directly, reading no keyfile and no management
endpoint, with the connection coming from the node's
operator-state.dsnsetting (or, when the address carries a password, fromVALIS_PG_DSN) and never from the command line. Reach for them on a node that is fresh or recovered and not answering yet. ⚠--offlineis what selects that route. secondary without it drives a running resident over the fabric, exactly like the owner-keyed verbs above, so on a node that is not serving yet it refuses and tells you to add the flag. load takes no such flag and is always direct. Either waysecondaryrecords the allowlist row; without--existing-keyit also mints the TSIG key and prints the BIND snippet to paste on the secondary. Both are walked through in first boot and operator moves.
A committed change is picked up by the running resident without a restart: the serving side re-reads the store, so the node begins answering the new data authoritatively.
At go-live. After a fresh node is up and healthy (serving :53, restore-proven per
the go-public gate above) the deploy recipe imports the node's own zone before the
address is cut over: whichever domain that node serves, with its apex SOA, its apex
NS, and the ns1 address record. The recipe runs valis zone import as
the account that can read the 0600 keyfile, then confirms the node answers the zone
over loopback (dig against :53) and that valis zone export round-trips before
declaring the node live. Zone master files are operator data staged onto the host, not
part of the delivery binary.
12. Publishing to a node, and updating one
Neither of these takes the node down, and neither wants you editing files on the host.
Publishing a page. publish places a publication into /pub on a running node,
owner-keyed like the zone verbs. It is an ordinary operator act at any time, not a
deploy-time one, and you run it from your own machine and name the node:
valis --node NODE publish --slug notes --file notes.html --content-type text/html
Its whole form is:
valis publish --slug <name> --file <path> --content-type <type>
All three are required. The slug is lower-case letters, digits, and hyphens, and
--file names a file on the machine you run the verb on, placed on the node byte for
byte. A slug that already exists is revised, not refused, and the node serves the
new text at once. With a key the node has not enrolled, the verb is refused with exit 2
and nothing is placed.
⚠ The gap. No verb retracts a publication. A page you publish stays served until you revise it, so publish only what you mean to leave up.
From the node's host, the break-glass route. When you cannot use the route from your
machine (the owner port closed, the pin or the device key lost, or the node not answering
on the owner port), run publish on the node's host, inside the node's namespace with
its keyfile and endpoint (see the break-glass route).
The file must then be on the host, readable by the run account:
sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \
/opt/valis/bin/valis publish --slug notes --file /var/lib/valis/notes.html \
--content-type text/html \
--keyfile /var/lib/valis/valis/keyfile \
--endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)"
Updating the binary. apply takes a delivery archive that is already on the host,
puts it in place, restarts the serving unit, and confirms the node still answers:
valis apply --artifact <path> --confirm-server <addr> --confirm-zone <fqdn>
It transfers nothing, so move the archive onto the host first. Without
--confirm-server it reads VALIS_APPLY_CONFIRM_SERVER; without --confirm-zone it
reads edge.domain in the configuration under your own account's data root, or
VALIS_EDGE_DOMAIN in your shell, and when neither is set, the node's own
edge.domain. When yours and the node's name different domains, the apply refuses and
names both: that disagreement is exactly the wrong-node case confirmation is there to
catch. Pass --confirm-zone to settle it; the flag always wins, since a node may
confirm for a zone other than its edge domain. The report names the zone it confirmed
and where that zone came from. There is no loopback default: name them or the apply refuses before anything moves. That refusal
is the point. Exit 2 means it refused and nothing moved; exit 3 means it moved and
could not confirm, which is the case that wants you looking. --dry-run walks it
without touching the host.
By default apply places the serving binary alone. --with-resident and --with-steer
are the explicit ask for the wider set, so the privileged parts of the delivery never
move because you forgot they were in the archive.
An apply leaves the host agent's configuration store as it is, so a node whose owner port was open answers on it again once the unit is back, under the same DID, and you go on managing it from your own machine with the pin you already hold.
12.1. Letting apply read the node's settings
Before it moves anything, apply reads the node's settings so it can judge what the
node will actually boot on. It does this by running its own binary as config list
under the node's run account, the owner of the unit's StateDirectory, through
sudo -n. It runs its own binary, never the one standing at the destination, because
an older binary there may not know the config verb and would start a node instead.
The read takes no lock and writes nothing.
Grant the deploy account that command in a file under /etc/sudoers.d/ (edit it with
visudo -f):
<deploy-account> ALL=(valis) NOPASSWD: /usr/bin/env XDG_DATA_HOME=/var/lib/valis /opt/valis/bin/valis config list
sudo matches the command word for word, so the grant fits only when:
- you run apply as
/opt/valis/bin/valis(a symbolic link to it is fine, since the binary names itself by its resolved path); - the unit's
StateDirectoryisvalis, which apply turns into/var/lib/valiswith no trailing slash; - the run account is
valis.
Understand what this grant is before you add it. It is not a narrow read permission.
apply needs /opt/valis/bin writable by the deploy account so it can replace the
binary, so the deploy account can put any binary it likes at /opt/valis/bin/valis and
run it as valis through this grant at once: no restart, no unit and no apply log
stand in the way. In effect the grant lets the deploy account run code it chose as the
node's run account, with everything that account can read and write, the node's store
included. That is the same trust the deploy account already holds through
sudo -n systemctl restart valis.service, which apply runs to restart the node, since
a restart runs whatever stands in /opt/valis/bin, but this one takes effect
immediately. Give it only to an account you already trust to deploy the node.
Through the grant the deploy account also sees every setting config list shows.
A secret shows only as set or unset, but a personal setting shows its value: today
that is acme.account-contact, the address the node's ACME account is registered to.
Without the grant, or when any of the three conditions above does not hold, the read
is refused, apply checks the flags and the unit's environment alone, as it did
before it could read the node, and its report says why the node's settings were not
read. A refusal for a missing site fact says so too.
One gap remains: config list does not show which keys the node's first boot already
settled from valis.env. A required key reset in the store after that boot reads as
unset to the node, but passes apply's check when valis.env still names it. The check
is never weaker than the environment-only check it replaced.
13. The module source root
module.source-root (seeded from VALIS_MODULE_SOURCE_ROOT) names the directory
holding the bytes of every module the node admits. Each module lives in a directory named for the identity of its bytes,
written once and never changed, and a new version of a module is a new directory
beside the old one. Never edit files under it. Keep it apart from the directory that
holds the owner keyfile, and keep it, and every directory above it, unwritable by
other users: an install refuses a root that is not.
I made the root a fact you state rather than a place valis picks, because it is where the code a node compiles into itself comes from. It is a boot-site requirement, so it is checked at three points:
- A resident whose configuration names no root refuses to boot, and the refusal names the key, and the variable that seeds it at a first boot. With a root named, the resident creates the directory at boot.
valis applyreads the node's settings, and the unit's environment, before it moves anything. When the root, or any other fact the resident needs to boot, is missing from both, the apply refuses with exit 2 and the node keeps serving what it served. Set it as the run account withvalis config set module.source-root ...with the node stopped, or, before the node's first boot, add the line to/etc/valis/valis.env, then apply.- Provisioning renders the env file from the same requirements, so a provisioned node carries the root.
What a node does with its modules. A module recorded as installed is admitted again from its identity directory at every boot, and its publisher's and owner's signatures are checked again then. A credential that expired after the module was admitted still admits it; a revocation recorded since stops it from loading. While a node serves real names, because it declares an edge domain or is authoritative for zones held on someone's behalf, it refuses to record a module install: a recorded module is stood up at every restart, and nothing yet falls back from a module that fails to stand up. A module that cannot be admitted again at boot does not run: the node logs it by name with the reason, comes up serving everything else, and names it on its status line. A backup carries no module bytes, so a node restored with modules recorded comes up without each one whose identity directory is not back under its module source root. Put those directories back and restart the unit to bring them up.
To install a module into a running node, or to supersede the version it carries, use
valis module install (Installing a module as the owner
below). Two verbs let you check a module first, on the node's host. Neither binds a port
or writes the node's stores, so either may run beside the running node:
valis module try-install /module/HEXinstalls the module into a process of its own, through the same gate a running node applies, reading the node's keys and stores, and records nothing. Run it as the run account on the node's data root, so it reads the node's configuration:sudo -u valis env XDG_DATA_HOME=/var/lib/valis \ /opt/valis/bin/valis module try-install /module/<hex>Exit 0 means the module was admitted and loaded. Exit 2 means you gave the verb no designation or more than one; 3, the argument is not a module designation; 4, no identity directory holds that module; 5, the module was refused before any of its code ran; 6, the module source root or the node state the verb needs is missing or unusable, or something else stopped the verb, such as a scratch directory it could not make; 7, both signatures admitted the module and its load then failed, so some of its code may have run. The verb's last line on standard output begins
valis-dry-run:and carries the same JSON verdicttry-supersedeprints, withphaseset tonew. That line is what a node reads when it tries an install first: before installing a module, a running node runs this verb in a child process of its own binary and goes on only when the verdict says the module loaded.valis module try-supersede /module/OLD /module/NEWloads the old version as a restart would and then the new one as an install would, again in a process of its own. It prints one line beginningvalis-dry-run:, carrying a JSON verdict that names what the new version redefined, and exits with the try-install status for the version named in the verdict'sphasefield. The node's log can write to standard output as well, so take the line with that prefix. A node runs this same check in a child process before it supersedes a running module.
Where a node compiles modules. A node compiles each module beneath module-fasls/
in its data root (/var/lib/valis/module-fasls/ on a packaged node), in a directory
named for the module's identity, and compiles a library a module needs that the binary
does not carry beneath module-fasls/deps/. Nothing is compiled into the run account's
home. A node upgraded from an earlier release keeps that release's compiled output under
module-fasls/, filed by each source's full path (.../var/lib/...); nothing reads it
any more. To reclaim the space, stop the unit, remove module-fasls/, and start it: the
next boot compiles its modules again.
A supersede is always tentative. A change which a running image cannot make in place is refused before any of it is compiled: a changed structure, macro, constant, type, or inline or ftype declaration, a removed file, a changed grouping form, or a change inside a top-level form the comparison compares only whole. The new version must then pass the child dry run and the node's own gate. Even then the old version's code can stay live: a closure that captured it keeps running it, and the scan for leftover code can miss some. So the node records its image as diverged from what a restart would build. Restart the unit when you need the running image to match its record.
13.1. Installing a module as the owner
valis module install asks a running node, as its owner, to install a module, or to
supersede in place the version of it the node already runs. You name the module by its
designation and say where its bytes are:
valis --node NODE module install /module/<hex>
valis --node NODE module install /module/<hex> --dist /path/to/release
valis --node NODE module install /module/<hex> --git REPOSITORY TAG
With no source, the node installs the bytes already in the module's identity directory
under its module source root. --dist names a release directory on the node by its
absolute path, and it must lie apart from the node's keys and data. --git names a
third party's tagged release in a repository the node reaches over https or ssh. Bytes
from either source land only when they are the module the designation names. The
request carries no system name, vouch, or signature: you file publishers and grants
through /cap/admit (Managing standing grants), and the
module loads only when the node's two-signature gate admits it. I kept trust out of the
request, so nothing a request carries can admit the bytes it names.
Run the verb from your own machine with --node, like every owner verb, or on the
node's host by the break-glass route.
What the node does, in order. Each step that refuses has loaded nothing.
- When the node already carries the identity you named (loaded from its identity
directory, recorded in its module manifest, and admitted in its vouch store), it
answers
already-currentand does nothing else. - It refuses the request when a newer instance of the node holds the instance fence
(
fenced), and when the node serves real names (serving-real-names). - It fetches or copies the release you named, giving a fetch 300 seconds, and lands it only when its bytes are the identity you named.
- It takes the module change lock, so one change runs at a time; a request that waits
more than 5 seconds for it is refused (
change-in-progress). - It tries the change in a child process of its own binary within 300 seconds: for a
new module the
try-installcheck above; for a module it already runs, thetry-supersedecheck, once a comparison of the two versions finds no change a running image cannot make in place. It goes on only when the child loaded the module. - It checks the fence again, then its gate checks the publisher's vouch and your grant and loads the module into the running image. It records the module in the module manifest, so a restart stands it up again, and puts any edge adapter a new module registered in front of the running edge, with no restart.
The reply. valis module install prints the node's reply as it came: a token on the
first line, then KEY VALUE lines, system and designation first.
| Token | What it means | What to do next |
|---|---|---|
installed-serving |
The module was admitted, loaded, and recorded. Each adapter line names an edge adapter answering now, then its ports. |
Nothing. valis module show SYSTEM reads it back. |
installed-unserved |
Admitted, loaded, and recorded, with nothing answering on the edge. The reason line says why: no-edge-adapter (the module registered none), edge-not-running (the edge binds it when it next starts), or attach-refused (the edge would not attach it). |
For attach-refused, read the journal line naming the adapter, below. Otherwise nothing. |
installed-unrecorded |
Admitted and loaded, so it runs now, but its record could not be published. The reason line says why, and runs-now yes and after-restart absent say that a restart does not stand it up again. |
Put right what reason names, then restart the unit before you ask again. Until a restart the node refuses another request for that module as not-carried. |
superseded |
The version the node ran, previous, was replaced in place, tentative until-restart. recorded says whether the module manifest names the new version. residue counts the old version's code the scan found still alive, or reads unscanned. |
Restart the unit when you need the image to match its record. With recorded no, a restart returns to the version the manifest names. |
already-current |
The node already carries, records, and admitted that identity. Nothing changed. | Nothing. |
After installed-unrecorded the verb also writes
valis: the module runs now and will not return after a restart on standard error.
Exit statuses.
- 0: the node answered with one of the five tokens.
- 1: a fault on this machine, or a reply this client cannot read.
- 2: refused with nothing installed: a command line the verb does not take (printed
with its usage), a refusal before your key was sent, a key the node gave no owner
view, a node with no
/modulesdoor, or a node refusal, printed asvalis: the node refused:and the node's text. - 3: the node could not be reached, or the session with it failed.
- 7: the install failed and code of the module may have run: the gate admitted it and its load failed, or the node's text does not say how far it got.
When the session failed or a fault on this machine stopped the verb after the request
was sent, the reply was lost, and a second line says
valis: module: the request may have reached the node; run valis module list to see what it holds before you ask again.
A reply this client cannot read prints only its own line.
Refusals. A refusal is an error beginning module install refused, and nothing of the
module was loaded. Apart from a malformed request or a caller who is not the owner,
neither of which the verb sends, the reason follows in parentheses:
not-a-designation: the request names no module.not-a-source: the request names no source an owner may name.no-source-root: the node has no module source root.fenced: a newer instance of the node has taken over.serving-real-names: the node declares an edge domain or is authoritative for zones held on someone's behalf. The text readsthis node serves real names, by its declared names, and a module recorded here would be stood up at every restart with nothing to fall back to, and names which:by its declared namesorby its owned zones.fetch-timed-out: the fetch ran out of time.not-the-requested-identity: the release holds other bytes.staging-refused: the release could not be fetched or staged.conflict: its identity directory already holds other bytes.vouch-refused: the node refused the vouch the release ships.bytes-not-found: no identity directory holds the module.not-one-system: the identity directory holds no system definition, or more than one.change-in-progress: another module change held the lock for more than 5 seconds.dry-run-unavailable: the child could not be asked, or did not answer within 300 seconds.dry-run-refused: the child answered that the module did not load.install-refused: the two-signature gate refused it, and the text after the reason says why.- For a module the node already runs:
not-carried(the running version was never admitted as a module, or an unrecorded change awaits a restart),same-identity,publisher-changed(the new version comes from another publisher),restart-required(a change a running image cannot make in place), anddry-run-failed.
A load that failed after the gate admitted the module is not a refusal: its error begins
module install failed after loading, the image is recorded as diverged, and a later
request for that module is refused as restart-required until you restart the unit.
The journal. The node logs each outcome, so journalctl -u valis.service shows what
it did with a request:
module /module/<hex> landed at <module-source-root>/<hex> module <system> installed as /module/<hex> module <system> (/module/<hex>) installed for the owner: installed-serving module <system> (/module/<hex>) is already current for the owner module <system> superseded, tentatively until a restart, from /module/<old> to /module/<new> module <system> superseded for the owner, from /module/<old> to /module/<new>
and at WARN, for what did not go as asked:
module <system> not installed from <directory>: <reason> module <system> installed from <directory> and running, not recorded: <reason> module <system> (/module/<hex>) installed for the owner and not recorded: <reason> the request for /module/<hex> was refused (<reason>): <cause> the module change for /module/<hex> was refused (<reason>): <detail>; nothing was loaded the module change for /module/<hex> failed after the gate admitted it (load-failed): <detail>; code of the module may have run in this image module adapter <name> was admitted and does not answer: <reason>
A supersede whose record could not be published logs its line with , not recorded
at the end.
How long an install holds its session. The verb waits for the node's reply, and the node answers only once the change is done: up to 5 seconds for the module change lock, up to 300 seconds for a fetch, up to 300 seconds for the dry run, and then the compile and load, which have no bound. Keep the session open until the reply arrives.
From a 9P client. The door is /modules/install in the owner's session. Write the
request as newline-framed lines, install, the designation, and then optionally the
source (here, dist and the absolute directory on the next line, or git and the
repository and then the tag), then read the reply, then clunk. The first read runs the
request and the fid keeps its outcome. A clunk before any read still runs it, but the
reply is lost. A session that is not the owner's finds no /modules.
⚠ Where this stands. A node that serves real names refuses every install
(serving-real-names, above) until a node can fall back from a module that fails to
stand up. Nothing retires an installed module or rolls one back on a
running node. ubik does not yet compose this door, and an install from a second machine
against a deployed node is not yet proven.
13.2. Reading the module set
valis module list prints one tab-separated line per module the node lists: system,
recorded, loaded, divergence, and boot. valis module show SYSTEM prints the five files
the node holds for that module, identity, generation, files, explain, and
state, each name on its own line and its lines indented beneath it. Both read and
change nothing, take --node like every owner verb, and exit as install does, without
The node serves these as read-only files under /modules/manifest/SYSTEM/, for the
owner alone. There is one directory for each module the module manifest records,
resident modules included, and one for each module the image loaded with no record
naming it, as installed-unrecorded leaves one.
identity- the recorded designation,
resident, ornone. generation- the generation the manifest pinned when it was published. It describes; nothing checks it.
files- the record's relative paths, one a line.
explain- which two signatures admitted the module, and when:
publisher(the did:key its vouch is rooted at),owner-credential(per-module-grantorstanding-grant),grant(the grant's hash), andadmitted-at(a Unix time), eachnonewhen unknown, andpublisher unreadablefor a vouch that does not decode; orresident yes. state- the running image beside the record:
recorded;loaded, the designation the module was loaded from,resident,unknown, orno;divergence, the reasons the image's divergence record holds, ornone;boot,ok,refused, ormanifest-unreadable; anddiverged,yesorno.
diverged yes means the running image is not what a restart would build, or the boot
left the module out; divergence and boot say which. A boot refusal shows its kind
alone, and its text stays in the journal and on the status line, since it can name a
host, an address, a key, or a path. No file carries vouch text, grant text, or key
material beyond the publisher's did:key. A manifest the node cannot read is an error
beginning module manifest unreadable and naming the kind of failure, never an empty
listing.
/modules is the module set: which modules run, from which bytes, and why. One
module's settings are somewhere else, under /config/modules/SYSTEM/, which
valis config reads and changes with --module SYSTEM (see
Changing a setting). /modules sits in the owner's own
session and never under the node root, so no grant, bearer name, or delegated token
reaches it. It is one letter from /module/, the prefix designations are written
under, and the two never meet: no directory is ever made at /module/.
14. Backup and restore
The backup-critical set is irreplaceable: the owner Ed25519 seed (0600), the
store head, the store blocks the head's root tree reaches, the vouch and revocation
stores, the grant audit trail beside them (see
Managing standing grants), the owner keyring of enrolled device keys (see
Enrolling and retiring an owner device key), the node's configuration (its own settings and every module's, under
<data-root>/config/), and the ACME account and leaf keys. Losing the owner seed is
losing the node's sovereign identity; there is no re-mint.
Run both verbs as the run account, never as root. The configuration directory is
private to the account that owns it, and a backup reads the grant audit trail only
when the account running it owns the file, so a root-run backup is refused at
whichever of the two it reaches first, before anything is written. A restore
run as any account other than the one owning --state-dir (or, for a directory not
made yet, the nearest directory above it) is refused before any write, with exit
code 2, because the node could not open the files it would land. A restore run as
root into a directory that does not exist yet is refused the same way, since
whatever it made would belong to root. The same holds for --acme-store-dir.
The valis binary carries the backup and restore verbs. Each seals or opens a single
encrypted, self-contained artifact, not a copy of the whole StateDirectory. The
passphrase is read interactively (or piped on standard input for an unattended run);
it is never a command-line flag, so it never lands in shell history or /proc.
Back up. Seal the backup-critical set into one artifact:
sudo -u valis env XDG_DATA_HOME=/var/lib/valis valis --backup --out /path/to/valis-backup.sealed # prompts: Backup passphrase:The artifact is small. It carries the node's configuration as written, so it carries what that configuration names: absolute paths such as the ACME store (
acme.store-path,/var/lib/valis/acme/on a standard install) and the module source root, and the node's own domains and addresses. It carries no database password: the configuration never holds one. A memory-hard key derivation (argon2id) turns the passphrase into the key, and authenticated encryption (ChaChaPoly) seals it: a tampered artifact is rejected rather than silently restored. Treat it as you would a private-key store, and store the artifact and the passphrase separately. An owner seed that is absent, or that is not exactly 32 octets, stops the backup with exit 1 before anything is sealed. A backup can also be driven remotely by the authenticated owner over the/backup/ctl9P door (backup <passphrase> [name];statusreports the last outcome). The passphrase rides one ctl line over the sealed owner session (never cleartext on the wire, never shell history) and the artifact always lands in the node's backup directory,<data-root>/backups/(/var/lib/valis/valis/backups/under the shipped unit), under NAME or, with no name, a timestamped bundle. NAME is a bare file name: a path, a name starting with a dot, or a name already present there is refused with anerror destinationreply, so no session can aim a sealed copy of the seed elsewhere. To write the artifact to a path of your choosing, runvalis --backup --out <path>on the host. A restore is deliberately not served over 9P: it requires the unit down and a pristine state directory, so it staysvalis --restoreon the host.Restore. Open the artifact into a pristine state directory. When the state directory does not exist yet, make it as the run account first; starting the unit once also has systemd make it:
sudo install -d -o valis -g valis -m 0700 /var/lib/valis sudo -u valis valis --restore --in /path/to/valis-backup.sealed --state-dir /var/lib/valis # prompts: Restore passphrase:The restore lands the configuration first, then places the ACME store where that configuration says the node keeps it. The ACME store must lie inside
--state-dir: one the configuration places elsewhere is refused before any write, with exit code 2, so a restore never overwrites certificates outside its target. To restore somewhere else, such as a scratch directory for a proof, name where the ACME store lands with--acme-store-dir:sudo -u valis mkdir -m 0700 /tmp/restore-proof sudo -u valis valis --restore --in /path/to/valis-backup.sealed \ --state-dir /tmp/restore-proof --acme-store-dir /tmp/restore-proof/acmeThe restored configuration still names the original ACME store, so a node you boot from a proof directory looks for its certificates there; the proof checks the keys under
--acme-store-dir. A non-pristine target (one that already holds valis state, configuration, or an ACME store where this restore lands one) is refused up front, before any write, with exit code 2; a wrong passphrase or corrupt artifact fails cleanly with a single message and exit code 1. The normal resident boot then condenses the durable namespace from the restored head: restore is the first-deploy path run from a saved head instead of from genesis, and a faithful restore reproduces the origin's durable identity byte-for-byte. Deploy, evacuate, and restore are one condense mechanism, see the contract's deploy/evacuate/restore section.The keyring on restore. A restore lands the owner keyring with its retirements, so a key retired before the backup was taken stays retired, and it prints every key it restores, live or retired, for you to review:
valis: restored owner key did:key:z6Mk... label=laptop live
A restore cannot know of a key retired after its backup was taken. Retire any such key again once the restored node is up. A backup taken before the node kept a keyring restores the seed alone, with a warning saying the node admits only its own key until you enrol your devices again. A binary older than the keyring refuses a backup that carries one, before writing anything. A keyring the backup cannot read whole, or one written for another owner, stops the backup before anything is written, and one a restore cannot land rolls the whole restore back.
The grant audit trail on restore. Backups carry the trail from container format 6. A restore lands it right after the revocation store and appends one
restoredentry, so the trail shows when the node came back. A backup in an older format restores with this warning, and the restored trail starts at the restore:valis: WARNING: this backup is in container format 5, which cannot carry the grant audit trail. The restored node's trail starts at the restore: when each grant was filed and reviewed before it is not known.
Each active standing grant then has no known filing and no review, so each is due a review at once: review them again (see Managing standing grants). A target that already holds a trail is refused as not pristine unless the backup carries one, which then replaces it whole. When the
restoredentry cannot be written, the restore stands and a warning beginningthe restore stands, but its restored entry was not written to the grant audit trailsays so. A backup that cannot read the trail is refused before anything is written.
The restore-proven backup is the go-public gate, and nothing else enforces it. Do not cut a node over to a public address until a restore has been proven: restore the artifact into a scratch state directory and confirm the three properties a good restore demonstrates (the owner axis resolves, the store generation matches the source, the TLS keys load). Those properties, and why a silently-wrong restore is treated as worse than an obvious failure, are set out in backup-and-observability.org.
15. Restart, stop, and teardown
Restart is just re-running bring-up: it is idempotent. The single-writer instance fence makes a re-claim of the current generation a no-op, so recovery is safe to repeat.
systemctl restart valis.serviceRestart as the owner with
valis restart, when you hold the owner key, or when the node has told you a restart clears threads a retired module left running. The node asksfulcrumto have systemd restartvalis.service, so it is the same stop and start assystemctl restart, with the same few seconds without service. Run it from your own machine and name the node:valis --node NODE restart- It waits for the new boot. After it asks, the verb waits, 120 seconds by default
(
--wait SECONDSsets the bound), until the node answers again over the same pin from a boot identity other than the one it had when asked. It then printsvalis: the node is back, bootwith the new identity, and exits 0. The owner port keeps its address across the restart, so the wait needs no shell on the host. - Every owner session ends. The restart ends every session the node held, including ubik's. Each valis verb opens a session of its own, so the next one you run attaches to the new boot. ubik attaches again on its own once the node is back, checking the pin as it does, and every panel reads the node again. When you restart the node from ubik's restart panel, the panel shows the node's boot identity before and after; Managing a node from ubik covers the panels.
- Exit statuses. 0: the node came back from a new boot, or you passed
--wait 0and the restart was accepted. 2: the node refused, or the client refused before asking (for example, for a key the node has not enrolled), so nothing was asked. 3: the node could not be reached to ask, or did not come back within the wait; the message says which. - A refusal says why. A node running without a supervisor that can restart it, such
as a development image or
--run, says so and tells you to restart it the way it was started. Whenfulcrumis too old to serve the restart request, the refusal says it cannot restart the node: restart the service on the host withsystemctl restart valis.service.
- It waits for the new boot. After it asks, the verb waits, 120 seconds by default
(
Restart as the owner from the node's host, the break-glass route, when you cannot use the route from your machine (the owner port closed, the pin or the device key lost, or the node not answering on the owner port). Run the verb inside the resident's namespace as the run account (see the break-glass route), and confirm from the host:
systemctl show -p MainPID -p ExecMainStartTimestamp valis.service sudo nsenter -t "$(pgrep -o -x valis)" -n -S "$(id -u valis)" -G "$(id -g valis)" \ /opt/valis/bin/valis restart --wait 0 \ --keyfile /var/lib/valis/valis/keyfile \ --endpoint "$(sudo cat /var/lib/valis/valis/mgmt-endpoint)" systemctl show -p MainPID -p ExecMainStartTimestamp valis.service- Pass
--wait 0from inside the namespace. A restart replaces the namespace and draws a new fabric port, so a verb entered into the old namespace, or pinned to the old port with--endpoint, can never see the new boot and would wait out the whole bound. With--wait 0the verb asks, prints the boot the node answered from, and exits. - Confirm on the host.
MainPIDisfulcrum, which every restart replaces, so a differentMainPIDand a laterExecMainStartTimestampafter the restart than before it say the node was restarted. Run any owner verb again through a freshnsenterto see it answering. Where the node's owner port is open and you name it with--endpoint, the address survives the restart and the verb's own wait works, with no root shell at all.
- Pass
Stop drives an ordered teardown. On
SIGTERMthe resident unwinds in reverse bring-up order (long-lived active modules drain first, then edge, then fabric, then listener);fulcrum(theMainPID) removes thesk_lookuplink and tears the netns down.KillMode=mixedlets fulcrum drive that ordered pair-teardown rather than systemd SIGTERMing both at once.systemctl stop valis.service- This stop does not depend on the stop notice.
systemctl stopsignalsfulcrumfirst, andfulcrumends the run on that request and reaps the resident. The resident gives its stop notice as it does on any orderly stop, but the outcome does not depend on it, so a warning about the notice during this stop needs no action. The same holds for an owner restart, which goes throughfulcrumand systemd.
- This stop does not depend on the stop notice.
- A signal sent to the resident's own process is a stop the resident begins itself:
SIGTERMorSIGINTdelivered to the resident directly, for examplekillwith its PID. The bullets below cover this stop only.- The resident tells fulcrum the stop is intended. Before it stops anything, the
resident sends
fulcruma stop notice over its control session, and it closes that session last. Once the session ends,fulcrumwaits up to 70 seconds for the resident to exit and exits with the resident's status; a resident still running at 70 seconds is killed andfulcrumexits 1. The journal showsthe host agent was told this stop is intendedat info level. - A warning about the notice means the stop can read as a failure. An older
fulcrumhas no file to take the notice, and the resident warnsthe host agent serves no stopping file; updatefulcrumto clear it. Any other warning about the notice (a request still held the session, the session had ended, orfulcrumrefused or did not acknowledge it) means the notice did not land this once. Either way the stop itself carries on, andfulcrumgives the resident only 2 seconds from the session's end: a resident that exits in that time has its status passed on, and one still running makesfulcrumexit 1.
- The resident tells fulcrum the stop is intended. Before it stops anything, the
resident sends
- A superseded writer fails closed. Two instances sharing one operator-state database
coordinate only through the single-row write-fence; a superseded writer is fenced out
(the
fenced-outcondition) and writes nothing. This is the split-brain stop that makes a cross-location handover safe. Expect the older instance to fail closed once a successor claims a higher generation.
16. Legacy from-$HOME operator mode
The default run account is a dedicated static valis:valis. A legacy operator running
the resident out of their own account is supported via a drop-in: install
legacy-operator.conf.example as
/etc/systemd/system/valis.service.d/legacy-operator.conf and set the <operator> /
<operator-home> placeholders:
install -d -m 0755 /etc/systemd/system/valis.service.d
install -m 0644 deploy/systemd/valis.service.d/legacy-operator.conf.example \
/etc/systemd/system/valis.service.d/legacy-operator.conf
# ... edit <operator>/<operator-home>, then ...
systemctl daemon-reload
Overriding User=/=Group and pointing XDG_DATA_HOME (and HOME) at the operator's
home keeps valis rooting durable state there; on the first boot after such a move, the
resident relocates pre-XDG ~/.valis state cleanly. To fall all the way back to
~/.local/share, clear the inherited StateDirectory (an empty StateDirectory=).
17. Troubleshooting
| Symptom | Cause / fix |
|---|---|
Error opening shared object … libfixposix.so |
iolib dlopen=s the *unversioned* SONAME. Install the runtime package *and* create the =libfixposix.so symlink (or install -dev). |
c3po's serving codec … not available at boot |
a plain :valis core was staged. Stage the serving-capable core from the make dist delivery tarball instead. |
:443 dark unexpectedly |
check edge.domain is set (v get edge.domain, with the v wrapper from changing a setting) and a cert is in the custody store; read the boot line for which of the two. A genesis dark :443 is healthy. |
setpriv: … Operation not permitted / no privilege drop |
Under NoNewPrivileges the execed setpriv helper keeps only the ambient caps, so emptying AmbientCapabilities breaks the privilege drop outright. Do not narrow the ambient set. |
| DNS boot cannot build its zone index | Postgres must be reachable over its local UNIX-domain socket under peer auth: verify the socket path in operator-state.dsn (valis config get operator-state.dsn, in the running-node or stopped-node form changing a setting gives) exists and the run account's uid maps to the database role. |
Type=notify times out despite the resident appearing to run |
NotifyAccess=all is required (valis is fulcrum's fork-child, not MainPID); and the launch chain must not clear the environment, or $NOTIFY_SOCKET never reaches valis. |
unit restarts but never opens :443 after a cert renewal |
renewal hot-swaps in place through the shared cell: a restart is not required and not the renewal path. Check the renewal manager's custody-store writes. |
new connections are refused while the unit stays active (running) and healthy |
descriptor exhaustion, which no instrument on the node reports: an accept loop that cannot accept is still an alive thread, so the liveness gate holds and the status line still says serving. Count /proc/<pid>/fd against Max open files in /proc/<pid>/limits for the resident, per descriptor headroom above. |
For the architectural why behind each of these, the privilege split, the descriptor
handoff, the sandbox directives that ship vs. those deliberately omitted
(MemoryDenyWriteExecute, LockPersonality), read the annotated
valis.service unit and the host-deployment contract.