valis / Running a node

valis - Backup and Observability

This note is for the operator running a deployed valis host. It documents two security-touching surfaces: how the irreplaceable durable state is backed up and restored, and what the running resident reports about its own health. Both exist so that a host can be recovered onto fresh hardware and watched for silent failure without the operator having to read the source. The code is the record of how each mechanism works; this note is the record of what is protected and why the design is shaped the way it is.

The backup-critical set: what must survive

Almost everything a valis host holds is recoverable. The content-addressed block store is rebuildable from its peers, and a leaf certificate can be re-issued. A small set of state is irreplaceable or expensive to reconstruct, and that set is exactly what the backup captures:

The owner Ed25519 seed, the root of the person's identity. Lose it and the person's did:key, and everything signed under it, is gone. This is the one item that cannot be regenerated.
The store head, and every block the head's root tree reaches. The content-addressed root the resident condenses from, together with the content it names. Restoring the head is what makes a fresh instance come up as this host rather than an empty one. The head alone is not enough to do it: it is a small pointer into the block store, so a node restored from a head without its blocks adopts an identity it then cannot read, and fails closed on the first block it reaches for while condensing.
The ACME store tree, the account key and the per-domain leaf keys and certificates. Restoring these lets the public TLS edge come up immediately, without a fresh certificate issuance round-trip on first boot.

The backup is small, portable, and host-agnostic: it carries no absolute paths or machine identifiers, so it restores onto any host by re-entering the passphrase.

The valis binary carries the two verbs that seal and open this set:

valis --backup  --out /path/to/valis-backup.sealed          # prompts: Backup passphrase:
valis --restore --in  /path/to/valis-backup.sealed --state-dir /var/lib/valis

--backup assembles the set and seals it into the single artifact at --out; --restore opens the artifact at --in into the --state-dir, which must be pristine (a directory that already holds valis state is refused before any write). Both read the passphrase interactively, never as a flag; the mechanics are below.

Passphrase escrow: how the off-host blob is protected

The backup artifact is encrypted before it ever leaves the host. Protection has two parts, and both matter:

A memory-hard key derivation (argon2id) turns the operator passphrase into the encryption key. Memory-hardness is what makes an offline brute-force of a stolen artifact expensive rather than trivial.
Authenticated encryption (ChaChaPoly) seals the plaintext and binds the key derivation parameters into the authentication tag. Any tampering (flipping a byte of ciphertext, or rewriting the stored parameters to force a weaker derivation) fails the tag, so a doctored artifact is rejected rather than silently restored.

The passphrase is never stored anywhere and is never passed as a command-line flag (a flag would land in shell history and in the process's command line under /proc). The operator supplies it interactively, or pipes it over standard input for an unattended restore driven from a controlled channel.

Why the on-disk owner seed stays a plain keyfile

On the running host the owner seed lives as a plain 0600 keyfile, not encrypted at rest. This is a deliberate choice, not an oversight. Encrypting the seed at rest would require an unlock secret to be present at every boot, which would either break unattended restart (the host could not come back up on its own after a reboot) or force the unlock secret to be stored on the same host, giving no real protection. The security boundary is therefore drawn at the host's own file permissions for the resident copy, and at the passphrase for the off-host backup blob. The blob is the only ciphertext; it is the copy that travels, so it is the copy that is encrypted.

Restoring into a fresh instance

Restore is the same operation as first deployment, run from a saved head instead of from genesis: the resident condenses the durable namespace from the restored state rather than building an empty one. The operator stages the encrypted artifact on the fresh host and runs the restore, entering the passphrase when prompted. A wrong passphrase or a corrupt artifact fails cleanly with a single message. The operator is never shown which of the two it was, and no partial state is written.

A restore is only trustworthy if it is proven before the host is exposed to the public internet. A good restore demonstrates three properties, and the deployment proving ground asserts all three against a real restored instance:

The owner is recognized: an owner-only axis of the namespace resolves, proving the seed landed and the identity is intact.
The store generation matches the source: the restored head carries the origin's generation, so the fresh instance is byte-faithful to the host it was taken from, not a divergent copy.
The TLS keys load: the ACME account and leaf keys are present and usable, so the certificate-gated edge comes up serving rather than dark.

A restore that copied the head faithfully but dropped the TLS keys, or whose owner axis did not resolve, fails this gate: a silently-wrong restore is treated as worse than an obvious failure.

Passphrase strength is the operator's responsibility

The cryptography bounds how expensive an offline attack on a stolen backup is; it cannot rescue a weak passphrase. Because the memory-hard derivation only multiplies the attacker's cost by a fixed factor, the practical security of an off-host backup rests on the operator choosing a strong, high-entropy passphrase and storing the artifact and the passphrase separately. Treat the passphrase as the master secret it is.

The observability surface

A deployed resident reports substrate facts about itself so the operator can see that it is genuinely serving. Three independent surfaces carry that reporting, and a fourth condition is set out below that none of them sees. I name the gap here rather than leave the three to imply a coverage they do not have: a surface trusted for a warning it cannot give is worse than no surface at all.

Structured substrate logs to the journal. The resident emits one structured line per lifecycle event (boot progress, certificate issuance and renewal, backup and restore outcomes, watchdog state) captured by the system journal and queryable there. These lines carry substrate and lifecycle facts only: they never contain message bodies, namespace contents, or any owner data. Logging is a health signal, not a data exhaust.
Health and liveness. The resident tells the service supervisor it is alive only while three things hold together: the accept loop's thread is alive, the operator-state database is reachable, and the fabric's listener is running. A serving thread that has died, or a lost database, stops the liveness signal, and the supervisor restarts the host rather than leaving it "running" while dead. The gate is a liveness gate and not a capacity one, which is the distinction the section below turns on. A periodic status line additionally reports the serving state, the days remaining on the soonest certificate, and the current store generation.
Independent certificate-expiry warning. A separate check watches the certificates against their own validity, independent of the renewal machinery, and raises an error-level warning if a leaf is approaching expiry without having been renewed. This is a deliberate belt against a silently failed renewal. A stale certificate is a condition to surface, never a reason to restart the host, so this warning is kept off the liveness path.

Descriptor exhaustion, which none of these surfaces sees

A resident that has run out of file descriptors goes on reporting itself healthy, and every instrument above stays green through it. The liveness gate asks three questions: is the accept loop's thread alive, is the operator-state database reachable, is the fabric serving. An exhausted resident answers yes to all three. The thread is alive and looping, the database is reachable, the fabric's listener is running. Each new connection fails on its own, the watchdog goes on being pinged, the supervisor is given no reason to restart anything, and the status line keeps saying the node is serving, which is true: it is serving everyone it already has and nobody new.

Nothing in the resident gauges descriptor headroom today. There is no count in the journal and no field in the status line, so this is checked from outside the process, by reading what the kernel already knows about it. The operations runbook carries the reading to take and what a leak looks like across two of them. I am not writing this from a hypothetical. valis shipped a descriptor release that could never succeed, in an error path, and no surface above would have shown it: reading the code is what found it.