valis / Direction

valis - Build TODO

The collaborator-facing status board for valis. It records where valis stands, what is done, what is next, and the order the pieces land in, across every delivery arc, not just the first. Read it alongside the design documents and the development guide:

the design of record
docs/ARCHITECTURE.org
working with the code
docs/DEVELOPMENT.org

The architecture is authoritative for how valis is shaped; this file is authoritative for where valis stands and delivery order and progress.

How to read the marks. An item is [ ] until it is merged to main, and [X] once it is. Progress moves the same way the code does: while a piece is open it names its feature branch and PR on an On branch :: line, and the tick flips to [X] (echoing to main) when that branch merges. So this file read on main always reflects what main actually carries; a feature branch carries its own items ticked ahead of main until it lands.

Where valis stands

valis's founding intent is a Lisp Machine for network protocols: a sovereign, malleable substrate that multiplexes one person's data, identity, and protocols, made durable as a namespace rather than a process so it can move between hosts. That thesis unpacks into five claims; here is where each one actually stands, read off the code rather than the aspiration.

  • Sovereign substrate: built and proven. The 9P namespace keystone (view roots, two-phase resolution, enforcement-by-mounting, union directories, read-only projection, revocation eviction) is the largest subsystem and the spine everything hangs off. Sovereignty is the absence of a name, set once at namespace construction, never a per-access guard.
  • Identity and capability: built and proven. NoiseXX transport identity, did:key principals, a custody store that is the sole holder of key material, UCAN-mould delegable tokens, and Tahoe-style bearer capability-names, all offline-verifiable. The transport cipher itself now lives in the mercer sibling; valis is plumbing over an opaque session.
  • Migratable namespace: built and proven. A content-addressed, venti-style block store under a generation-fenced fossil head; reassembly-not-transfer migration with evacuation fencing; and the code-side symmetry: a module manifest that reassembles the running protocol set from owner-vouched hashes.
  • For network protocols: one proven end to end, the rest externalised by charter. A conformant HTTP/1.1 server answers on the steered :80 with identity-selected views on one URL space. The wire engines themselves now live in the c3po sibling (HTTP and Gopher adopted, SMTP/IMAP/DNS following); valis core deliberately names no concrete wire engine. The mail-transport spine and the authoritative-DNS substrate are built here as substrate-plus-seam.
  • A Lisp Machine all the way down: the resident path is live on a real host; the public-TLS spine is the arc in progress. The eBPF sk_lookup steering is compiled from Lisp by Whistler and proven end to end on a real kernel in the privileged fulcrum sibling. valis consumes the steered descriptor in its own process (minting, receiving across a fail-closed boundary, and booting its edge on it, with a race-free wire push), and the privileged launcher brings the unit up under a real network namespace: the resident serves public authoritative DNS on :53, proven end to end off-host in a real-kernel integration gate (worldwide multi-region reachability has its own harness, gating the actual go-live). ⚠ DNS does not arrive by the steer. This is the one sanctioned departure from the one-socket model. sk_lookup cannot carry UDP at all, and a UDP reply must source from the socket it was received on, so fulcrum binds :53 on both transports inside the namespace and hands the descriptors down across the exec. valis still performs no privileged bind. This is a necessary aberration from the one-socket model rather than a second sanctioned way to reach the edge: every other port is steered, and a new protocol requiring its own inherited descriptor is requesting an exception, not following the rule. The ACME/dns-01 spine, server-TLS termination, the public HTTPS edge on :443, and the systemd deployment wrapping (a Type=notify service around the fulcrum launcher with per-tier sandbox hardening, proven on a live Debian cold-boot VM) are all landed. Deployment is framed and tested as condense-from-genesis (the proving ground). The go-public gate (a restore-TESTED backup of the irreplaceable owner seed, structured journald logging, and watchdog liveness) is landed and proven green on a Debian guest: a restored node condenses from its backup and serves authoritatively. The owner control plane landed: zone, obtain, publish and apply each drive a running resident, owner-keyed over the loopback fabric, so a node is updated and published to over an authenticated path rather than by editing files on the host. So did the live-host cutover: valis is the primary authoritative server for the registered domains, the lame delegation is closed, a sibling host runs as an AXFR secondary, and zone data lives in PostgreSQL as operator state rather than in files. Connections that outlive a request are supported: the edge no longer assumes a client-speaks-first, reply-only protocol. A production certificate is issued and serving: the public :443 answers for deepsky.systems under a Let's Encrypt chain that verifies off-host. The certificate machinery behind it is complete: one credential held per served name, a renewal proven end to end against an authority that resolved the challenge for real, and a deploy that refuses before it moves anything a node cannot run. ⚠ The machinery is in the tree; the live host is behind it. It is authoritative for four zones and holds one certificate, because it is running an older build. Closing that gap is a deployment to a node serving real domains, not a build. What remains on the go-public arc is narrow and specific: that deployment, a cold verification that steered traffic reaches the accept path with the original port recovered, and a TLS-terminated connection the event loop can drive, which is what a continuous protocol over TLS needs and which gates owner mail access.

The shape of it: the hard, novel part (the substrate that makes every protocol a uniform, capability-gated, user-owned plugin over a migratable namespace) is done and proven by one real protocol on a real kernel. What remains is breadth (more protocols, now a c3po exercise against a versioned seam), the live-internet host (the host is up and authoritative for the registered domains; the ACME/TLS spine, the :443 edge, the supervised systemd deployment, the owner control plane, the go-public gate (restore-tested backup, observability, watchdog) and a production certificate on the public :443 are all landed; what remains is deploying the current tree so every served name is answered under its own certificate, the cold steered-delivery verification, and a TLS-terminated connection the event loop can drive), and one genuine external unknown (native content-addressed transport). The arc in progress and the horizon items at the foot of this file track each of those.

Scope of the first arc

The first arc builds the substrate as horizontal layers and then assembles them into one real protocol: it extends the proven steering foundation into a real listener, stands up the 9P namespace fabric, lays the full capability seam, models a minimal semantic substrate, lands real eBPF-steered file descriptors, and finishes by answering one IETF protocol (HTTP on :80) end to end as a standards-compliant server that serves the anonymous published view and the owner-proven management view on the same port, selected by identity.

Beyond this arc: the durable store and migration (fossil-over-venti, single-writer-per-subtree, generation fencing on evacuation) - now under way as the second arc, tracked below - and, later still, further IETF protocols and content-addressing (libp2p/IPFS). These are designed in docs/ARCHITECTURE.org.

Foundation (done)

  • [X] Protocol-plugin contract: the protocol class and the handle-connection generic every plugin specialises. (src/protocol.lisp)
  • [X] Port → protocol registry and the dispatch entry point. (src/registry.lisp)
  • [X] Process lifecycle: run / daemon / dev modes. (src/main.lisp)
  • [X] sklookup steering, proven externally: compiled, kernel-verified, and attached to a network namespace via the pure-CL loader, in the Whistler fork.
  • [X] Userland recovery of the original destination port via getsockname: proven and kernel-independent over loopback.

Build sequence (in order)

The listener and the capability layers can be built over loopback before real kernel steering lands: getsockname port recovery works identically without the kernel in the loop. The privileged host agent is independent of the capability and substrate layers and can proceed in parallel.

  1. [X] Event-loop listener. An iolib(epoll) multiplexer accepts steered connections, recovers each connection's original destination port, and hands it to the existing registry: no synchronous accept loop in the path.
    Done when
    a client dialing several loopback ports is accepted by one event loop, each recovered port matches the dialed port, and each connection routes through the registry.
    Note
    the event source sits behind valis's own multiplexer interface (so an iouring backend can replace epoll later) and accepts datagram as well as stream sources (sklookup steers both TCP and UDP).
    Depends on
    nothing - extends the proven port-recovery foundation.
  2. [X] 9P fabric and bus: consume seven. valis exposes a person's data, identity, and names as a mountable 9P2000 namespace served by seven, and carries inter-module and inter-node dispatch as 9P file operations.
    Done when
    a 9P client walks the namespace and reads a published file; one module reaches another's mailbox as a 9P file identically whether the target is a local thread or a remote node; and the host kernel's v9fs mounts the namespace so ordinary tools (ls, cat) traverse it.
    Depends on
    seven, the sibling 9P library, built first in its own repo.
    Delegatable
    seven is a self-contained sibling repository (the 9P2000 client + server), a natural standalone work unit. Consumed: namespace, bus, and host v9fs mount all verified; seven evolves on as a sibling.
  3. [X] Transport identity and factotum agent. A Noise-style handshake yields a principal bound to a peer key/DID (or anonymous for a keyless request) with no explicit login step; a factotum-shaped 9P agent is the sole holder of the owner's keys, so protocol modules never see key material.
    Done when
    a keyed peer is presented to the rest of valis as a principal; a keyless peer is presented as anonymous; and the factotum agent is a 9P service holding all keys, handing modules only a resolved principal.
    Depends on
    the 9P fabric (the factotum agent is a 9P service).
    Landed
    a NoiseXX handshake over the 9P auth path binds the keyed peer's did:key principal to its connection; a keyless attach resolves to anonymous; the factotum directory (/id) is the sole key custodian, reached only as a 9P service. End-to-end handshake, anonymous-attach, and key-stability tests are green.
  4. [X] Capability tokens and names. Authority is carried as UCAN-mould tokens (delegable, attenuatable, public-key-rooted, offline-verifiable, with the owner as the authorization server), and published data is named by a Tahoe-style capability that embeds its own access right and key.
    Done when
    a token can be minted, attenuated, delegated, and verified offline with no central server; an attenuated delegation grants strictly less and reads back as provenance; and a capability-name yields exactly the access it encodes, with no identity required of the bearer.
    Depends on
    transport identity (tokens and caps root in principals).
    Landed
    tokens mint, attenuate, delegate, and verify offline against the owner's key; attenuation is strictly narrowing and the delegation chain reads back as provenance; capability-names grant exactly the embedded right with no bearer identity required.
  5. [X] Enforcement by mounting and two-phase resolution. Resolved authority is enforced by what is mounted into a connection's namespace (a module can name only what is mounted, with no per-access guard and no ambient authority), and authority resolves in two phases: a connect-time principal sets a base namespace, and a per-request capability may mount further within it.
    Done when
    a connection can name and reach only what is mounted (an uncovered resource is absent, not merely refused: sovereignty is the absence of a name); there is no per-access check on the hot path; and a request-scoped capability mounts a further subtree without granting ambient authority.
    Depends on
    capability tokens and names. This closes the capability seam.
    Landed
    the assembler builds each connection's view from its resolved grants; an ungranted name is absent from the tree, never refused after admission; read-only grants wrap the target in a server-side projection so write denial is structural, not adapter courtesy.
  6. [X] Minimal semantic substrate. The substrate models the semantic object (a publication / file) independent of any wire protocol, and the first protocol projects over it as a wire-format adapter gated by the connection's capabilities.
    Done when
    a publication exists as a semantic object in a 9P subtree, independent of any protocol; a wire adapter reads it only through the mounted namespace; and it is reachable only where mounted.
    Depends on
    enforcement by mounting (the substrate is gated by the seam).
    Landed
    a publication is a directory of attribute files under /pub, backed by a swappable store seam (the v2 fossil/venti replacement point); two wire adapters (HTTP/1.0 and Gopher) project the same object read-only through capability-mounted views over a real in-process 9P attach, and absence reaches the wire as the protocol's own absence response. Proven end to end over real TCP: both adapters served the same 9P-authored publication. See the publication sections of docs/ARCHITECTURE.org for the seam contracts and validation map.
  7. [X] Edge↔core seam and anonymous scoping. The IETF edge is port-addressed (the recovered destination port selects the module), and an edge module reaches the sovereign core only by carrying its connection's principal and capabilities across a 9P/capability seam (co-located in one image now, sandbox-ready later, never a shared-memory assumption); an anonymous caller reaches a narrow, capability-scoped namespace, never ambient authority.
    Done when
    the recovered port alone selects the module; an anonymous caller can name nothing of the owner's authority; and the same edge module works co-located in-image or split across a pipe with no code change.
    Depends on
    the minimal semantic substrate.
    Landed
    the recovered destination port alone selects the adapter (HTTP/Gopher), which reaches the core only by carrying its connection's principal and capabilities through a 9P seam: the same adapter serves byte-identical in-process, over loopback TCP, and over a socketpair with no code change. The fabric mints one anonymous read-only /pub grant; an anonymous caller reads only that view and every out-of-scope axis (/proto, /id, /cap, /edge) is absent from its wire, never merely refused. Request-carried capabilities mount into a fresh private per-request view; per-port connection budgets apply fail-closed backpressure (overflow waits in the kernel backlog, never accept-then-close); a read-only /edge axis renders live per-port status.
  8. [X] Privileged host agent and real steering. An external host agent provisions eBPF sklookup steering so every designated port on the unit's IP fans out to valis's single socket, and hands valis its steered listening fd; valis runs unprivileged and never binds the designated ports itself.
    Done when
    a connection to a designated port on the real IP is steered to valis's socket and accepted with its original port recovered; valis holds no privileged kernel state; and the agent steers exactly the declared set of ports, nothing outside it.
    Depends on
    the event-loop listener (it consumes the listener's fd). Independent of items 2-7, parallelizable.
    Delegatable
    a self-contained systems unit (eBPF provisioning, the designated-ports contract, and the fd handoff) for someone comfortable with eBPF and Linux network namespaces. It replaces loopback with a real port.
    Landed
    fulcrum, a privileged sibling agent holding CAP_BPF=/=CAP_NET_ADMIN (the only privileged component: the privilege boundary drawn as a repository boundary). valis declares its ports over a 9P control plane, then hands fulcrum its LISTEN descriptor over a fail-closed, validated SCMRIGHTS receive; fulcrum populates the sockmap before attaching the catchall sk_lookup program. valis stays unprivileged and binds only loopback. Proven on a real kernel by fulcrum's make steer-test netns gate: a connection dialed at the designated port lands on valis's single steered socket with the original dialed port recovered (four-tuple preserved under sk_assign), valis never binds the designated port, and the netns is the boundary: no leak.
  9. [X] First IETF protocol end-to-end: HTTP on :80. valis answers HTTP as a standards-compliant server on its steered port, serving the anonymous published view and the owner-proven management view on the same port, selected by identity, against a real namespace through the full capability seam. The capstone that assembles every layer below.
    Done when
    an ordinary GET returns a well-formed HTTP response; an anonymous GET returns the published view while the same port with the owner key proven returns the management view; and the owner view is reachable only by proving the owner key.
    Depends on
    the edge↔core seam and real steering.
    Landed
    the HTTP/1.0 adapter grew into a conformant-but-bounded HTTP/1.1 server: Host required (400 when absent), GET + HEAD, persistent keep-alive with a per-request read-deadline, conditional GET304 mapped onto the 9P qid.version as a strong ETag, and the real status taxonomy with Date=/=Server headers. Faithful HTTP, not a skin. The owner proves possession of the Ed25519 custody key per request via RFC 9421 HTTP Message Signatures, verified as a third method on the authenticate seam; the edge relays only the signed material and never sees the key. One URL space, identity selects the mounted namespace: an anonymous GET resolves to the read-only /pub published view, an owner-signed request to the full canonical frame as a browsable read projection, and an anonymous caller names nothing of management (404, never 403). The same anonymous-/pub published view is now consistent across both edges: an unauthenticated HTTP GET and an unauthenticated kernel v9fs mount see the same published floor, while /proto /id /cap /edge /bus stay credential-gated. Replay is bounded (created=/=expires window + a nonce cache committed only after the signature verifies). Proven end to end: the full cold suite is green (1557/0, incl. the real-NIC auth-seam crossing and the kernel v9fs mount), the auth surface is threat-audited (09-SECURITY.md, no open threats), and the operator make steer-test gate answers a real client on the steered :80. The seven dependency floor is pinned at the version that carries the close/EOF and node-qid capabilities valis requires. The first delivery arc is complete.

Where work can be delegated to collaborators

Through the first arc, the core foundation - listener, 9P fabric, the capability seam, and the semantic substrate (items 1-7) - stays architect-held: it is the spine, and the dependency ordering is reviewed before each piece begins. Delegation opens at the edges and after:

  • seven (the 9P library): already a separable sibling repository and the keystone dependency for item 2. The clearest standalone unit; in progress.
  • The privileged host agent (item 8): independent and parallelizable; a contained eBPF/Linux systems task.
  • The IETF protocol modules: the prime collaborator surface, opening once the substrate interfaces and the first protocol (HTTP on :80) exist. Each further protocol (SMTP, IMAP, NNTP, DNS, and the rest) is a self-contained wire-format adapter over an already-established semantic subtree, gated by capabilities, addable without inventing new storage or identity machinery.
  • libp2p/IPFS bindings (a later milestone): a separable CFFI-binding sibling repository for content addressing.

Second arc - durable store and migration (complete)

The second arc gives the sovereign namespace a durable, migratable backing store: content-addressed immutable blocks under one generation-fenced mutable head (fossil-over-venti), single-writer-per-subtree authority, reassembly-not-transfer migration with evacuation fencing, and coincident-instance caching that preserves read-your-writes. It is built in-tree under src/store/ from pieces valis already owns (no new dependencies, no new 9P wire surface) as an import-clean package cluster a later extraction can move out as files plus an .asd. The on-disk format and algorithms are grounded against the Plan 9 venti source; fossil is absent there, so the mutable head is venti-style content addressing plus a compare-and-swap on a monotonic root, not fossil. The design of record is docs/ARCHITECTURE.org.

Build sequence (in order)

The store is built strictly bottom-up: block, then tree, then the generation-fenced head (the spine), then the manifest, the fabric adapter, reassembly, the fence↔revocation unification, caching, and the end-to-end migration proof. The fossil/venti replacement point named in the first arc's item 6 is the store-store adapter of item 14.

  1. [X] Content-addressed block store. A write-once, hash-addressed, integrity-verified block primitive: the foundation everything else hangs off.
    Done when
    writing octets returns a content address whose hash recomputes from the stored bytes; a corrupted block is detected on read, never served; identical content dedups to a no-op; and a block reported written survives a kill -9 plus drop_caches.
    Landed
    a file-per-block store over portable Lisp stream IO with the proven fs-store fdatasync→rename→dir-fsync durability discipline, SHA-256 behind a swappable store-block-digest seam, and an in-memory index plus bloom filter so a miss avoids a seek.
  2. [X] Tree codec. A single hash names a whole subtree; snapshots share unmodified blocks by content identity.
    Done when
    one root hash resolves a directory tree to its leaf blocks and the same logical tree always encodes to the same root hash; changing one leaf writes only new blocks on the path to the root; and a snapshot is an O(1) retained root that copies no bytes.
    Landed
    deterministic, dedup-friendly directory and file node encoding through the block seam, importing nothing from valis's capability/namespace packages.
  3. [X] Generation-fenced mutable head: the spine. A monotonic, generation- fenced root pointer advanced by compare-and-swap, where a superseded write is rejected at the commit and a recovered head always names a fully-durable tree.
    Done when
    a write carrying a superseded generation is rejected at the commit even after an injected stop-the-world pause lands it arbitrarily late (the Kleppmann fencing-race proof); the generation is strictly monotonic, 64-bit, store-allocated; every block is durable before the head names it; and a recovered head needs no repair pass.
    Landed
    CAS-advance of the root pointer via an atomic rename as the single linearization point, validated by a kill -9 + drop_caches + injected-pause fault-injection harness.
  4. [X] Namespace manifest. The mount table made literal, living inside the mutable tree, reachable from the head, so it migrates atomically.
    Done when
    the serialized manifest contains no key bytes and no derivation seed (a structural bar with a regression test); it pins a specific generation and everything it references is durable before it is published; a first boot with no manifest reproduces today's axes exactly; and a manifest with a bad hash or a missing root is rejected fail-closed.
    Landed
    a versioned manifest inside the durable tree referencing only key locations and trust-domain identifiers, defaulting to the live six-axis set so a fresh boot is byte-identical to the first arc.
  5. [X] store-store adapter and fabric wiring. The durable store dropped into the running fabric behind seven's backend node layer: no new 9P wire surface.
    Done when
    /pub serves byte-identical results over the durable store as over fs-store (no first-arc regression); start-fabric selects the durable backend with a one-line swap; a /pub write goes through the head CAS as a single-writer mutation; and the subsystem cold-builds with the new backend selected.
    Landed
    the publication store seam implemented over head and tree, selected at fabric start-up, with mtime/version synthesized from the head generation rather than a POSIX mtime.
  6. [X] Assembler manifest path and reassembly. A fresh instance reconstructs the namespace from the manifest plus the owner's keys: reassembly-not-transfer.
    Done when
    a fresh instance reads head → manifest → roots and reassembles the same namespace; reachability of every referenced root is verified before the namespace is declared assembled, fail-closed on any missing root; a clean install with no manifest reassembles today's axes; and the destination cannot come up without an out-of-band key acquisition step.
    Landed
    the assembler's canonical-frame builder gained a manifest-driven path; keys arrive out of band through the existing factotum/custody path and never travel in the manifest.
  7. [X] Fence ↔ revocation unification. Evacuation fencing and capability revocation collapsed into one append-only monotonic authority log.
    Done when
    standing up a new instance bumps the generation as its first durable act and the store rejects the superseded instance's subsequent writes; that superseded generation is expressed as a revoked write capability on the existing append-only revocation log, checked on the same write path as a revoked hash; live write-capable grants under a fenced generation are swept from active sessions; and single-writer authority is enforced at the store, not by a clock-based lease.
    Landed
    a write capability carries a generation stamp inside its owner-signed bytes and is rejected at verify time on the same membership check as a revoked token, with the head CAS as the commit-point backstop and a write-right eviction sweep for live grants, so fencing and revocation cannot drift apart.
  8. [X] Caching coherence and qid.version. Coincident instances get read-your-writes against the single-writer head without funneling all reads through it.
    Done when
    a write through one instance is visible to a coincident instance within the staleness bound, without the reader funneling through the writer; an immutable-block cache hit is always valid (the score is the validator) and only the head needs revalidation, keyed to the current generation; qid.version tracks the generation so a conditional GET / ETag returns 304 and kernel v9fs caching stays correct under mutation across instances; and the cache honours an explicit byte budget with every pin released on every exit path: no pin leak, no finalizer-based release.
    Depends on
    the generation-fenced head (item 12) and the store-store adapter (item 14).
    Landed
    qid.version tracks the head generation so a conditional GET / ETag returns 304 and kernel v9fs caching stays correct under cross-instance mutation; an immutable-block cache hit validates by its score and only the head revalidates, keyed to the current generation; the cache honours an explicit byte budget with every pin released on every exit path. The seven sibling carries the matching qid.version backend stamp.
  9. [X] Migration/evacuation end-to-end. The thesis proven end to end, with the auditor-facing architecture document shipped.
    Done when
    a fresh instance reassembles from manifest plus out-of-band keys and serves the namespace while the evacuated instance's writes are fenced and its live grants swept; an old instance paused across the generation bump cannot land its commit (the fencing race re-verified end to end, across a GC pause); a confused-deputy audit confirms no store, backup, or observability path ever reaches decrypted content (the store handles opaque blocks only: encryption-ready, not encrypting); and docs/ARCHITECTURE.org is updated to capture the fencing/CAS/migration flow.
    Depends on
    reassembly (15), the fence↔revocation unification (16), and caching coherence (17).
    Landed
    a fresh instance reassembles from manifest plus out-of-band keys and serves the namespace while the evacuated instance's writes are fenced and its live grants swept; an instance paused across the generation bump cannot land its commit (the fencing race re-verified across a GC pause); a confused-deputy audit confirmed no store, backup, or observability path ever reaches decrypted content (opaque blocks only: encryption-ready, not encrypting); and docs/ARCHITECTURE.org carries the fencing/CAS/migration flow. The seven sibling carries the matching drain-on-evacuation teardown. The second arc is complete.

Where work can be delegated in the second arc

The store is the spine of the durable substrate and stays architect-held, built in-tree with the dependency ordering reviewed before each piece begins. The boundaries open in the same places as before:

  • The one seven dependency (a bulk-transfer msize ceiling for the read/walk/write path) lands in seven as a sibling repository; this arc may depend on its arrival but plans no seven-internal work.
  • libp2p/IPFS content-addressed block transport remains a later, separable sibling unit: the future repository boundary is the block transport, not the store itself, which is why the store cluster is kept import-clean from day one.

Third arc - plugin composition (complete)

Every protocol becomes a lifecycled plugin. A protocol is a module (a registered, lifecycled unit on Shinmera's modularize), and the edge controller's hard-coded adapter list becomes an *edge-adapters* registry that modules install into on load. The same module abstraction then carries through capability-gated admission and a migratable module manifest, so an instance's identity is its namespace manifest plus its module set. valis owns the plumbing; the protocol adapters themselves live in the c3po sibling repository. The design of record is docs/ARCHITECTURE.org §"Plugin composition".

  1. [X] The plugin seam. The edge controller's hard-coded adapter list becomes an installable registry of self-registering protocol modules, and valis gains the modularize-facing plugin infrastructure (the combined module class, the option-expander, and an ordered teardown hook), so adding a protocol needs no controller edit.
    Done when
    valis defines and exports one class that is both an asdf:package-inferred-system and a modularize:virtual-module, so a module's .asd names it as :class and keeps one-package-per-file derivation while gaining a lifecycle; the edge controller binds adapters by iterating the registry and names no concrete adapter; deleting a module runs an ordered teardown (deregister source → drain in-flight connections → unregister adapter → remove the edge entry) before its packages unbind; and the resident HTTP and Gopher adapters self-register with the full cold suite green.
    Landed
    a combined module-inferred-system class (package-inferred-system before modularize's virtual-module in the precedence list), the *edge-adapters* registry replacing the hard-coded %adapter-specs, an idempotent :edge-adapter option-expander firing in expand-option (after modularization, never as a too-early hook), and a drain-then-retire delete-module hook. HTTP and Gopher self-register purely by being loaded; a compile-then-load double-fire is a benign no-op; the start-edge compat surface keeps every existing call-site unchanged. The exported seam surface (registry, register fn, :edge-adapter keyword, combined class) was handed to c3po, which carries the protocol adapters on its own roadmap.
  2. [X] Capability-gated module admission. Loading a module is a capability decision over its content hash: an instance compiles or loads a module only when an owner-rooted authority has vouched for that exact hash, so a recondensing instance is never an open code-execution endpoint.
    Done when
    a module whose content hash an owner-rooted authority vouched for loads, the decision made as a capability check over the hash and not a bare fetch; an unvouched, wrong-hash, data-mount-only, or revoked hash is refused fail-closed: it never reaches load and leaves no partial module state; and the vouching authority chains to the owner root on the same verification path as the rest of the capability machinery, never a parallel trust check.
    Depends on
    the module abstraction of item 19.
    Landed
    an additive :admit right distinct from the data :mount right (a data-mount capability can never authorize code execution), a canonical bundle hasher content-addressing a module's .asd plus its source with length-prefixed, path-sorted framing so no re-partitioning of the same bytes collides and on-disk discovery order does not change the address, and a single admit-module chokepoint that verifies an owner-rooted :admit vouch over the module's /module/<hash> and loads exactly once. Revoked, wrong-hash, :mount-only, and unvouched requests refuse before the loader ever runs. The fail-closed and load-once invariants are pinned by direct assertions (a counting loader proven to fire once on a valid admit and zero times on every refusal, with the adapter registry asserted unchanged after a refusal), and the gate reuses the existing owner-rooted authority and revocation machinery, minting no new crypto. Cold suite green (1122/0).
  3. [X] Module manifest and migration. A durable, content-addressed record of an instance's active module set (distinct from the namespace manifest), so recondensation reassembles the module set from the vouched hashes: reassembly-not-transfer for code.
    Done when
    an instance records its active module set as a durable, content-addressed manifest distinct from the namespace manifest; a destination reads the manifest, pulls the vouched hashes, verifies and compiles each through the item-20 admission gate, and serves the same protocols as the evaporated instance; and a manifest naming an unvouched or unreachable module hash is rejected fail-closed, never standing up a partial module set.
    Depends on
    capability-gated admission (item 20).
    Landed
    a durable, content-addressed module manifest distinct from the namespace manifest, co-published with it under one genesis root; the evacuative stand-up reads it, pulls the vouched hashes, verifies and compiles each through the admission gate, and stands the same protocols up at the successor. A refusal aborts fail-closed with the adapter registry restored, so no partial module set ever boots. A post-delivery review pass hardened the re-vouch path (a fresh-entropy nonce per mint, a fail-closed set-completeness guard, and refusal of source paths that escape the module root). The third arc is complete.

Where work can be delegated in the third arc

valis owns the plugin plumbing: the module class, the registry, the admission gate, and the manifest stay architect-held. The boundary opens at the adapters:

  • c3po (the protocol adapters): a separate sibling repository. With the seam landed and its exported surface signalled, c3po has carried its HTTP, Gopher, and Finger adapters onto valis's real edge seam (the conversion is complete); they ride reassembly as ordinary vouched modules through the admission gate. SMTP is parked for the messaging substrate (architect-led).
  • libp2p/IPFS as the live module-distribution transport remains the named gate for a later milestone: content addressing stays local and seam-shaped here, so the admission gate is transport-agnostic when that distribution path arrives.

Fourth arc - factotum and transport-identity adoption (complete)

valis's transport identity and key custody were built in-tree during the first arc. The fourth arc moved the factotum (the identity and key-custody agent) into its own sibling repository, mercer, so the constellation has one home for transport authentication and custody that valis and other consumers depend on rather than each carrying its own copy. It adopted the extracted NoiseXX authenticator, consolidated the post-attach transport cipher into mercer with each session sealed end to end, and proved the responder against an independent published transcript carried over the real wire. The design of record is docs/ARCHITECTURE.org.

  1. [X] Adopt mercer as the factotum home. valis cut over from its in-tree NoiseXX transport identity and RFC 8439 AEAD to depend on the mercer sibling, and the handshake gained validation against independent published vectors.
    Done
    valis depends on mercer for the transport authenticator and the in-tree copies are retired, with the handshake/anonymous-attach/key-stability behaviour unchanged; the NoiseXX handshake is checked against independent published vectors (not only valis's own two-party self-interop), backed by a short cryptographic review; and the 9P auth seam the authenticator sits over is agreed with the seven sibling.
  2. [X] Wrap the post-attach session. The transport cipher is consolidated into mercer (mercer owns seal/unseal/close-session); valis owns the keyed accept loop and installs the sealing endpoint at the attach boundary, so every record after attach is sealed end to end.
    Done
    a keyed peer's post-attach 9P traffic is sealed under per-session transport keys; the record framing bounds allocation against a hostile length prefix; a tampered record fails closed to teardown with no plaintext; and the wrap is validated directly against the real keyed accept loop.
  3. [X] Over-the-wire initiator. A remote initiator authenticates end to end over the wire and reaches its authenticated view, with the on-wire transcript anchored to a standard-conformant NoiseXX vector rather than valis's own self-interop.
    Done
    a real initiator dials valis over the wire, completes the keyed handshake and 9P attach, and reads a principal-scoped node over the sealed session; the handshake and transport bytes match an independent published vector; and the documented real-NIC sealed-attach procedure is shipped for the operator to run out of band.

Where work can be delegated in the fourth arc

The factotum is identity-critical and its seam stays architect-held, but the work is inherently cross-repo:

  • mercer (the factotum): the sibling repository that becomes the home of the transport authenticator and, later, the broader key-custody surface. It owns the independent-vector validation and the cryptographic review.
  • seven (the 9P keystone): owns the 9P auth seam the authenticator sits over; coordinates the interface mercer's handshake rides.

Fifth arc - the mail transport spine (complete)

The fifth arc stands up valis's sovereign mail-transport core: the substrate that moves a message end to end through the capability-gated namespace, with the wire engines that speak SMTP and IMAP left to the c3po protocol sibling. It models a message as an immutable content-addressed body plus a separate transport-state record, moves it through a durable queue and a fail-closed router (no open relay), projects the whole spine as a capability-scoped /mail namespace axis where a 9P write is the authenticated submission, and proves the assembled spine moves real mail end to end against the live image with the wire stubbed, the body an opaque block no operations path can name. Modern exchangers reject unauthenticated mail, so being a legitimate sender ultimately means being authoritative for our own DNS; that, and the SMTP/IMAP wire engines, are the arcs beyond this one. The design of record is docs/ARCHITECTURE.org.

  1. [X] Message-object model over the store. A submitted or received message is an immutable, content-addressed body (opaque RFC 5322 octets: never parsed, never mutated) plus a transport-state record carrying the envelope and lifecycle, so the deferred retry / bounce / relay builds are seam-fills.
    Done
    bodies dedup by content address and ride namespace migration as ordinary store blocks; the envelope and transport state live in a separate mutable record keyed by the body's address.
  2. [X] The durable transport queue. A fenced, fsync-durable queue over the content-addressed head, with the acknowledged-write contract: a submission returns success only after its queue entry is durable.
    Done
    queue entries advance by compare-and-swap off the immutable body so the body needs no lock; the incoming / active tiers carry the day-one fields (attempt count, next-attempt time, last status) the deferred drivers fill.
  3. [X] The router with a fail-closed open-relay gate. Each recipient is classified local-delivery versus relay; relay is deny-by-default, decided by a submit-capability check reusing the existing capability machinery.
    Done
    a message relays only for an authorized submitter; an unauthenticated non-local relay attempt (peer-landed mail for a non-local recipient) is refused, never silently forwarded: the blocking open-relay guard. Sovereignty as a capability check, not a flag.
  4. [X] The mail-adapter seam. A *mail-adapters* registry plus an inbound landing entrypoint and an outbound drain driver, the opaque "hand these octets to the wire" contract with the c3po protocol sibling.
    Done
    the spine is decoupled from c3po's delivery: the adapter returns a delivered / deferred / bounced disposition, and the spine is provable end to end with the wire adapter stubbed.
  5. [X] The /mail axis and 9P-native submission. A capability-scoped /mail namespace axis (outbox / queue / inbox / ctl) on the qmail Maildir model, where the owner submits by writing a message into the outbox.
    Done
    a capability-mounted 9P write is the authenticated submission, durable before the write returns; sovereignty-as-absence holds on the queue projection: an ops grant sees envelope and status only and has no name for any body or for the inbox.
  6. [X] Local end-to-end proof and the sovereignty audit. The spine moves a message end to end against the live image (native submit → ingest → queue → route → {local-deliver | relay-drain to a stub adapter}, one each way) with the open-relay guard re-proven and the content boundary audited.
    Done
    start-fabric binds the mail seams so /mail is live in the owner frame; a confused-deputy audit confirms no operations / backup / observability path reaches a message body; and docs/ARCHITECTURE.org ships the mail-flow, open-relay-guard, and content-boundary deliverable.

Where work can be delegated in the fifth arc

The spine is the substrate; the wire is the sibling's. The split holds across the subsystem:

  • c3po (the protocols): owns the SMTP and IMAP wire engines that ride the mail spine through the adapter seam, and (for the deliverability arc beyond) the SPF / DKIM / DMARC evaluators. valis owns the substrate, queue, router, and the adapter seam they plug into.
  • mercer (the factotum): for the deliverability arc, owns the DKIM key custody and the sign / verify crypto; the contract with c3po stays opaque ("sign these octets") so the private key never crosses into the parser.

Sixth arc - authoritative DNS substrate and the PostgreSQL operator-state seam (in progress)

The sixth arc gives valis a mutable, name-keyed PostgreSQL operator-state seam - the ACID counterpart to the content-addressed store, holding the records an operator mutates (domain names, social graph, message state) where a body never enters the database - and, over it, an authoritative DNS substrate: the domains valis terminates, authored through the capability-gated :names axis and served from an in-memory view that answers off the database hot path and degrades to serve-last-good across an outage, behind a versioned, fail-closed adapter seam a capability-admitted wire engine binds to. It follows the fifth arc's split: valis owns the substrate, the controller, and capability admission; the runciter sibling owns the DNS substrate (zone model over PG, :names semantics, BIND text parsing, resolver policy); the binary DNS wire engine on :53 is c3po-dns, stubbed locally here. Sequencing is locked: the operator-state seam is built first as the foundation, and the DNS substrate reads zone data from it. The design of record is docs/ARCHITECTURE.org.

Build sequence (in order)

The seam is built foundation-first: the PostgreSQL operator-state seam, then the zone schema and the :names door over it, then the DNS adapter seam and in-memory zone view, then the secondary-first cutover substrate. The runciter sibling scaffold and the versioned seam contract overlap the foundation because the cross-repo coordination with c3po-dns has lead time.

  1. [X] Scaffold runciter and the DNS adapter seam contract. Stand up the runciter DNS-substrate sibling and define the versioned seam contract the :53 wire engine binds to, stubbed locally, so the cross-repo interface is versioned before the substrate that fills it is built.
    Landed
    the runciter sibling exists and carries the DNS substrate; valis names a versioned, fail-closed adapter seam and no concrete wire engine, with the c3po-dns binding deferred to its own roadmap and stubbed here.
  2. [X] The PostgreSQL operator-state seam: the foundation. The mutable, pooled, fenced dao-class seam every later axis reads through, where a body never enters the database and two valis sharing one database cannot both write.
    Done when
    the seam is wired into the fabric boot/teardown path; the store generation is projected into PostgreSQL fence-first on evacuation, so a superseded instance cannot write; and a body is structurally barred from the database (pointer-hash only).
    Landed
    the fenced dao-class pool stands up and tears down with the fabric; the generation projects fence-first on evacuation, proven across the real fabric evacuation path so coincident instances on one database cannot both write.
  3. [X] Zone schema, the :names axis, and the BIND text parser. Zones live in PostgreSQL and are mutated only through the capability-gated :names axis; legacy BIND zone files parse correctly into :names rows.
    Done when
    a zone is authored and mutated only through :names, never by reaching the table directly; and a legacy BIND zone parses into canonical rows.
    Landed
    the zone schema lives behind the operator-state seam, the :names axis is the sole capability-gated door to it, and the BIND text parser canonicalises legacy zones into rows.
  4. [X] The DNS adapter seam, the in-memory zone view, and the stub-adapter proof. Prove the seam end to end with the wire stubbed: a decoded query resolves through the in-memory zone view, independent of the database hot path, admission fail-closed.
    Done when
    a decoded query resolves through the zone view to a record set and result code; a kill-PG-mid-load leaves the last good view serving; and a DNS wire module is admitted only as a capability decision, fail-closed.
    Landed
    the in-memory serving view answers off the database hot path and degrades to serve-last-good across a PG outage; the versioned adapter seam resolves a decoded query end to end against a stub wire engine; admission is a capability decision.
  5. [X] The cutover substrate: secondary-first zone sink/source. Ingest a legacy zone as a secondary, track its SOA-timer and cutover policy, and support the outbound-feed and serial-bump data side, so a domain can be moved onto valis before valis answers as its primary.
    Done when
    a zone is ingested as a secondary and tracked through the cutover lifecycle (held → answer-as-secondary → primary, operator-gated); a held or expired secondary never enters the serving view; serve-time expiry is enforced fail-closed against the persisted absolute deadline; and cutover legality is decided by runciter through the seam, never re-implemented.
    Landed
    the per-zone /ctl flips (answer-as-secondary → promote-to-primary) with runciter-decided legality through the serving seam and a fenced persist; the answerability gate is centralised at the refresh-zone install chokepoint; serve- time expiry is enforced against the persisted absolute deadline (restart-safe bigint epoch timers); and the outbound zone-change feed is fault-isolated. The runciter pin is tightened to 0.3.0 with the typed rdata-parse-error adopted fail-closed on the import and per-zone read boundaries. Merged to main via PR #32 (352c2ff).
  6. [X] Resolver policy and the mail-drain seam. runciter's MX resolution policy, wired into the v1.4 mail drain to close the deferred MX-over-DNS slice, so an outbound message resolves its recipient's exchanger through the sovereign DNS substrate rather than a stubbed lookup (the resolver wire itself stubbed).
    Done when
    the mail drain resolves a recipient MX through runciter's resolver policy behind the seam; the resolver wire is stubbed but the policy and the seam are exercised end to end; and the deferred MX-over-DNS slice in the mail spine is closed.
    Landed
    the mail drain routes relay entries through an injected resolver and maps runciter's four MX outcomes onto the delivered/deferred/bounced queue vocabulary, proven end to end over stub-wire DNS with no live wire. The seam is versioned around runciter's resolve-mail-exchangers; the wire client fills it later. Merged to main via PR #37. The real-wire cutover (inject the live resolver at the production boot seam and retire the smarthost) is item 41 below, still open.
  7. [X] Operator-state migration and the per-axis sovereignty model. Carry the mail-transport state and the social graph onto the PostgreSQL seam, then document and audit the per-axis sovereignty boundaries.
    Done when
    mail-transport state and the social graph live on the operator-state seam alongside the DNS zones; and the per-axis sovereignty model is documented and audited. One principle, three boundaries: a content boundary for mail bodies, write-authority for DNS zones, capability-scoped access for the social graph.
    Landed
    mail-transport state and the social graph sit on the operator-state seam alongside the DNS zones; the per-axis sovereignty model is documented and audited: the content boundary for mail bodies, write-authority for DNS zones, and capability-scoped access for the social graph each hold as a distinct axis.
  8. [X] valis becomes the pure ndb DATA owner: the nameserver serving layer moves to runciter. Conform the DNS code to the architecture of record: valis holds the authoritative zone data and dispatches connections; the logical nameserver (runciter) holds the serving index and composes every answer.
    Done when
    valis holds no DNS answer logic and no serving cache: it exposes the zone data through the zone-data-source read protocol (plus the held-inclusive origin set and the per-zone cutover status the nameserver gates answerability on), and the bind registers runciter's serving handler at the agreed contract version, fail-closed; a held or expired secondary still never answers, now enforced from the cutover-status data plus runciter's gate rather than a valis serving view.
    Landed
    the in-memory serving layer (the loaded-zone index and the RFC 2308 answer composition) is shed from valis and relocated to runciter, parity-checked branch-for-branch before the shed; bind-dns-view registers (make-dns-service-handler <pg-zone-source>) via late resolution at contract-version 1; the transitional serving-seam fallbacks are deleted (the seam is thin late-bound delegation, fail-closed); serve-last-good is handed to runciter with an explicit audit confirmation rather than dropped; docs/CONSTELLATION.org and docs/ARCHITECTURE.org record the split. Pending merge to main; the runciter pin tightens to its serving-module tag when that repo is merged and tagged.
  9. [X] TSIG-gated zone-transfer allowlist and serial monotonicity. valis owns the transfer-peer allowlist as capability-mediated operator-state and enforces RFC 1982 serial monotonicity on the transfer path; no zone reaches the wire except to an allowlisted, authenticated peer.
    Done when
    a zone leaves valis only to an allowlisted peer, the allowlist is mutated only through the capability-gated operator-state door, and a non-monotonic serial is rejected before it can be emitted.
    Landed
    the allowlist verdict is injected into runciter's authorize-before-emit gate through the seam (bind-transfer-authorizer mirrors bind-dns-view), serial monotonicity is carried on the transfer path, and the gate is proven end to end across the repo boundary. Merged to main via PR #42 and PR #43 (04f41a4).
  10. [X] Authoritative answers on :53 through fail-closed admission. valis answers authoritative DNS on :53: a steered connection is dispatched to the admitted DNS module fail-closed, and with runciter's serving orchestration and c3po's wire codec landed, the answer is externally probeable: non-recursive and amplification-bounded.
    Done when
    a steered :53 UDP/TCP descriptor is admitted fail-closed and pumped through the wire codec onto runciter's serving exchange; the serve loop starts from boot once the DNS view binds and disarms on module revocation; and an out-of-band probe from a stranger vantage confirms authoritative, non-recursive answers with EDNS0@1232 and TC/TCP fallback bounding amplification.
    Landed
    two steered :53 fds sit behind one admitted handler; queries pump through c3po's codec onto runciter's serve-exchange; the loop boots with the DNS view and retires fail-closed on revocation (a revoked fd stays live yet serves nothing); and an out-of-band probe from a stranger vantage confirms authoritative, non-recursive answers with amplification bounded. Merged to main via PR #44 (7b6579d).
  11. [X] Real-wire mail-drain resolver cutover: retire the smarthost. Inject runciter's live MX resolver into the production outbound drain and remove the static smarthost fallback, so an outbound message resolves its recipient's exchanger over the real DoT wire, fail-closed against a hung or hostile upstream. The stub-wire seam (item 36) is the substrate; this closes it onto the live wire.
    Done when
    the production drain boots with the live resolver as the MX source and the smarthost fallback is gone from the delivery path; a resolvable recipient delivers to its preference-sorted MX; a transient failure (SERVFAIL, timeout, TLS or auth failure, budget exhaustion) defers and never loses the message; a null-MX or NXDOMAIN bounces; the resolve is deadline-bounded so a hung upstream defers rather than stalling the drain thread; a wrong-cert DoT upstream defers, never delivering over an unauthenticated or downgraded channel; and the resolver stays client-only, with no inbound listener or accept loop.
    Landed
    the production drain builds runciter's live DoT resolver at boot (late-bound, so valis stays cold-loadable without it) and resolves each relay recipient's MX over the real wire; the static smarthost is gone; an unresolved, unconfigured, or empty-trust upstream DEFERS rather than delivering; the open-relay guard is unchanged. All mail suites green, cold-hermetic. Merged to main via PR #45 (4ba6004), over runciter's make-mail-resolver (PR #19).

Where work can be delegated in the sixth arc

The substrate and the seams stay architect-held; the wire and the resolver engine are the siblings'. The split holds as in the prior arcs:

  • runciter (the DNS substrate): owns the zone model over PostgreSQL, the :names semantics, BIND text parsing, and the resolver policy; coordinated cross-repo, with merge and tag operator-gated.
  • c3po-dns (the wire): owns the RFC 1035 binary engine on :53 (serving, AXFR/NOTIFY, the resolver wire client), on its own roadmap and gated on valis signalling the DNS adapter seam; stubbed locally through this arc.

Seventh arc - the resident steered path (in progress)

The first arc proved eBPF sk_lookup steering end to end on a real kernel, but in the privileged fulcrum sibling, external to the valis process: valis still listened on a loopback stand-in. The seventh arc makes the steered descriptor the valis process's own resident listening path: valis mints its designated-port LISTEN descriptor in production, receives it across a fail-closed process boundary, and boots its steered edge sources on the pushed descriptor, so the target data flow of the first arc becomes the running one. What remains is hardening the cross-process push to be race-free at the wire and the privileged launcher that brings the whole resident unit up under a real network namespace. The design of record is docs/ARCHITECTURE.org.

Build sequence (in order)

The primitive lands first (the descriptor crossing and the shared fd-passing seam), then the resident boot that consumes it, then the two hardening pieces: a race-free push at the wire and the privileged launcher for the live-kernel bring-up.

  1. [X] The descriptor peer and the shared fd-passing seam. An SCM_RIGHTS descriptor-passing primitive with fail-closed admission: a single caller-owned descriptor crosses the process boundary and every crossing goes through one shared seam.
    Landed
    the SCM_RIGHTS primitive gates each crossing through fail-closed admission (a multi-descriptor control message is refused, a sent descriptor stays caller-owned, syscall returns are checked), proven cold across a real process boundary by a fresh-SBCL-child spawn harness; the edge passes every descriptor through the one shared seam. Merged to main via PR #48 and PR #49.
  2. [X] The resident boot on the pushed descriptor. valis boots its edge on the steered LISTEN descriptor pushed from the privileged host agent, fail-closed: the resident listening path replaces the loopback stand-in.
    Depends on
    the descriptor peer and the shared seam (item 42).
    Landed
    valis mints its designated-port LISTEN descriptor in production and a resident boot brings the steered edge sources up on the descriptor pushed over a 9P control client; a --resident dispatch boots fail-closed with the shipped adapter set, and the bounded control-push read closes a failed dial. The push sends the descriptor only after the control-port write drains, and the crossing is proven over a real process boundary. Merged to main via PR #51.
  3. [X] A race-free descriptor push. The production descriptor push crosses race-free at the wire, so no buffered read on the receiver can over-read the carrier byte and silently discard the passed descriptor.
    Done when
    the sender pushes the descriptor immediately after the control frame with no out-of-band readiness marker and the descriptor still crosses race-free; the carrier byte is only ever consumed by a control-armed receive, never a plain buffered read; and ordinary control frames stay byte-identical.
    Depends on
    the resident boot (item 43). Today's production push is fail-fast: it makes the race loud with a read deadline, not gone.
    Note
    the reader that guarantees this is owned by the seven sibling (its AF_LOCAL control-endpoint reader); valis's sender adopts it by swapping its push seam onto seven's primitive with no construction change, retiring the in-test readiness barrier and the hand-rolled raw-frame push while keeping the read deadline as defence-in-depth. fulcrum's receiver adopts the matching guard. Merged to main via PR #55.
  4. [ ] The privileged resident launcher. The host agent brings the resident unit up under a real network namespace and hands over the descriptor across a well-known control socket, so the steered path comes up from a single privileged bring-up rather than a manual stitch-together.
    Done when
    the fulcrum host agent launches valis into the network namespace it steers and hands over the LISTEN descriptor across a well-known control socket, and the resident unit is reachable on its steered port without a manual bring-up.
    Depends on
    the resident boot (item 43); independent of the race-free push (item 44).
  5. [ ] Live-kernel capstone. The whole resident path (steer, push, resident boot, answer) proven end to end on a real kernel from an out-of-band operator run.
    Done when
    an out-of-band operator run steers a designated port on a real kernel, pushes the descriptor to a resident valis, and a stranger-vantage client is answered on the steered port through the resident edge, with valis holding no privileged kernel state.
    Depends on
    the race-free push (item 44) and the privileged launcher (item 45).

Where work can be delegated in the seventh arc

The resident path is the valis process's own and stays architect-held; the privilege stays across the repository boundary, as in every prior arc:

  • seven (the 9P keystone): owns the race-free control-endpoint reader the descriptor push rides; valis and fulcrum adopt it as consumers with no re-plumb.
  • fulcrum (the privileged host agent): owns the network-namespace launcher and the well-known control socket that bring the resident unit up on a real kernel, and adopts the receiver-side guard for the race-free push.

Eighth arc - the live internet host (in progress)

valis becomes publicly reachable. The resident path proven in the seventh arc is turned outward: the privileged fulcrum launcher binds the public ports in a real network namespace and hands the descriptors to an unprivileged valis, which serves without a privileged bind. Deployment is unified with migration (standing a node up is the genesis case of condense) and made reproducible and testable rather than hand-stitched. The remaining pieces obtain and terminate TLS so the edge is publicly reachable over HTTPS. The design of record is docs/ARCHITECTURE.org.

Build sequence (in order)

  1. [X] Public authoritative DNS on :53. The resident serves authoritative DNS to the public internet on UDP and TCP, no geoblock, CAA answered correctly, reachability proven off-host, the dns-01 precondition.
    Landed
    fulcrum binds :53 UDP/TCP in the netns and exec-inherits the descriptors to an unprivileged valis, whose resident boot adopts them and drives the sibling serve loop over them with a pg-zone-source: no privileged/loopback bind. CAA and TYPE257 answer NOERROR/NODATA; the serving path is proven end to end off-host in a real-kernel integration gate, and a multi-region reachability harness proves worldwide answerability and gates the public go-live. Merged to main via PR #58.
  2. [X] The host-deployment contract and the deployment proving ground. A durable, declarative harness that stands Debian guests up from source and drives valis's own deploy == condense-from-genesis path under assertion, plus the written contract.
    Landed
    docs/deployment/host-deployment-contract.org states the prerequisites, bring-up order, provisioning, teardown, and fail-closed invariants. proving-ground/ pins a Nix verification fabric, boots a two-location VLAN-tagged Debian topology via Terraform-libvirt, provisions with cloud-init, and drives three scenarios (deploy, evacuate, restore) over one mechanism through an in-image Lisp/SSH assertion layer (valis-proving-ground, its own ASDF system). The scenarios assert the shipped condense/fence path; the harness is unit-green against fixtures, with a live libvirtd run the one out-of-band step. Merged to main via PR #59.
  3. [ ] The active-module (resident/lifecycle) seam. A start/stop/supervise module kind with capability-scoped egress, boot-supervised, proven with a trivial no-op module: the non-reactive second module archetype the ACME manager will consume.
    Built
    the archetype exists and the renewal manager registers through it; module admission runs from the durable manifest.
    Not yet true on the live node
    no active module has ever been admitted there.
    Corrected since
    this used to read as a certificate fault, the node reporting no certificate in custody while holding a complete chain and key on disk, because the status line read an enumerator that was never bound to the credential store. It no longer does. The cert gate loads the credential for the configured edge domain straight from the ACME store path, and the readiness status is handed that same resolved fact rather than re-deriving one. The independent cert-expiry watch still goes through a custody enumerator seam, which is a different surface and was not the one reporting wrongly.
  4. [ ] The ACME/dns-01 manager and public HTTPS on :443. Obtain and renew a real TLS certificate over ACME dns-01 (mercer custody), terminate server-TLS, and serve the anonymous /pub view over public HTTPS on :443, with the fabric absent from the public wire and a restore-tested backup before exposure.
    Already on main
    the server-TLS termination seam with renewed-cert hot-reload; the :443 edge itself, terminating to the anonymous /pub view with the fabric structurally absent from the public wire; the challenge machinery, proven against a real CA, with five Let's Encrypt vantages observed querying the challenge name on the wire; and the order lifecycle through to a downloaded chain.
    Done since
    a production order was placed and the public :443 serves it. The live node answers for deepsky.systems under a Let's Encrypt chain that verifies off-host. The edge stays dark by design without a certificate, and that gate behaves correctly.
    Remaining
    the live node holds that one certificate while it is authoritative for four zones, so the other three are answered under a certificate that does not name them. Item 52 carries the rest.
    ⚠ Known before the next attempt
    opening :443 requires the port to be declared to the host agent and admitted by its firewall, which today admits :53 only. Declaring the port on the valis side without the matching firewall admission took the nameserver down for two minutes on 2026-08-13; a missing edge port now degrades to a dark :443 instead of killing the resident.
  5. [X] The owner control plane over a running node. Owner authority over a running node became withdrawable, and content became placeable on a node that is already up rather than only on one being created.
    Landed
    Live sessions are registered with an identity an operator can read off a listing and type back, and the identity axis carries a door that reports the live sessions and fences the one it is told to. A session is also fenced by idleness, checked by the operation that wakes it, with the bound unset by default so no deployment is locked out by an upgrade it did not ask for; and by an authority transfer, so an instance that has surrendered write authority stops serving an owner after an evacuation. An owner-keyed publication verb places a page on the running node, and the client verbs list and fence a session from outside the image. Merged to main via PR #116 and PR #117.
    Off-host reach
    UNPROVEN, and it is the half that is easy to read as done. The serving-side wiring is built: an inherited owner-port descriptor is adopted as the routable owner terminus and bound as a second keyed listener, fail-closed unless the auth seam is engaged, and the launcher carries an :open-owner-port knob for the firewall. But the port ships closed, no deployed node has it opened, and every exercise of the owner client so far has been against a same-host loopback. Nobody here has reproduced a cross-host attach. A single-host "verified live" proves the cryptography, not the reach through the namespace and firewall boundary.
    The remaining half
    A fence does not revoke a handle already open below the session root, so a client holding one keeps it until the revocation reaches it. The remedy is a generic per-session handle revoke in the 9P library, which needs to know nothing about owners or fences. That mechanism has since landed upstream; valis is not yet wired to it, so the limit stands here for now.

Where work can be delegated in the eighth arc

The public exposure and privilege stay across the repository boundary, as in every arc:

  • fulcrum (the privileged host agent): owns the :53=/:443= exposure, the netns and firewall, and the launcher that brings the resident unit up.
  • mercer (the credential authority): owns the ACME lifecycle, the dns-01 digest, cert custody, and the server-TLS termination seam.
  • runciter (the DNS service module): serves the transient challenge TXT for dns-01.

Ninth arc - the nameservice surface (in progress)

The arc that turns a reachable host into a serving one. Its subject is DNS and TLS in service of the actual goal, which is websites answered from the operator's own machine on the operator's own domains. Named here because the eighth arc's remaining items are being closed out under it rather than separately.

What is delivered

  • valis is the primary authoritative server for the registered domains. The lame delegation is closed, a sibling host runs as an AXFR secondary, and zone data lives in PostgreSQL as operator state. Zone editing is typed and structured; the master-file text format is a boundary format for the secondary, never the edit format.
  • A node can be brought up from nothing and made to serve DNS. The acceptance gate provisions a guest, installs the delivery artifact, and drives the resident up on it. Two checks in that gate could not have reported a failure and were corrected: the gate's exit status now carries its verdict, and the write-fence assertion authenticates as the account that holds the row.
  • Connections may outlive a request, in the codebase. The edge no longer assumes a client-speaks-first, reply-only protocol, which is the precondition for every continuous protocol after it. Not yet exercised on the live node.
  • A running node can be updated and published to without being rebuilt, and the operator reaches it over an authenticated path rather than by editing files on the host.
  • A certificate renewal completes end to end in the deployed shape. Proven on a two-NIC guest in the deployed shape, against an authority that resolved the dns-01 challenge rather than being told to accept it. The defect that had stalled every local proof for months was ours: an outbound connection was not asked for as a duplex stream, so the request sat unsent. A public authority's handshake happens not to arrive in the order that exposes it, which is why certificates kept coming back from the field while every local proof hung.
  • The edge holds one credential per served name, so a request for one name is no longer answered with another name's certificate.
  • A deploy refuses before it moves anything the node cannot run, from one declaration that the boot path and the deploy path both consult. The refusal comes before the binary moves, so a node that cannot run the new build keeps serving the old one.
  • The two-NIC production topology passes its acceptance gate. The dedicated interface moves wholesale into the serving namespace, and :53 answers authoritatively on its routable address there. Verified on a guest destroyed and rebuilt rather than on the one that had been running. The fault that stalled this for weeks turned out not to be ours: a guest given no CPU model gets a default that lacks POPCNT, and SBCL's own bignum logcount faulted on its non-POPCNT path. ⚠ The gate is not yet the whole property, and I am recording the gate: :53 is asserted from outside the node and not yet from inside it, and the topology is written down nowhere under docs/ as the contract a host-install recipe would encode.
  • Two sockets on one address and port no longer collide silently. A UDP pair both carrying SO_REUSEADDR are both accepted and the second takes every datagram, which is a total DNS outage that reports nothing. The DNS datagram socket is bound without that option and first, so the kernel refuses the second bind instead.

What remains

I settled the order of these on 2026-09-06, after looking at how far the tree still is from where it is going: item 53 first, then item 55, then item 52. Item 54 depends on none of them and can run alongside. The recipe that installs a two-NIC host comes last of all.

Item 53 leads because it is the only one with no evidence either way. The recipe comes last because one written today would capture a node whose ports are handed down one at a time and whose steer carries nothing, and that is the version that costs the most to undo once it is the durable contract.

  1. [ ] Every served name answered under its own certificate, on the live host. The machinery is built and listed under What is delivered above: one credential per served name, the order lifecycle, the challenge machinery, and a renewal proven end to end in the deployed shape. deepsky.systems is answered correctly today.
    What is left
    the deployed node. It holds one certificate and is authoritative for four zones, so three of them are answered with a certificate that does not name them. The node is behind the tree. This is a production change to a host serving real domains, which is why it is its own item and not a loose end on a build.
    Known going in
    never SAN-pack the four names into one certificate. One certificate per registrable domain, because Certificate Transparency records the pairing permanently. Reaching the edge at all is item 53.
  2. [ ] Cold verification that steered traffic is delivered. Real traffic reaching the resident accept path from a fresh image, with the four-tuple preserved and the original destination port recovered in userland. ⚠ I want this one first because nothing else would have caught what the entry below records, and it stayed invisible for weeks while every part of it read as correct.
    Known going in
    on the live host no port arrives through the steer at all. The :53 pair and the :443 edge are both inherited descriptors, and the http and gopher adapters bind loopback, so every public port is a privileged bind the host agent hands down. I went looking for the boot-path defect this item used to record and it is gone: the resident composes both arrival paths, and it declines to mint a socket only when the port it was told to serve is the one already arriving inherited. The live node is told :53 and inherits :53, so nothing is minted and the steering map stays empty. Name a port that is not inherited and the steer carries it with no code change, which is what makes this item a verification rather than a repair.
    Worth knowing
    the steer has carried public traffic before. gopher and http were served through it until the DNS work shadowed them, and the catchall object ships and sits dormant on the node. A failure here is a regression, never a first proof.
  3. [ ] A TLS-terminated connection the event loop can drive. Termination hands back a blocking stream bound to one thread, while the loop drives raw descriptors and a TLS connection's raw descriptor carries ciphertext. Until this is answered, a continuous protocol over TLS costs a thread per idle session, which is the cost the continuous-protocol work exists to remove. Gates owner mail access.
  4. [ ] A node declares the port set it serves, not one port. The resident takes a single port today and refuses a set, so a node serves one steered port plus whatever it inherited. I want the set, because a person's node runs the protocols that person chose and one port cannot carry them.
    What is left
    the open question is in the control session, which pushes one descriptor per write. Whether several are expressible in it as it stands decides whether this is a small cut or a change to the contract between valis and the host agent. That bound is not settled, and settling it comes before cutting.
    Known going in
    this is sequenced nowhere else. It appears in no arc and in no design document, only as a note beside the code that refuses it, which is why it is written down here.
  5. [ ] The inherited :80 reaches the documentation. Every node ships :80 enabled and serves its own documentation there in the clear, so a new owner can read the manual for the machine off the machine itself before a certificate exists and before anything else on the node is working. The node does not do that yet: the port arrives and nothing answers on it.
    Done when
    the inherited :80 descriptor reaches the http adapter the way the inherited :443 descriptor reaches TLS termination today, and a client speaking plain HTTP/1.1 with no TLS is served the documentation by a node that holds no certificate.
    Known going in
    the resident parses the descriptor as --edge-http-fd and declares it ignorable; start-edge binds the registered adapters' loopback ports and takes no descriptor at all, while :443 is threaded separately into start-tls-serve as :tcp-fd. That threading is the shape to copy. The host-agent half is a config-gated bind and firewall accept, shipping enabled, so the descriptor will be there to consume.
    Note
    this is the valis half only. The design it serves is in docs/ARCHITECTURE.org, under what :80 is for.

The constraint behind all five items

Each of the five items above is a whole-path property, and every failure found in this arc was a feature that bound its own listener instead of arriving through the single steered socket. The steering model is that every port arrives on one socket, the operator's declared set decides what is answered, and the registry decides which module answers it. A feature that binds, inherits, or accepts on its own is departing from that, and :53 is the one place where departing is correct.

Beyond - open and on the horizon

The substrate thesis is proven; what remains is breadth and one genuine external unknown. These are tracked here as open items, not yet sequenced into an arc.

  • [ ] Native content-addressed transport (libp2p/IPFS). The one external dependency the ecosystem does not yet answer in Common Lisp. The store, the module-distribution puller, and cross-host coherence are all seam-shaped for it; the gate stays open. The intended path is a CFFI binding to a C-ABI libp2p, with an external-daemon HTTP bridge as an interim stopgap.
  • [ ] At-rest key encryption. The custody keyfile is a mode-0600 dev keyfile today; real at-rest protection (passphrase / OS keyring / HSM) is owed once real steering and real external authority are outstanding.
  • [ ] Owner-key rotation with continuity. The /id/ctl custody model gains an additional-keys keyring first: the DID stays fixed, so device enrolment and co-signing land without disturbing the sovereignty gate; this is the near-term step that unblocks the owner-facing viewer's key-rotation runbook. True rotation of the owner master (a new DID with a signed continuity record an external verifier can prove old→new) is the sovereignty fork the substrate's gate exists to catch, and is owed its own design arc: a continuity/attestation format, a mercer verifier, the question of whether a raw did:key is even the right identifier for something that must survive rotation, and migration of every capability, revocation, and durable head keyed to the DID.
  • [ ] The evaluation axis. Reaching a running node's live image to inspect, patch and repair it while it serves, which is one of the reasons the substrate is written in Common Lisp at all. The design is settled and written up in docs/REPL-AXIS.org: where the axis is placed, the shapes that were rejected and why, and one question still open (whether revocation must interrupt an evaluation already in flight, or only prevent the next one). None of the axis is in the tree: no right bit, no bridge process, and no listener wired to a namespace file. ⚠ The loopback Slynk listener the delivery image carries is not the axis and the resemblance is the trap. It is an ambient switch read from the host at boot, it carries no grant hash, revocation does not reach it, and it cannot be withheld from a delegate.
  • [ ] Further IETF protocols over the established subtrees. SMTP, IMAP, and NNTP wire engines (and the DNS engine) ride the established substrate seams as c3po modules; valis owns the substrate and the seam, c3po owns each wire engine. Messaging is one substrate with many protocol skins.
  • [ ] Sovereign identity provider (OAuth/OIDC). valis as its own issuer for the domains it terminates, decomposed across the constellation: valis owns the identity records (a DID/principal-keyed dao-class, distinct from the DNS ndb) and orchestrates issuance over the /cap spine; mercer owns the token-signing crypto; c3po owns the OAuth/OIDC wire engine. Horizon, not yet sequenced.
  • [ ] The upper doctrine layers: messaging beyond mail, publishing-as-social, contracts. Framed in the architecture as semantic substrates with protocols as projections; built only as far as the mail spine so far. Each is one substrate the later protocol modules project over.
  • [ ] What a production deployment is, and what sequesters a build. Lisp dependency resolution folds into our own private dist: that is where a release pin can record what a build was made from without freezing what development builds against. What remains open is a separate question that has been wearing the same clothes: what sequesters the rest of the build environment, meaning the C toolchain, the C libraries, and the host configuration a build leans on without saying so. A cold load regrovels iolib and cffi from source and so needs pkg-config and a C compiler present; the live host needs libfixposix and ambient capabilities; and a tracked gate driver can fail at load rather than at a test on any machine missing a developer's own source-registry file, which reads as a broken gate rather than a missing dependency. A dist answers none of those three. NixOS as a build engine, roswell with an explicit system-package manifest, and a documented host contract are the candidates; they are not mutually exclusive. One production deployment is already live while this is undecided.

The edge serves a name set, not a configured name

Direction from fade, 2026-08-16. valis must be able to hold potentially many thousands of domains in large deployments, and every domain a node holds is potentially a live edge, because any of them can point at the node's external interface for routing. ⇒ The served name set is therefore a property of what the node holds, not something an operator restates in configuration.

VALIS_EDGE_DOMAIN is the weakness, and one string answers three unrelated questions. Read in valis src/main.lisp and src/apply/delivery.lisp: it selects which credential the edge loads; it is a precondition for :443 coming up at all, since an unset value leaves the public edge dark before a certificate is even looked for; and the delivery-apply readiness confirmation reads it directly as the zone to confirm the node against after a restart. That third question has no answer at all once the node holds a set. It is also singular by construction: the resident sources one shared credential cell at boot.

What is already right, and what is not

Custody scales. Custody on the live host is keyed per domain on disk, so a second order has somewhere of its own to land rather than a slot to take over.

Selection does not. boomer already accepts an SNI callback, returning a certificate chain and key for the hostname a client asked for, and it can also refuse a name outright with an unrecognized-name alert, which is the answer a node needs for the names it does not hold. mercer's server-TLS wrapper does not forward it, so the capability exists upstream and does not come through the path valis uses. The seam carries a contract version; whether forwarding the callback disturbs it is open.

And the edge has no path to what the node serves. Zone state is keyed by zone identifier in operator state; nothing exposes "the names this node answers for" to the edge. Until something does, a name set cannot be derived and can only be restated.

What large numbers change, beyond correctness

A design that works for two names can fail at scale in ways that do not look like defects:

  • The issuing CA rate limits per registrable domain, so standing up many names is a pacing problem rather than a loop. ⛔ Never widen a certificate to cover more names instead: a SAN-packed certificate publishes the whole name list to certificate transparency permanently, correlating every domain on it.
  • Renewal becomes scheduling. A watcher that renews one credential is a different thing from one that keeps thousands current without a stampede.
  • Cost must remain measurable per domain, not in aggregate, or there is no way to say how many domains one node can carry and still pay for itself.
  • One cell sourced at start is fine for one name and wrong for thousands, which puts the timing of selection in question rather than settling it.

⇒ Nothing here is designed yet, and the first question is not SNI. It is where the served name set comes from.

dsmr carries the operations and delivery axis

Ruled by fade, 2026-08-16. dsmr is a constellation member, not tooling that happens to live nearby, and its role grows as the work turns toward operations and continuous delivery. It is registered in the responsibility map as a third kind of member: supply, beside the constellation rather than carved out of it.

What it owns today is the constellation's dependency closure, what every repository here is to be built from, pinned to exact commits and published as a Quicklisp distribution. That is the mechanism the constellation has not had for pinning what it builds against, and it is the reason the ordinary case can track main while a release records what it was built from.

The sequencing gate on this has expired. The custom distribution and qlot were deferred behind the public go-live, which has happened. The deferral named its own condition, and that condition has been met: the work is no longer held.

It closes the dependency half only. The open item above, What a production deployment is, and what sequesters a build, states plainly that a dist answers neither the C toolchain, nor the C libraries, nor the host configuration a build leans on. Nothing here changes that.

Node machinery belongs in the Lisp image

Ruled by fade, 2026-08-16. All node machinery should exist in the Lisp image unless there is a very good reason for it not to, because the nodes must be able to control their environment. A node that reaches its own state through an external command is a node whose state model is that command's, not its own.

The standing residue this names first: namespace creation, entry and teardown shell out to ip(8). Creation, entry and deletion are syscalls – unshare(CLONE_NEWNET), setns, netlink RTM_NEWLINK – and are reachable from the image. The precedent is already here: whistler loads and attaches eBPF objects in Common Lisp, which is why a deploy node needs no BPF toolchain at all.

The motivating defect, and the reason this is not a matter of taste: a resident could fail to start after any ungraceful death because /run/netns/<name> was a plain file rather than a namespace, and the liveness test was probe-file. That is a filesystem question standing in for a liveness question, and it exists only because ip netns models namespaces as bind mounts over files in a directory and that state model leaks through. Holding the namespace descriptor in the image makes liveness a property of a descriptor the node owns rather than of a path it inspects.

Not scoped to any current branch. The attach-proof work in flight lands first and still shells to ip(8); this is the direction it should be brought under afterwards.

The discriminator: which side of the node boundary the act sits on

Refined by fade, 2026-08-16, and it is what keeps the ruling above from over-applying. The rule is not "shell is wrong." Standing up host OS services is a perfectly normal and correct use of shell code. What requires Lisp is anything a node must manipulate from valis.

Preparing the ground a node runs on shell is right scripts/pg-dev.sh stands up a dev PostgreSQL; valis consumes a DSN and never touches the container
Acting through a node at runtime, under its authority Lisp namespace create, enter and teardown; placing publications into /pub

The test to apply: does valis itself have to do this, while running, as part of being a node? If a person or a build host prepares something before valis starts, shell is fine. If the node reaches its own state or an owner's through it, that path belongs in the image. A namespace is not host provisioning, which is why ip(8) falls on the Lisp side: the resident creates it, enters it, and must know whether it is live.

Publishing a tree into /pub is the second instance, and its reason is different

Called out by fade, 2026-08-16, the same day as the ruling above. The first driver that placed the manual into /pub was written as a Python build script that shells out to the publish verb once per page. It works and it is what put the documentation site up, so it is not being torn out to reach that aim.

The reason it is the wrong shape is not the reason ip(8) is. That one is about a node controlling its own environment. This one is about whose authority the act runs under: a stranger operating their own valis must be able to publish their own words within their own instance's authority, and that makes tree publishing an owner-facing capability of the node, not scaffolding on a build host. Filing it as site tooling mistook our first use of a general capability for the capability itself, in the same way :443 is an existence proof for the http server rather than the point of it.

What follows from that, for whoever picks this up:

  • The act belongs in the image, reachable by an owner who has no build host, no python3, and no checkout. A capability a stranger cannot reach is not a capability they have.
  • It should open one owner session and publish through it, rather than one session per page.
  • It must call valid-slug-p rather than restate it. The Python driver copied that charset into a regular expression, which puts a fail-closed traversal gate in two languages with nothing keeping them honest.
  • The link rewriting is a separate matter and can stay a build step: turning ARCHITECTURE.html into /architecture is post-processing an Org export, and it sits beside split-api.py, which already post-processes the same export.

⇒ The split to carry forward: rewriting names is a build concern, placing publications is a node concern. Bundling them is what hid the second inside the first.

The precedent was already in the tree. tools/api-doc.lisp, contributed by Glenn Thompson on 2026-06-08, walks valis's own packages through sb-introspect and writes docs/API.org. The documentation pipeline has carried the right division since then:

code to docs tools/api-doc.lisp Lisp, because it introspects the loaded image
org to HTML site.el, split-api.py text transformation, and non-Lisp is fine

Placing publications is the first kind of work, not the second: it acts on a running node under an owner's authority. Nothing needed inventing here; the shape was already demonstrated, and consulting tools/ would have settled it.

🄯 Brian O'Reilly <fade@deepsky.com>, 2026