valis / Direction
valis - Build TODO
The collaborator-facing status board for valis. It records where valis stands, what is done, what is next, and the order the pieces land in, across every delivery arc, not just the first. Read it alongside the design documents and the development guide:
- the design of record
- docs/ARCHITECTURE.org
- working with the code
- docs/DEVELOPMENT.org
The architecture is authoritative for how valis is shaped; this file is authoritative for where valis stands and delivery order and progress.
How to read the marks. An item is [ ] until it is merged to main, and [X]
once it is. Progress moves the same way the code does: while a piece is open it names
its feature branch and PR on an On branch :: line, and the tick flips to [X]
(echoing to main) when that branch merges. So this file read on main always
reflects what main actually carries; a feature branch carries its own items ticked
ahead of main until it lands.
Where valis stands
valis's founding intent is a Lisp Machine for network protocols: a sovereign, malleable substrate that multiplexes one person's data, identity, and protocols, made durable as a namespace rather than a process so it can move between hosts. That thesis unpacks into five claims; here is where each one actually stands, read off the code rather than the aspiration.
- Sovereign substrate: built and proven. The 9P namespace keystone (view roots, two-phase resolution, enforcement-by-mounting, union directories, read-only projection, revocation eviction) is the largest subsystem and the spine everything hangs off. Sovereignty is the absence of a name, set once at namespace construction, never a per-access guard.
- Identity and capability: built and proven. NoiseXX transport identity,
did:keyprincipals, a custody store that is the sole holder of key material, UCAN-mould delegable tokens, and Tahoe-style bearer capability-names, all offline-verifiable. The transport cipher itself now lives in the mercer sibling; valis is plumbing over an opaque session. - Migratable namespace: built and proven. A content-addressed, venti-style block store under a generation-fenced fossil head; reassembly-not-transfer migration with evacuation fencing; and the code-side symmetry: a module manifest that reassembles the running protocol set from owner-vouched hashes.
- For network protocols: one proven end to end, the rest externalised by charter.
A conformant HTTP/1.1 server answers on the steered
:80with identity-selected views on one URL space. The wire engines themselves now live in the c3po sibling (HTTP and Gopher adopted, SMTP/IMAP/DNS following); valis core deliberately names no concrete wire engine. The mail-transport spine and the authoritative-DNS substrate are built here as substrate-plus-seam. - A Lisp Machine all the way down: the resident path is live on a real host; the
public-TLS spine is the arc in progress. The eBPF
sk_lookupsteering is compiled from Lisp by Whistler and proven end to end on a real kernel in the privileged fulcrum sibling. valis consumes the steered descriptor in its own process (minting, receiving across a fail-closed boundary, and booting its edge on it, with a race-free wire push), and the privileged launcher brings the unit up under a real network namespace: the resident serves public authoritative DNS on:53, proven end to end off-host in a real-kernel integration gate (worldwide multi-region reachability has its own harness, gating the actual go-live). ⚠ DNS does not arrive by the steer. This is the one sanctioned departure from the one-socket model.sk_lookupcannot carry UDP at all, and a UDP reply must source from the socket it was received on, so fulcrum binds:53on both transports inside the namespace and hands the descriptors down across the exec. valis still performs no privileged bind. This is a necessary aberration from the one-socket model rather than a second sanctioned way to reach the edge: every other port is steered, and a new protocol requiring its own inherited descriptor is requesting an exception, not following the rule. The ACME/dns-01 spine, server-TLS termination, the public HTTPS edge on:443, and the systemd deployment wrapping (aType=notifyservice around the fulcrum launcher with per-tier sandbox hardening, proven on a live Debian cold-boot VM) are all landed. Deployment is framed and tested as condense-from-genesis (the proving ground). The go-public gate (a restore-TESTED backup of the irreplaceable owner seed, structured journald logging, and watchdog liveness) is landed and proven green on a Debian guest: a restored node condenses from its backup and serves authoritatively. The owner control plane landed:zone,obtain,publishandapplyeach drive a running resident, owner-keyed over the loopback fabric, so a node is updated and published to over an authenticated path rather than by editing files on the host. So did the live-host cutover: valis is the primary authoritative server for the registered domains, the lame delegation is closed, a sibling host runs as an AXFR secondary, and zone data lives in PostgreSQL as operator state rather than in files. Connections that outlive a request are supported: the edge no longer assumes a client-speaks-first, reply-only protocol. A production certificate is issued and serving: the public:443answers fordeepsky.systemsunder a Let's Encrypt chain that verifies off-host. The certificate machinery behind it is complete: one credential held per served name, a renewal proven end to end against an authority that resolved the challenge for real, and a deploy that refuses before it moves anything a node cannot run. ⚠ The machinery is in the tree; the live host is behind it. It is authoritative for four zones and holds one certificate, because it is running an older build. Closing that gap is a deployment to a node serving real domains, not a build. What remains on the go-public arc is narrow and specific: that deployment, a cold verification that steered traffic reaches the accept path with the original port recovered, and a TLS-terminated connection the event loop can drive, which is what a continuous protocol over TLS needs and which gates owner mail access.
The shape of it: the hard, novel part (the substrate that makes every protocol a
uniform, capability-gated, user-owned plugin over a migratable namespace) is done and
proven by one real protocol on a real kernel. What remains is breadth (more
protocols, now a c3po exercise against a versioned seam), the live-internet host (the
host is up and authoritative for the registered domains; the ACME/TLS spine, the :443
edge, the supervised systemd deployment, the owner control plane, the go-public gate
(restore-tested backup, observability, watchdog) and a production certificate on the
public :443 are all landed; what remains is deploying the current tree so every served
name is answered under its own certificate, the cold steered-delivery verification, and a
TLS-terminated connection the event loop can drive), and
one genuine external unknown (native content-addressed transport). The arc in progress
and the horizon items at the foot of this file track each of those.
Scope of the first arc
The first arc builds the substrate as horizontal layers and then assembles them
into one real protocol: it extends the proven steering foundation into a real
listener, stands up the 9P namespace fabric, lays the full capability seam,
models a minimal semantic substrate, lands real eBPF-steered file descriptors,
and finishes by answering one IETF protocol (HTTP on :80) end to end as a
standards-compliant server that serves the anonymous published view and the
owner-proven management view on the same port, selected by identity.
Beyond this arc: the durable store and migration (fossil-over-venti, single-writer-per-subtree, generation fencing on evacuation) - now under way as the second arc, tracked below - and, later still, further IETF protocols and content-addressing (libp2p/IPFS). These are designed in docs/ARCHITECTURE.org.
Foundation (done)
[X]Protocol-plugin contract: theprotocolclass and thehandle-connectiongeneric every plugin specialises. (src/protocol.lisp)[X]Port → protocol registry and the dispatch entry point. (src/registry.lisp)[X]Process lifecycle: run / daemon / dev modes. (src/main.lisp)[X]sklookup steering, proven externally: compiled, kernel-verified, and attached to a network namespace via the pure-CL loader, in the Whistler fork.[X]Userland recovery of the original destination port viagetsockname: proven and kernel-independent over loopback.
Build sequence (in order)
The listener and the capability layers can be built over loopback before real
kernel steering lands: getsockname port recovery works identically without
the kernel in the loop. The privileged host agent is independent of the
capability and substrate layers and can proceed in parallel.
[X]Event-loop listener. An iolib(epoll) multiplexer accepts steered connections, recovers each connection's original destination port, and hands it to the existing registry: no synchronous accept loop in the path.- Done when
- a client dialing several loopback ports is accepted by one event loop, each recovered port matches the dialed port, and each connection routes through the registry.
- Note
- the event source sits behind valis's own multiplexer interface (so an iouring backend can replace epoll later) and accepts datagram as well as stream sources (sklookup steers both TCP and UDP).
- Depends on
- nothing - extends the proven port-recovery foundation.
[X]9P fabric and bus: consume seven. valis exposes a person's data, identity, and names as a mountable 9P2000 namespace served by seven, and carries inter-module and inter-node dispatch as 9P file operations.- Done when
- a 9P client walks the namespace and reads a published file;
one module reaches another's mailbox as a 9P file identically whether the
target is a local thread or a remote node; and the host kernel's
v9fsmounts the namespace so ordinary tools (ls,cat) traverse it. - Depends on
- seven, the sibling 9P library, built first in its own repo.
- Delegatable
- seven is a self-contained sibling repository (the 9P2000
client + server), a natural standalone work unit. Consumed: namespace,
bus, and host
v9fsmount all verified; seven evolves on as a sibling.
[X]Transport identity and factotum agent. A Noise-style handshake yields a principal bound to a peer key/DID (oranonymousfor a keyless request) with no explicit login step; a factotum-shaped 9P agent is the sole holder of the owner's keys, so protocol modules never see key material.- Done when
- a keyed peer is presented to the rest of valis as a principal;
a keyless peer is presented as
anonymous; and the factotum agent is a 9P service holding all keys, handing modules only a resolved principal. - Depends on
- the 9P fabric (the factotum agent is a 9P service).
- Landed
- a NoiseXX handshake over the 9P auth path binds the keyed peer's
did:keyprincipal to its connection; a keyless attach resolves toanonymous; the factotum directory (/id) is the sole key custodian, reached only as a 9P service. End-to-end handshake, anonymous-attach, and key-stability tests are green.
[X]Capability tokens and names. Authority is carried as UCAN-mould tokens (delegable, attenuatable, public-key-rooted, offline-verifiable, with the owner as the authorization server), and published data is named by a Tahoe-style capability that embeds its own access right and key.- Done when
- a token can be minted, attenuated, delegated, and verified offline with no central server; an attenuated delegation grants strictly less and reads back as provenance; and a capability-name yields exactly the access it encodes, with no identity required of the bearer.
- Depends on
- transport identity (tokens and caps root in principals).
- Landed
- tokens mint, attenuate, delegate, and verify offline against the owner's key; attenuation is strictly narrowing and the delegation chain reads back as provenance; capability-names grant exactly the embedded right with no bearer identity required.
[X]Enforcement by mounting and two-phase resolution. Resolved authority is enforced by what is mounted into a connection's namespace (a module can name only what is mounted, with no per-access guard and no ambient authority), and authority resolves in two phases: a connect-time principal sets a base namespace, and a per-request capability may mount further within it.- Done when
- a connection can name and reach only what is mounted (an uncovered resource is absent, not merely refused: sovereignty is the absence of a name); there is no per-access check on the hot path; and a request-scoped capability mounts a further subtree without granting ambient authority.
- Depends on
- capability tokens and names. This closes the capability seam.
- Landed
- the assembler builds each connection's view from its resolved grants; an ungranted name is absent from the tree, never refused after admission; read-only grants wrap the target in a server-side projection so write denial is structural, not adapter courtesy.
[X]Minimal semantic substrate. The substrate models the semantic object (a publication / file) independent of any wire protocol, and the first protocol projects over it as a wire-format adapter gated by the connection's capabilities.- Done when
- a publication exists as a semantic object in a 9P subtree, independent of any protocol; a wire adapter reads it only through the mounted namespace; and it is reachable only where mounted.
- Depends on
- enforcement by mounting (the substrate is gated by the seam).
- Landed
- a publication is a directory of attribute files under
/pub, backed by a swappable store seam (the v2 fossil/venti replacement point); two wire adapters (HTTP/1.0 and Gopher) project the same object read-only through capability-mounted views over a real in-process 9P attach, and absence reaches the wire as the protocol's own absence response. Proven end to end over real TCP: both adapters served the same 9P-authored publication. See the publication sections of docs/ARCHITECTURE.org for the seam contracts and validation map.
[X]Edge↔core seam and anonymous scoping. The IETF edge is port-addressed (the recovered destination port selects the module), and an edge module reaches the sovereign core only by carrying its connection's principal and capabilities across a 9P/capability seam (co-located in one image now, sandbox-ready later, never a shared-memory assumption); an anonymous caller reaches a narrow, capability-scoped namespace, never ambient authority.- Done when
- the recovered port alone selects the module; an anonymous caller can name nothing of the owner's authority; and the same edge module works co-located in-image or split across a pipe with no code change.
- Depends on
- the minimal semantic substrate.
- Landed
- the recovered destination port alone selects the adapter
(HTTP/Gopher), which reaches the core only by carrying its connection's
principal and capabilities through a 9P seam: the same adapter serves
byte-identical in-process, over loopback TCP, and over a socketpair with no
code change. The fabric mints one anonymous read-only
/pubgrant; an anonymous caller reads only that view and every out-of-scope axis (/proto,/id,/cap,/edge) is absent from its wire, never merely refused. Request-carried capabilities mount into a fresh private per-request view; per-port connection budgets apply fail-closed backpressure (overflow waits in the kernel backlog, never accept-then-close); a read-only/edgeaxis renders live per-port status.
[X]Privileged host agent and real steering. An external host agent provisions eBPF sklookup steering so every designated port on the unit's IP fans out to valis's single socket, and hands valis its steered listening fd; valis runs unprivileged and never binds the designated ports itself.- Done when
- a connection to a designated port on the real IP is steered to valis's socket and accepted with its original port recovered; valis holds no privileged kernel state; and the agent steers exactly the declared set of ports, nothing outside it.
- Depends on
- the event-loop listener (it consumes the listener's fd). Independent of items 2-7, parallelizable.
- Delegatable
- a self-contained systems unit (eBPF provisioning, the designated-ports contract, and the fd handoff) for someone comfortable with eBPF and Linux network namespaces. It replaces loopback with a real port.
- Landed
- fulcrum, a privileged sibling agent holding
CAP_BPF=/=CAP_NET_ADMIN(the only privileged component: the privilege boundary drawn as a repository boundary). valis declares its ports over a 9P control plane, then hands fulcrum its LISTEN descriptor over a fail-closed, validated SCMRIGHTS receive; fulcrum populates the sockmap before attaching the catchallsk_lookupprogram. valis stays unprivileged and binds only loopback. Proven on a real kernel by fulcrum'smake steer-testnetns gate: a connection dialed at the designated port lands on valis's single steered socket with the original dialed port recovered (four-tuple preserved undersk_assign), valis never binds the designated port, and the netns is the boundary: no leak.
[X]First IETF protocol end-to-end: HTTP on:80. valis answers HTTP as a standards-compliant server on its steered port, serving the anonymous published view and the owner-proven management view on the same port, selected by identity, against a real namespace through the full capability seam. The capstone that assembles every layer below.- Done when
- an ordinary
GETreturns a well-formed HTTP response; an anonymousGETreturns the published view while the same port with the owner key proven returns the management view; and the owner view is reachable only by proving the owner key. - Depends on
- the edge↔core seam and real steering.
- Landed
- the HTTP/1.0 adapter grew into a conformant-but-bounded HTTP/1.1
server:
Hostrequired (400 when absent),GET+HEAD, persistent keep-alive with a per-request read-deadline, conditionalGET→304mapped onto the 9Pqid.versionas a strongETag, and the real status taxonomy withDate=/=Serverheaders. Faithful HTTP, not a skin. The owner proves possession of the Ed25519 custody key per request via RFC 9421 HTTP Message Signatures, verified as a third method on theauthenticateseam; the edge relays only the signed material and never sees the key. One URL space, identity selects the mounted namespace: an anonymousGETresolves to the read-only/pubpublished view, an owner-signed request to the full canonical frame as a browsable read projection, and an anonymous caller names nothing of management (404, never403). The same anonymous-/pubpublished view is now consistent across both edges: an unauthenticated HTTPGETand an unauthenticated kernelv9fsmount see the same published floor, while/proto /id /cap /edge /busstay credential-gated. Replay is bounded (created=/=expireswindow + a nonce cache committed only after the signature verifies). Proven end to end: the full cold suite is green (1557/0, incl. the real-NIC auth-seam crossing and the kernel v9fs mount), the auth surface is threat-audited (09-SECURITY.md, no open threats), and the operatormake steer-testgate answers a real client on the steered:80. Thesevendependency floor is pinned at the version that carries the close/EOF and node-qid capabilities valis requires. The first delivery arc is complete.
Where work can be delegated to collaborators
Through the first arc, the core foundation - listener, 9P fabric, the capability seam, and the semantic substrate (items 1-7) - stays architect-held: it is the spine, and the dependency ordering is reviewed before each piece begins. Delegation opens at the edges and after:
- seven (the 9P library): already a separable sibling repository and the keystone dependency for item 2. The clearest standalone unit; in progress.
- The privileged host agent (item 8): independent and parallelizable; a contained eBPF/Linux systems task.
- The IETF protocol modules: the prime collaborator surface, opening once the
substrate interfaces and the first protocol (HTTP on
:80) exist. Each further protocol (SMTP, IMAP, NNTP, DNS, and the rest) is a self-contained wire-format adapter over an already-established semantic subtree, gated by capabilities, addable without inventing new storage or identity machinery. - libp2p/IPFS bindings (a later milestone): a separable CFFI-binding sibling repository for content addressing.
Second arc - durable store and migration (complete)
The second arc gives the sovereign namespace a durable, migratable backing
store: content-addressed immutable blocks under one generation-fenced mutable
head (fossil-over-venti), single-writer-per-subtree authority,
reassembly-not-transfer migration with evacuation fencing, and
coincident-instance caching that preserves read-your-writes. It is built in-tree
under src/store/ from pieces valis already owns (no new dependencies, no new
9P wire surface) as an import-clean package cluster a later extraction can move
out as files plus an .asd. The on-disk format and algorithms are grounded
against the Plan 9 venti source; fossil is absent there, so the mutable head is
venti-style content addressing plus a compare-and-swap on a monotonic root, not
fossil. The design of record is docs/ARCHITECTURE.org.
Build sequence (in order)
The store is built strictly bottom-up: block, then tree, then the generation-fenced head (the spine), then the manifest, the fabric adapter, reassembly, the fence↔revocation unification, caching, and the end-to-end migration proof. The fossil/venti replacement point named in the first arc's item 6 is the store-store adapter of item 14.
[X]Content-addressed block store. A write-once, hash-addressed, integrity-verified block primitive: the foundation everything else hangs off.- Done when
- writing octets returns a content address whose hash recomputes
from the stored bytes; a corrupted block is detected on read, never served;
identical content dedups to a no-op; and a block reported written survives a
kill -9plusdrop_caches. - Landed
- a file-per-block store over portable Lisp stream IO with the
proven
fs-storefdatasync→rename→dir-fsync durability discipline, SHA-256 behind a swappablestore-block-digestseam, and an in-memory index plus bloom filter so a miss avoids a seek.
[X]Tree codec. A single hash names a whole subtree; snapshots share unmodified blocks by content identity.- Done when
- one root hash resolves a directory tree to its leaf blocks and the same logical tree always encodes to the same root hash; changing one leaf writes only new blocks on the path to the root; and a snapshot is an O(1) retained root that copies no bytes.
- Landed
- deterministic, dedup-friendly directory and file node encoding through the block seam, importing nothing from valis's capability/namespace packages.
[X]Generation-fenced mutable head: the spine. A monotonic, generation- fenced root pointer advanced by compare-and-swap, where a superseded write is rejected at the commit and a recovered head always names a fully-durable tree.- Done when
- a write carrying a superseded generation is rejected at the commit even after an injected stop-the-world pause lands it arbitrarily late (the Kleppmann fencing-race proof); the generation is strictly monotonic, 64-bit, store-allocated; every block is durable before the head names it; and a recovered head needs no repair pass.
- Landed
- CAS-advance of the root pointer via an atomic rename as the single
linearization point, validated by a
kill -9+drop_caches+ injected-pause fault-injection harness.
[X]Namespace manifest. The mount table made literal, living inside the mutable tree, reachable from the head, so it migrates atomically.- Done when
- the serialized manifest contains no key bytes and no derivation seed (a structural bar with a regression test); it pins a specific generation and everything it references is durable before it is published; a first boot with no manifest reproduces today's axes exactly; and a manifest with a bad hash or a missing root is rejected fail-closed.
- Landed
- a versioned manifest inside the durable tree referencing only key locations and trust-domain identifiers, defaulting to the live six-axis set so a fresh boot is byte-identical to the first arc.
[X]store-store adapter and fabric wiring. The durable store dropped into the running fabric behind seven's backend node layer: no new 9P wire surface.- Done when
/pubserves byte-identical results over the durable store as overfs-store(no first-arc regression);start-fabricselects the durable backend with a one-line swap; a/pubwrite goes through the head CAS as a single-writer mutation; and the subsystem cold-builds with the new backend selected.- Landed
- the publication store seam implemented over head and tree, selected at fabric start-up, with mtime/version synthesized from the head generation rather than a POSIX mtime.
[X]Assembler manifest path and reassembly. A fresh instance reconstructs the namespace from the manifest plus the owner's keys: reassembly-not-transfer.- Done when
- a fresh instance reads head → manifest → roots and reassembles the same namespace; reachability of every referenced root is verified before the namespace is declared assembled, fail-closed on any missing root; a clean install with no manifest reassembles today's axes; and the destination cannot come up without an out-of-band key acquisition step.
- Landed
- the assembler's canonical-frame builder gained a manifest-driven path; keys arrive out of band through the existing factotum/custody path and never travel in the manifest.
[X]Fence ↔ revocation unification. Evacuation fencing and capability revocation collapsed into one append-only monotonic authority log.- Done when
- standing up a new instance bumps the generation as its first durable act and the store rejects the superseded instance's subsequent writes; that superseded generation is expressed as a revoked write capability on the existing append-only revocation log, checked on the same write path as a revoked hash; live write-capable grants under a fenced generation are swept from active sessions; and single-writer authority is enforced at the store, not by a clock-based lease.
- Landed
- a write capability carries a generation stamp inside its owner-signed bytes and is rejected at verify time on the same membership check as a revoked token, with the head CAS as the commit-point backstop and a write-right eviction sweep for live grants, so fencing and revocation cannot drift apart.
[X]Caching coherence andqid.version. Coincident instances get read-your-writes against the single-writer head without funneling all reads through it.- Done when
- a write through one instance is visible to a coincident
instance within the staleness bound, without the reader funneling through
the writer; an immutable-block cache hit is always valid (the score is the
validator) and only the head needs revalidation, keyed to the current
generation;
qid.versiontracks the generation so a conditionalGET/ ETag returns304and kernelv9fscaching stays correct under mutation across instances; and the cache honours an explicit byte budget with every pin released on every exit path: no pin leak, no finalizer-based release. - Depends on
- the generation-fenced head (item 12) and the store-store adapter (item 14).
- Landed
qid.versiontracks the head generation so a conditionalGET/ ETag returns304and kernelv9fscaching stays correct under cross-instance mutation; an immutable-block cache hit validates by its score and only the head revalidates, keyed to the current generation; the cache honours an explicit byte budget with every pin released on every exit path. Thesevensibling carries the matchingqid.versionbackend stamp.
[X]Migration/evacuation end-to-end. The thesis proven end to end, with the auditor-facing architecture document shipped.- Done when
- a fresh instance reassembles from manifest plus out-of-band keys and serves the namespace while the evacuated instance's writes are fenced and its live grants swept; an old instance paused across the generation bump cannot land its commit (the fencing race re-verified end to end, across a GC pause); a confused-deputy audit confirms no store, backup, or observability path ever reaches decrypted content (the store handles opaque blocks only: encryption-ready, not encrypting); and docs/ARCHITECTURE.org is updated to capture the fencing/CAS/migration flow.
- Depends on
- reassembly (15), the fence↔revocation unification (16), and caching coherence (17).
- Landed
- a fresh instance reassembles from manifest plus out-of-band keys and
serves the namespace while the evacuated instance's writes are fenced and its
live grants swept; an instance paused across the generation bump cannot land
its commit (the fencing race re-verified across a GC pause); a confused-deputy
audit confirmed no store, backup, or observability path ever reaches decrypted
content (opaque blocks only: encryption-ready, not encrypting); and
docs/ARCHITECTURE.org carries the fencing/CAS/migration flow. The
sevensibling carries the matching drain-on-evacuation teardown. The second arc is complete.
Where work can be delegated in the second arc
The store is the spine of the durable substrate and stays architect-held, built in-tree with the dependency ordering reviewed before each piece begins. The boundaries open in the same places as before:
- The one
sevendependency (a bulk-transfermsizeceiling for the read/walk/write path) lands in seven as a sibling repository; this arc may depend on its arrival but plans no seven-internal work. - libp2p/IPFS content-addressed block transport remains a later, separable sibling unit: the future repository boundary is the block transport, not the store itself, which is why the store cluster is kept import-clean from day one.
Third arc - plugin composition (complete)
Every protocol becomes a lifecycled plugin. A protocol is a module (a
registered, lifecycled unit on Shinmera's modularize), and the edge
controller's hard-coded adapter list becomes an *edge-adapters* registry that
modules install into on load. The same module abstraction then carries through
capability-gated admission and a migratable module manifest, so an instance's
identity is its namespace manifest plus its module set. valis owns the
plumbing; the protocol adapters themselves live in the c3po sibling
repository. The design of record is docs/ARCHITECTURE.org §"Plugin composition".
[X]The plugin seam. The edge controller's hard-coded adapter list becomes an installable registry of self-registering protocol modules, and valis gains the modularize-facing plugin infrastructure (the combined module class, the option-expander, and an ordered teardown hook), so adding a protocol needs no controller edit.- Done when
- valis defines and exports one class that is both an
asdf:package-inferred-systemand amodularize:virtual-module, so a module's.asdnames it as:classand keeps one-package-per-file derivation while gaining a lifecycle; the edge controller binds adapters by iterating the registry and names no concrete adapter; deleting a module runs an ordered teardown (deregister source → drain in-flight connections → unregister adapter → remove the edge entry) before its packages unbind; and the resident HTTP and Gopher adapters self-register with the full cold suite green. - Landed
- a combined
module-inferred-systemclass (package-inferred-system before modularize's virtual-module in the precedence list), the*edge-adapters*registry replacing the hard-coded%adapter-specs, an idempotent:edge-adapteroption-expander firing inexpand-option(after modularization, never as a too-early hook), and a drain-then-retiredelete-modulehook. HTTP and Gopher self-register purely by being loaded; a compile-then-load double-fire is a benign no-op; thestart-edgecompat surface keeps every existing call-site unchanged. The exported seam surface (registry, register fn,:edge-adapterkeyword, combined class) was handed to c3po, which carries the protocol adapters on its own roadmap.
[X]Capability-gated module admission. Loading a module is a capability decision over its content hash: an instance compiles or loads a module only when an owner-rooted authority has vouched for that exact hash, so a recondensing instance is never an open code-execution endpoint.- Done when
- a module whose content hash an owner-rooted authority vouched for loads, the decision made as a capability check over the hash and not a bare fetch; an unvouched, wrong-hash, data-mount-only, or revoked hash is refused fail-closed: it never reaches load and leaves no partial module state; and the vouching authority chains to the owner root on the same verification path as the rest of the capability machinery, never a parallel trust check.
- Depends on
- the module abstraction of item 19.
- Landed
- an additive
:admitright distinct from the data:mountright (a data-mount capability can never authorize code execution), a canonical bundle hasher content-addressing a module's.asdplus its source with length-prefixed, path-sorted framing so no re-partitioning of the same bytes collides and on-disk discovery order does not change the address, and a singleadmit-modulechokepoint that verifies an owner-rooted:admitvouch over the module's/module/<hash>and loads exactly once. Revoked, wrong-hash,:mount-only, and unvouched requests refuse before the loader ever runs. The fail-closed and load-once invariants are pinned by direct assertions (a counting loader proven to fire once on a valid admit and zero times on every refusal, with the adapter registry asserted unchanged after a refusal), and the gate reuses the existing owner-rooted authority and revocation machinery, minting no new crypto. Cold suite green (1122/0).
[X]Module manifest and migration. A durable, content-addressed record of an instance's active module set (distinct from the namespace manifest), so recondensation reassembles the module set from the vouched hashes: reassembly-not-transfer for code.- Done when
- an instance records its active module set as a durable, content-addressed manifest distinct from the namespace manifest; a destination reads the manifest, pulls the vouched hashes, verifies and compiles each through the item-20 admission gate, and serves the same protocols as the evaporated instance; and a manifest naming an unvouched or unreachable module hash is rejected fail-closed, never standing up a partial module set.
- Depends on
- capability-gated admission (item 20).
- Landed
- a durable, content-addressed module manifest distinct from the namespace manifest, co-published with it under one genesis root; the evacuative stand-up reads it, pulls the vouched hashes, verifies and compiles each through the admission gate, and stands the same protocols up at the successor. A refusal aborts fail-closed with the adapter registry restored, so no partial module set ever boots. A post-delivery review pass hardened the re-vouch path (a fresh-entropy nonce per mint, a fail-closed set-completeness guard, and refusal of source paths that escape the module root). The third arc is complete.
Where work can be delegated in the third arc
valis owns the plugin plumbing: the module class, the registry, the admission gate, and the manifest stay architect-held. The boundary opens at the adapters:
- c3po (the protocol adapters): a separate sibling repository. With the seam landed and its exported surface signalled, c3po has carried its HTTP, Gopher, and Finger adapters onto valis's real edge seam (the conversion is complete); they ride reassembly as ordinary vouched modules through the admission gate. SMTP is parked for the messaging substrate (architect-led).
- libp2p/IPFS as the live module-distribution transport remains the named gate for a later milestone: content addressing stays local and seam-shaped here, so the admission gate is transport-agnostic when that distribution path arrives.
Fourth arc - factotum and transport-identity adoption (complete)
valis's transport identity and key custody were built in-tree during the first arc. The fourth arc moved the factotum (the identity and key-custody agent) into its own sibling repository, mercer, so the constellation has one home for transport authentication and custody that valis and other consumers depend on rather than each carrying its own copy. It adopted the extracted NoiseXX authenticator, consolidated the post-attach transport cipher into mercer with each session sealed end to end, and proved the responder against an independent published transcript carried over the real wire. The design of record is docs/ARCHITECTURE.org.
[X]Adopt mercer as the factotum home. valis cut over from its in-tree NoiseXX transport identity and RFC 8439 AEAD to depend on the mercer sibling, and the handshake gained validation against independent published vectors.- Done
- valis depends on mercer for the transport authenticator and the in-tree copies are retired, with the handshake/anonymous-attach/key-stability behaviour unchanged; the NoiseXX handshake is checked against independent published vectors (not only valis's own two-party self-interop), backed by a short cryptographic review; and the 9P auth seam the authenticator sits over is agreed with the seven sibling.
[X]Wrap the post-attach session. The transport cipher is consolidated into mercer (mercer owns seal/unseal/close-session); valis owns the keyed accept loop and installs the sealing endpoint at the attach boundary, so every record after attach is sealed end to end.- Done
- a keyed peer's post-attach 9P traffic is sealed under per-session transport keys; the record framing bounds allocation against a hostile length prefix; a tampered record fails closed to teardown with no plaintext; and the wrap is validated directly against the real keyed accept loop.
[X]Over-the-wire initiator. A remote initiator authenticates end to end over the wire and reaches its authenticated view, with the on-wire transcript anchored to a standard-conformant NoiseXX vector rather than valis's own self-interop.- Done
- a real initiator dials valis over the wire, completes the keyed handshake and 9P attach, and reads a principal-scoped node over the sealed session; the handshake and transport bytes match an independent published vector; and the documented real-NIC sealed-attach procedure is shipped for the operator to run out of band.
Where work can be delegated in the fourth arc
The factotum is identity-critical and its seam stays architect-held, but the work is inherently cross-repo:
- mercer (the factotum): the sibling repository that becomes the home of the transport authenticator and, later, the broader key-custody surface. It owns the independent-vector validation and the cryptographic review.
- seven (the 9P keystone): owns the 9P auth seam the authenticator sits over; coordinates the interface mercer's handshake rides.
Fifth arc - the mail transport spine (complete)
The fifth arc stands up valis's sovereign mail-transport core: the substrate that
moves a message end to end through the capability-gated namespace, with the wire
engines that speak SMTP and IMAP left to the c3po protocol sibling. It models a
message as an immutable content-addressed body plus a separate transport-state
record, moves it through a durable queue and a fail-closed router (no open relay),
projects the whole spine as a capability-scoped /mail namespace axis where a 9P
write is the authenticated submission, and proves the assembled spine moves real
mail end to end against the live image with the wire stubbed, the body an opaque
block no operations path can name. Modern exchangers reject unauthenticated mail,
so being a legitimate sender ultimately means being authoritative for our own DNS;
that, and the SMTP/IMAP wire engines, are the arcs beyond this one. The design of
record is docs/ARCHITECTURE.org.
[X]Message-object model over the store. A submitted or received message is an immutable, content-addressed body (opaque RFC 5322 octets: never parsed, never mutated) plus a transport-state record carrying the envelope and lifecycle, so the deferred retry / bounce / relay builds are seam-fills.- Done
- bodies dedup by content address and ride namespace migration as ordinary store blocks; the envelope and transport state live in a separate mutable record keyed by the body's address.
[X]The durable transport queue. A fenced, fsync-durable queue over the content-addressed head, with the acknowledged-write contract: a submission returns success only after its queue entry is durable.- Done
- queue entries advance by compare-and-swap off the immutable body so the body needs no lock; the incoming / active tiers carry the day-one fields (attempt count, next-attempt time, last status) the deferred drivers fill.
[X]The router with a fail-closed open-relay gate. Each recipient is classified local-delivery versus relay; relay is deny-by-default, decided by a submit-capability check reusing the existing capability machinery.- Done
- a message relays only for an authorized submitter; an unauthenticated non-local relay attempt (peer-landed mail for a non-local recipient) is refused, never silently forwarded: the blocking open-relay guard. Sovereignty as a capability check, not a flag.
[X]The mail-adapter seam. A*mail-adapters*registry plus an inbound landing entrypoint and an outbound drain driver, the opaque "hand these octets to the wire" contract with the c3po protocol sibling.- Done
- the spine is decoupled from c3po's delivery: the adapter returns a delivered / deferred / bounced disposition, and the spine is provable end to end with the wire adapter stubbed.
[X]The/mailaxis and 9P-native submission. A capability-scoped/mailnamespace axis (outbox / queue / inbox / ctl) on the qmail Maildir model, where the owner submits by writing a message into the outbox.- Done
- a capability-mounted 9P write is the authenticated submission, durable before the write returns; sovereignty-as-absence holds on the queue projection: an ops grant sees envelope and status only and has no name for any body or for the inbox.
[X]Local end-to-end proof and the sovereignty audit. The spine moves a message end to end against the live image (native submit → ingest → queue → route → {local-deliver | relay-drain to a stub adapter}, one each way) with the open-relay guard re-proven and the content boundary audited.- Done
start-fabricbinds the mail seams so/mailis live in the owner frame; a confused-deputy audit confirms no operations / backup / observability path reaches a message body; and docs/ARCHITECTURE.org ships the mail-flow, open-relay-guard, and content-boundary deliverable.
Where work can be delegated in the fifth arc
The spine is the substrate; the wire is the sibling's. The split holds across the subsystem:
- c3po (the protocols): owns the SMTP and IMAP wire engines that ride the mail spine through the adapter seam, and (for the deliverability arc beyond) the SPF / DKIM / DMARC evaluators. valis owns the substrate, queue, router, and the adapter seam they plug into.
- mercer (the factotum): for the deliverability arc, owns the DKIM key custody and the sign / verify crypto; the contract with c3po stays opaque ("sign these octets") so the private key never crosses into the parser.
Sixth arc - authoritative DNS substrate and the PostgreSQL operator-state seam (in progress)
The sixth arc gives valis a mutable, name-keyed PostgreSQL operator-state seam - the
ACID counterpart to the content-addressed store, holding the records an operator
mutates (domain names, social graph, message state) where a body never enters the
database - and, over it, an authoritative DNS substrate: the domains valis
terminates, authored through the capability-gated :names axis and served from an
in-memory view that answers off the database hot path and degrades to serve-last-good
across an outage, behind a versioned, fail-closed adapter seam a capability-admitted
wire engine binds to. It follows the fifth arc's split: valis owns the substrate,
the controller, and capability admission; the runciter sibling owns the DNS
substrate (zone model over PG, :names semantics, BIND text parsing, resolver
policy); the binary DNS wire engine on :53 is c3po-dns, stubbed locally here.
Sequencing is locked: the operator-state seam is built first as the foundation, and
the DNS substrate reads zone data from it. The design of record is
docs/ARCHITECTURE.org.
Build sequence (in order)
The seam is built foundation-first: the PostgreSQL operator-state seam, then the zone
schema and the :names door over it, then the DNS adapter seam and in-memory zone
view, then the secondary-first cutover substrate. The runciter sibling scaffold and
the versioned seam contract overlap the foundation because the cross-repo coordination
with c3po-dns has lead time.
[X]Scaffold runciter and the DNS adapter seam contract. Stand up the runciter DNS-substrate sibling and define the versioned seam contract the:53wire engine binds to, stubbed locally, so the cross-repo interface is versioned before the substrate that fills it is built.- Landed
- the runciter sibling exists and carries the DNS substrate; valis names a versioned, fail-closed adapter seam and no concrete wire engine, with the c3po-dns binding deferred to its own roadmap and stubbed here.
[X]The PostgreSQL operator-state seam: the foundation. The mutable, pooled, fenceddao-classseam every later axis reads through, where a body never enters the database and two valis sharing one database cannot both write.- Done when
- the seam is wired into the fabric boot/teardown path; the store generation is projected into PostgreSQL fence-first on evacuation, so a superseded instance cannot write; and a body is structurally barred from the database (pointer-hash only).
- Landed
- the fenced
dao-classpool stands up and tears down with the fabric; the generation projects fence-first on evacuation, proven across the real fabric evacuation path so coincident instances on one database cannot both write.
[X]Zone schema, the:namesaxis, and the BIND text parser. Zones live in PostgreSQL and are mutated only through the capability-gated:namesaxis; legacy BIND zone files parse correctly into:namesrows.- Done when
- a zone is authored and mutated only through
:names, never by reaching the table directly; and a legacy BIND zone parses into canonical rows. - Landed
- the zone schema lives behind the operator-state seam, the
:namesaxis is the sole capability-gated door to it, and the BIND text parser canonicalises legacy zones into rows.
[X]The DNS adapter seam, the in-memory zone view, and the stub-adapter proof. Prove the seam end to end with the wire stubbed: a decoded query resolves through the in-memory zone view, independent of the database hot path, admission fail-closed.- Done when
- a decoded query resolves through the zone view to a record set and result code; a kill-PG-mid-load leaves the last good view serving; and a DNS wire module is admitted only as a capability decision, fail-closed.
- Landed
- the in-memory serving view answers off the database hot path and degrades to serve-last-good across a PG outage; the versioned adapter seam resolves a decoded query end to end against a stub wire engine; admission is a capability decision.
[X]The cutover substrate: secondary-first zone sink/source. Ingest a legacy zone as a secondary, track its SOA-timer and cutover policy, and support the outbound-feed and serial-bump data side, so a domain can be moved onto valis before valis answers as its primary.- Done when
- a zone is ingested as a secondary and tracked through the cutover lifecycle (held → answer-as-secondary → primary, operator-gated); a held or expired secondary never enters the serving view; serve-time expiry is enforced fail-closed against the persisted absolute deadline; and cutover legality is decided by runciter through the seam, never re-implemented.
- Landed
- the per-zone
/ctlflips (answer-as-secondary → promote-to-primary) with runciter-decided legality through the serving seam and a fenced persist; the answerability gate is centralised at the refresh-zone install chokepoint; serve- time expiry is enforced against the persisted absolute deadline (restart-safe bigint epoch timers); and the outbound zone-change feed is fault-isolated. The runciter pin is tightened to0.3.0with the typedrdata-parse-erroradopted fail-closed on the import and per-zone read boundaries. Merged tomainvia PR #32 (352c2ff).
[X]Resolver policy and the mail-drain seam. runciter's MX resolution policy, wired into the v1.4 mail drain to close the deferred MX-over-DNS slice, so an outbound message resolves its recipient's exchanger through the sovereign DNS substrate rather than a stubbed lookup (the resolver wire itself stubbed).- Done when
- the mail drain resolves a recipient MX through runciter's resolver policy behind the seam; the resolver wire is stubbed but the policy and the seam are exercised end to end; and the deferred MX-over-DNS slice in the mail spine is closed.
- Landed
- the mail drain routes relay entries through an injected resolver and maps
runciter's four MX outcomes onto the delivered/deferred/bounced queue vocabulary,
proven end to end over stub-wire DNS with no live wire. The seam is versioned around
runciter's
resolve-mail-exchangers; the wire client fills it later. Merged tomainvia PR #37. The real-wire cutover (inject the live resolver at the production boot seam and retire the smarthost) is item 41 below, still open.
[X]Operator-state migration and the per-axis sovereignty model. Carry the mail-transport state and the social graph onto the PostgreSQL seam, then document and audit the per-axis sovereignty boundaries.- Done when
- mail-transport state and the social graph live on the operator-state seam alongside the DNS zones; and the per-axis sovereignty model is documented and audited. One principle, three boundaries: a content boundary for mail bodies, write-authority for DNS zones, capability-scoped access for the social graph.
- Landed
- mail-transport state and the social graph sit on the operator-state seam alongside the DNS zones; the per-axis sovereignty model is documented and audited: the content boundary for mail bodies, write-authority for DNS zones, and capability-scoped access for the social graph each hold as a distinct axis.
[X]valis becomes the pure ndb DATA owner: the nameserver serving layer moves to runciter. Conform the DNS code to the architecture of record: valis holds the authoritative zone data and dispatches connections; the logical nameserver (runciter) holds the serving index and composes every answer.- Done when
- valis holds no DNS answer logic and no serving cache: it exposes
the zone data through the
zone-data-sourceread protocol (plus the held-inclusive origin set and the per-zone cutover status the nameserver gates answerability on), and the bind registers runciter's serving handler at the agreed contract version, fail-closed; a held or expired secondary still never answers, now enforced from the cutover-status data plus runciter's gate rather than a valis serving view. - Landed
- the in-memory serving layer (the loaded-zone index and the RFC 2308
answer composition) is shed from valis and relocated to runciter, parity-checked
branch-for-branch before the shed;
bind-dns-viewregisters(make-dns-service-handler <pg-zone-source>)via late resolution at contract-version 1; the transitional serving-seam fallbacks are deleted (the seam is thin late-bound delegation, fail-closed); serve-last-good is handed to runciter with an explicit audit confirmation rather than dropped;docs/CONSTELLATION.organddocs/ARCHITECTURE.orgrecord the split. Pending merge tomain; the runciter pin tightens to its serving-module tag when that repo is merged and tagged.
[X]TSIG-gated zone-transfer allowlist and serial monotonicity. valis owns the transfer-peer allowlist as capability-mediated operator-state and enforces RFC 1982 serial monotonicity on the transfer path; no zone reaches the wire except to an allowlisted, authenticated peer.- Done when
- a zone leaves valis only to an allowlisted peer, the allowlist is mutated only through the capability-gated operator-state door, and a non-monotonic serial is rejected before it can be emitted.
- Landed
- the allowlist verdict is injected into runciter's authorize-before-emit
gate through the seam (
bind-transfer-authorizermirrorsbind-dns-view), serial monotonicity is carried on the transfer path, and the gate is proven end to end across the repo boundary. Merged tomainvia PR #42 and PR #43 (04f41a4).
[X]Authoritative answers on:53through fail-closed admission. valis answers authoritative DNS on:53: a steered connection is dispatched to the admitted DNS module fail-closed, and with runciter's serving orchestration and c3po's wire codec landed, the answer is externally probeable: non-recursive and amplification-bounded.- Done when
- a steered
:53UDP/TCP descriptor is admitted fail-closed and pumped through the wire codec onto runciter's serving exchange; the serve loop starts from boot once the DNS view binds and disarms on module revocation; and an out-of-band probe from a stranger vantage confirms authoritative, non-recursive answers with EDNS0@1232 and TC/TCP fallback bounding amplification. - Landed
- two steered
:53fds sit behind one admitted handler; queries pump through c3po's codec onto runciter'sserve-exchange; the loop boots with the DNS view and retires fail-closed on revocation (a revoked fd stays live yet serves nothing); and an out-of-band probe from a stranger vantage confirms authoritative, non-recursive answers with amplification bounded. Merged tomainvia PR #44 (7b6579d).
[X]Real-wire mail-drain resolver cutover: retire the smarthost. Inject runciter's live MX resolver into the production outbound drain and remove the static smarthost fallback, so an outbound message resolves its recipient's exchanger over the real DoT wire, fail-closed against a hung or hostile upstream. The stub-wire seam (item 36) is the substrate; this closes it onto the live wire.- Done when
- the production drain boots with the live resolver as the MX source and the smarthost fallback is gone from the delivery path; a resolvable recipient delivers to its preference-sorted MX; a transient failure (SERVFAIL, timeout, TLS or auth failure, budget exhaustion) defers and never loses the message; a null-MX or NXDOMAIN bounces; the resolve is deadline-bounded so a hung upstream defers rather than stalling the drain thread; a wrong-cert DoT upstream defers, never delivering over an unauthenticated or downgraded channel; and the resolver stays client-only, with no inbound listener or accept loop.
- Landed
- the production drain builds runciter's live DoT resolver at boot (late-bound,
so valis stays cold-loadable without it) and resolves each relay recipient's MX over the
real wire; the static smarthost is gone; an unresolved, unconfigured, or empty-trust
upstream DEFERS rather than delivering; the open-relay guard is unchanged. All mail suites
green, cold-hermetic. Merged to
mainvia PR #45 (4ba6004), over runciter'smake-mail-resolver(PR #19).
Where work can be delegated in the sixth arc
The substrate and the seams stay architect-held; the wire and the resolver engine are the siblings'. The split holds as in the prior arcs:
- runciter (the DNS substrate): owns the zone model over PostgreSQL, the
:namessemantics, BIND text parsing, and the resolver policy; coordinated cross-repo, with merge and tag operator-gated. - c3po-dns (the wire): owns the RFC 1035 binary engine on
:53(serving, AXFR/NOTIFY, the resolver wire client), on its own roadmap and gated on valis signalling the DNS adapter seam; stubbed locally through this arc.
Seventh arc - the resident steered path (in progress)
The first arc proved eBPF sk_lookup steering end to end on a real kernel, but in the
privileged fulcrum sibling, external to the valis process: valis still listened on a
loopback stand-in. The seventh arc makes the steered descriptor the valis process's own
resident listening path: valis mints its designated-port LISTEN descriptor in
production, receives it across a fail-closed process boundary, and boots its steered
edge sources on the pushed descriptor, so the target data flow of the first arc
becomes the running one. What remains is hardening the cross-process push to be
race-free at the wire and the privileged launcher that brings the whole resident unit
up under a real network namespace. The design of record is
docs/ARCHITECTURE.org.
Build sequence (in order)
The primitive lands first (the descriptor crossing and the shared fd-passing seam), then the resident boot that consumes it, then the two hardening pieces: a race-free push at the wire and the privileged launcher for the live-kernel bring-up.
[X]The descriptor peer and the shared fd-passing seam. AnSCM_RIGHTSdescriptor-passing primitive with fail-closed admission: a single caller-owned descriptor crosses the process boundary and every crossing goes through one shared seam.- Landed
- the
SCM_RIGHTSprimitive gates each crossing through fail-closed admission (a multi-descriptor control message is refused, a sent descriptor stays caller-owned, syscall returns are checked), proven cold across a real process boundary by a fresh-SBCL-child spawn harness; the edge passes every descriptor through the one shared seam. Merged tomainvia PR #48 and PR #49.
[X]The resident boot on the pushed descriptor. valis boots its edge on the steeredLISTENdescriptor pushed from the privileged host agent, fail-closed: the resident listening path replaces the loopback stand-in.- Depends on
- the descriptor peer and the shared seam (item 42).
- Landed
- valis mints its designated-port
LISTENdescriptor in production and a resident boot brings the steered edge sources up on the descriptor pushed over a 9P control client; a--residentdispatch boots fail-closed with the shipped adapter set, and the bounded control-push read closes a failed dial. The push sends the descriptor only after the control-port write drains, and the crossing is proven over a real process boundary. Merged tomainvia PR #51.
[X]A race-free descriptor push. The production descriptor push crosses race-free at the wire, so no buffered read on the receiver can over-read the carrier byte and silently discard the passed descriptor.- Done when
- the sender pushes the descriptor immediately after the control frame with no out-of-band readiness marker and the descriptor still crosses race-free; the carrier byte is only ever consumed by a control-armed receive, never a plain buffered read; and ordinary control frames stay byte-identical.
- Depends on
- the resident boot (item 43). Today's production push is fail-fast: it makes the race loud with a read deadline, not gone.
- Note
- the reader that guarantees this is owned by the seven sibling (its
AF_LOCALcontrol-endpoint reader); valis's sender adopts it by swapping its push seam onto seven's primitive with no construction change, retiring the in-test readiness barrier and the hand-rolled raw-frame push while keeping the read deadline as defence-in-depth. fulcrum's receiver adopts the matching guard. Merged tomainvia PR #55.
[ ]The privileged resident launcher. The host agent brings the resident unit up under a real network namespace and hands over the descriptor across a well-known control socket, so the steered path comes up from a single privileged bring-up rather than a manual stitch-together.- Done when
- the fulcrum host agent launches valis into the network namespace it
steers and hands over the
LISTENdescriptor across a well-known control socket, and the resident unit is reachable on its steered port without a manual bring-up. - Depends on
- the resident boot (item 43); independent of the race-free push (item 44).
[ ]Live-kernel capstone. The whole resident path (steer, push, resident boot, answer) proven end to end on a real kernel from an out-of-band operator run.- Done when
- an out-of-band operator run steers a designated port on a real kernel, pushes the descriptor to a resident valis, and a stranger-vantage client is answered on the steered port through the resident edge, with valis holding no privileged kernel state.
- Depends on
- the race-free push (item 44) and the privileged launcher (item 45).
Where work can be delegated in the seventh arc
The resident path is the valis process's own and stays architect-held; the privilege stays across the repository boundary, as in every prior arc:
- seven (the 9P keystone): owns the race-free control-endpoint reader the descriptor push rides; valis and fulcrum adopt it as consumers with no re-plumb.
- fulcrum (the privileged host agent): owns the network-namespace launcher and the well-known control socket that bring the resident unit up on a real kernel, and adopts the receiver-side guard for the race-free push.
Eighth arc - the live internet host (in progress)
valis becomes publicly reachable. The resident path proven in the seventh arc is turned outward: the privileged fulcrum launcher binds the public ports in a real network namespace and hands the descriptors to an unprivileged valis, which serves without a privileged bind. Deployment is unified with migration (standing a node up is the genesis case of condense) and made reproducible and testable rather than hand-stitched. The remaining pieces obtain and terminate TLS so the edge is publicly reachable over HTTPS. The design of record is docs/ARCHITECTURE.org.
Build sequence (in order)
[X]Public authoritative DNS on:53. The resident serves authoritative DNS to the public internet on UDP and TCP, no geoblock, CAA answered correctly, reachability proven off-host, the dns-01 precondition.- Landed
- fulcrum binds
:53UDP/TCP in the netns and exec-inherits the descriptors to an unprivileged valis, whose resident boot adopts them and drives the sibling serve loop over them with apg-zone-source: no privileged/loopback bind. CAA and TYPE257 answer NOERROR/NODATA; the serving path is proven end to end off-host in a real-kernel integration gate, and a multi-region reachability harness proves worldwide answerability and gates the public go-live. Merged tomainvia PR #58.
[X]The host-deployment contract and the deployment proving ground. A durable, declarative harness that stands Debian guests up from source and drives valis's own deploy == condense-from-genesis path under assertion, plus the written contract.- Landed
docs/deployment/host-deployment-contract.orgstates the prerequisites, bring-up order, provisioning, teardown, and fail-closed invariants.proving-ground/pins a Nix verification fabric, boots a two-location VLAN-tagged Debian topology via Terraform-libvirt, provisions with cloud-init, and drives three scenarios (deploy, evacuate, restore) over one mechanism through an in-image Lisp/SSH assertion layer (valis-proving-ground, its own ASDF system). The scenarios assert the shipped condense/fence path; the harness is unit-green against fixtures, with a live libvirtd run the one out-of-band step. Merged tomainvia PR #59.
[ ]The active-module (resident/lifecycle) seam. A start/stop/supervise module kind with capability-scoped egress, boot-supervised, proven with a trivial no-op module: the non-reactive second module archetype the ACME manager will consume.- Built
- the archetype exists and the renewal manager registers through it; module admission runs from the durable manifest.
- Not yet true on the live node
- no active module has ever been admitted there.
- Corrected since
- this used to read as a certificate fault, the node reporting no certificate in custody while holding a complete chain and key on disk, because the status line read an enumerator that was never bound to the credential store. It no longer does. The cert gate loads the credential for the configured edge domain straight from the ACME store path, and the readiness status is handed that same resolved fact rather than re-deriving one. The independent cert-expiry watch still goes through a custody enumerator seam, which is a different surface and was not the one reporting wrongly.
[ ]The ACME/dns-01 manager and public HTTPS on:443. Obtain and renew a real TLS certificate over ACME dns-01 (mercer custody), terminate server-TLS, and serve the anonymous/pubview over public HTTPS on:443, with the fabric absent from the public wire and a restore-tested backup before exposure.- Already on
main - the server-TLS termination seam with renewed-cert hot-reload; the
:443edge itself, terminating to the anonymous/pubview with the fabric structurally absent from the public wire; the challenge machinery, proven against a real CA, with five Let's Encrypt vantages observed querying the challenge name on the wire; and the order lifecycle through to a downloaded chain. - Done since
- a production order was placed and the public
:443serves it. The live node answers fordeepsky.systemsunder a Let's Encrypt chain that verifies off-host. The edge stays dark by design without a certificate, and that gate behaves correctly. - Remaining
- the live node holds that one certificate while it is authoritative for four zones, so the other three are answered under a certificate that does not name them. Item 52 carries the rest.
- ⚠ Known before the next attempt
- opening
:443requires the port to be declared to the host agent and admitted by its firewall, which today admits:53only. Declaring the port on the valis side without the matching firewall admission took the nameserver down for two minutes on 2026-08-13; a missing edge port now degrades to a dark:443instead of killing the resident.
- Already on
[X]The owner control plane over a running node. Owner authority over a running node became withdrawable, and content became placeable on a node that is already up rather than only on one being created.- Landed
- Live sessions are registered with an identity an operator can read off
a listing and type back, and the identity axis carries a door that reports the
live sessions and fences the one it is told to. A session is also fenced by
idleness, checked by the operation that wakes it, with the bound unset by default
so no deployment is locked out by an upgrade it did not ask for; and by an
authority transfer, so an instance that has surrendered write authority stops
serving an owner after an evacuation. An owner-keyed publication verb places a
page on the running node, and the client verbs list and fence a session from
outside the image. Merged to
mainvia PR #116 and PR #117. - Off-host reach
- UNPROVEN, and it is the half that is easy to read as done. The
serving-side wiring is built: an inherited owner-port descriptor is adopted as the
routable owner terminus and bound as a second keyed listener, fail-closed unless the
auth seam is engaged, and the launcher carries an
:open-owner-portknob for the firewall. But the port ships closed, no deployed node has it opened, and every exercise of the owner client so far has been against a same-host loopback. Nobody here has reproduced a cross-host attach. A single-host "verified live" proves the cryptography, not the reach through the namespace and firewall boundary. - The remaining half
- A fence does not revoke a handle already open below the session root, so a client holding one keeps it until the revocation reaches it. The remedy is a generic per-session handle revoke in the 9P library, which needs to know nothing about owners or fences. That mechanism has since landed upstream; valis is not yet wired to it, so the limit stands here for now.
Where work can be delegated in the eighth arc
The public exposure and privilege stay across the repository boundary, as in every arc:
- fulcrum (the privileged host agent): owns the
:53=/:443= exposure, the netns and firewall, and the launcher that brings the resident unit up. - mercer (the credential authority): owns the ACME lifecycle, the dns-01 digest, cert custody, and the server-TLS termination seam.
- runciter (the DNS service module): serves the transient challenge TXT for dns-01.
Ninth arc - the nameservice surface (in progress)
The arc that turns a reachable host into a serving one. Its subject is DNS and TLS in service of the actual goal, which is websites answered from the operator's own machine on the operator's own domains. Named here because the eighth arc's remaining items are being closed out under it rather than separately.
What is delivered
- valis is the primary authoritative server for the registered domains. The lame delegation is closed, a sibling host runs as an AXFR secondary, and zone data lives in PostgreSQL as operator state. Zone editing is typed and structured; the master-file text format is a boundary format for the secondary, never the edit format.
- A node can be brought up from nothing and made to serve DNS. The acceptance gate provisions a guest, installs the delivery artifact, and drives the resident up on it. Two checks in that gate could not have reported a failure and were corrected: the gate's exit status now carries its verdict, and the write-fence assertion authenticates as the account that holds the row.
- Connections may outlive a request, in the codebase. The edge no longer assumes a client-speaks-first, reply-only protocol, which is the precondition for every continuous protocol after it. Not yet exercised on the live node.
- A running node can be updated and published to without being rebuilt, and the operator reaches it over an authenticated path rather than by editing files on the host.
- A certificate renewal completes end to end in the deployed shape. Proven on a two-NIC guest in the deployed shape, against an authority that resolved the dns-01 challenge rather than being told to accept it. The defect that had stalled every local proof for months was ours: an outbound connection was not asked for as a duplex stream, so the request sat unsent. A public authority's handshake happens not to arrive in the order that exposes it, which is why certificates kept coming back from the field while every local proof hung.
- The edge holds one credential per served name, so a request for one name is no longer answered with another name's certificate.
- A deploy refuses before it moves anything the node cannot run, from one declaration that the boot path and the deploy path both consult. The refusal comes before the binary moves, so a node that cannot run the new build keeps serving the old one.
- The two-NIC production topology passes its acceptance gate. The dedicated interface moves
wholesale into the serving namespace, and
:53answers authoritatively on its routable address there. Verified on a guest destroyed and rebuilt rather than on the one that had been running. The fault that stalled this for weeks turned out not to be ours: a guest given no CPU model gets a default that lacks POPCNT, and SBCL's own bignum logcount faulted on its non-POPCNT path. ⚠ The gate is not yet the whole property, and I am recording the gate::53is asserted from outside the node and not yet from inside it, and the topology is written down nowhere underdocs/as the contract a host-install recipe would encode. - Two sockets on one address and port no longer collide silently. A UDP pair both carrying
SO_REUSEADDRare both accepted and the second takes every datagram, which is a total DNS outage that reports nothing. The DNS datagram socket is bound without that option and first, so the kernel refuses the second bind instead.
What remains
I settled the order of these on 2026-09-06, after looking at how far the tree still is from where it is going: item 53 first, then item 55, then item 52. Item 54 depends on none of them and can run alongside. The recipe that installs a two-NIC host comes last of all.
Item 53 leads because it is the only one with no evidence either way. The recipe comes last because one written today would capture a node whose ports are handed down one at a time and whose steer carries nothing, and that is the version that costs the most to undo once it is the durable contract.
[ ]Every served name answered under its own certificate, on the live host. The machinery is built and listed under What is delivered above: one credential per served name, the order lifecycle, the challenge machinery, and a renewal proven end to end in the deployed shape.deepsky.systemsis answered correctly today.- What is left
- the deployed node. It holds one certificate and is authoritative for four zones, so three of them are answered with a certificate that does not name them. The node is behind the tree. This is a production change to a host serving real domains, which is why it is its own item and not a loose end on a build.
- Known going in
- never SAN-pack the four names into one certificate. One certificate per registrable domain, because Certificate Transparency records the pairing permanently. Reaching the edge at all is item 53.
[ ]Cold verification that steered traffic is delivered. Real traffic reaching the resident accept path from a fresh image, with the four-tuple preserved and the original destination port recovered in userland. ⚠ I want this one first because nothing else would have caught what the entry below records, and it stayed invisible for weeks while every part of it read as correct.- Known going in
- on the live host no port arrives through the steer at all. The
:53pair and the:443edge are both inherited descriptors, and the http and gopher adapters bind loopback, so every public port is a privileged bind the host agent hands down. I went looking for the boot-path defect this item used to record and it is gone: the resident composes both arrival paths, and it declines to mint a socket only when the port it was told to serve is the one already arriving inherited. The live node is told:53and inherits:53, so nothing is minted and the steering map stays empty. Name a port that is not inherited and the steer carries it with no code change, which is what makes this item a verification rather than a repair. - Worth knowing
- the steer has carried public traffic before. gopher and http were served through it until the DNS work shadowed them, and the catchall object ships and sits dormant on the node. A failure here is a regression, never a first proof.
[ ]A TLS-terminated connection the event loop can drive. Termination hands back a blocking stream bound to one thread, while the loop drives raw descriptors and a TLS connection's raw descriptor carries ciphertext. Until this is answered, a continuous protocol over TLS costs a thread per idle session, which is the cost the continuous-protocol work exists to remove. Gates owner mail access.[ ]A node declares the port set it serves, not one port. The resident takes a single port today and refuses a set, so a node serves one steered port plus whatever it inherited. I want the set, because a person's node runs the protocols that person chose and one port cannot carry them.- What is left
- the open question is in the control session, which pushes one descriptor per write. Whether several are expressible in it as it stands decides whether this is a small cut or a change to the contract between valis and the host agent. That bound is not settled, and settling it comes before cutting.
- Known going in
- this is sequenced nowhere else. It appears in no arc and in no design document, only as a note beside the code that refuses it, which is why it is written down here.
[ ]The inherited:80reaches the documentation. Every node ships:80enabled and serves its own documentation there in the clear, so a new owner can read the manual for the machine off the machine itself before a certificate exists and before anything else on the node is working. The node does not do that yet: the port arrives and nothing answers on it.- Done when
- the inherited
:80descriptor reaches the http adapter the way the inherited:443descriptor reaches TLS termination today, and a client speaking plain HTTP/1.1 with no TLS is served the documentation by a node that holds no certificate. - Known going in
- the resident parses the descriptor as
--edge-http-fdand declares it ignorable;start-edgebinds the registered adapters' loopback ports and takes no descriptor at all, while:443is threaded separately intostart-tls-serveas:tcp-fd. That threading is the shape to copy. The host-agent half is a config-gated bind and firewall accept, shipping enabled, so the descriptor will be there to consume. - Note
- this is the valis half only. The design it serves is in
docs/ARCHITECTURE.org, under what
:80is for.
The constraint behind all five items
⚠ Each of the five items above is a whole-path property, and every failure found in this
arc was a feature that bound its own listener instead of arriving through the single
steered socket. The steering model is that every port arrives on one socket, the
operator's declared set decides what is answered, and the registry decides which module
answers it. A feature that binds, inherits, or accepts on its own is departing from that,
and :53 is the one place where departing is correct.
Beyond - open and on the horizon
The substrate thesis is proven; what remains is breadth and one genuine external unknown. These are tracked here as open items, not yet sequenced into an arc.
[ ]Native content-addressed transport (libp2p/IPFS). The one external dependency the ecosystem does not yet answer in Common Lisp. The store, the module-distribution puller, and cross-host coherence are all seam-shaped for it; the gate stays open. The intended path is a CFFI binding to a C-ABI libp2p, with an external-daemon HTTP bridge as an interim stopgap.[ ]At-rest key encryption. The custody keyfile is a mode-0600 dev keyfile today; real at-rest protection (passphrase / OS keyring / HSM) is owed once real steering and real external authority are outstanding.[ ]Owner-key rotation with continuity. The/id/ctlcustody model gains an additional-keys keyring first: the DID stays fixed, so device enrolment and co-signing land without disturbing the sovereignty gate; this is the near-term step that unblocks the owner-facing viewer's key-rotation runbook. True rotation of the owner master (a new DID with a signed continuity record an external verifier can prove old→new) is the sovereignty fork the substrate's gate exists to catch, and is owed its own design arc: a continuity/attestation format, a mercer verifier, the question of whether a rawdid:keyis even the right identifier for something that must survive rotation, and migration of every capability, revocation, and durable head keyed to the DID.[ ]The evaluation axis. Reaching a running node's live image to inspect, patch and repair it while it serves, which is one of the reasons the substrate is written in Common Lisp at all. The design is settled and written up in docs/REPL-AXIS.org: where the axis is placed, the shapes that were rejected and why, and one question still open (whether revocation must interrupt an evaluation already in flight, or only prevent the next one). None of the axis is in the tree: no right bit, no bridge process, and no listener wired to a namespace file. ⚠ The loopback Slynk listener the delivery image carries is not the axis and the resemblance is the trap. It is an ambient switch read from the host at boot, it carries no grant hash, revocation does not reach it, and it cannot be withheld from a delegate.[ ]Further IETF protocols over the established subtrees. SMTP, IMAP, and NNTP wire engines (and the DNS engine) ride the established substrate seams as c3po modules; valis owns the substrate and the seam, c3po owns each wire engine. Messaging is one substrate with many protocol skins.[ ]Sovereign identity provider (OAuth/OIDC). valis as its own issuer for the domains it terminates, decomposed across the constellation: valis owns the identity records (a DID/principal-keyed dao-class, distinct from the DNS ndb) and orchestrates issuance over the/capspine; mercer owns the token-signing crypto; c3po owns the OAuth/OIDC wire engine. Horizon, not yet sequenced.[ ]The upper doctrine layers: messaging beyond mail, publishing-as-social, contracts. Framed in the architecture as semantic substrates with protocols as projections; built only as far as the mail spine so far. Each is one substrate the later protocol modules project over.[ ]What a production deployment is, and what sequesters a build. Lisp dependency resolution folds into our own private dist: that is where a release pin can record what a build was made from without freezing what development builds against. What remains open is a separate question that has been wearing the same clothes: what sequesters the rest of the build environment, meaning the C toolchain, the C libraries, and the host configuration a build leans on without saying so. A cold load regrovelsiolibandcffifrom source and so needspkg-configand a C compiler present; the live host needslibfixposixand ambient capabilities; and a tracked gate driver can fail at load rather than at a test on any machine missing a developer's own source-registry file, which reads as a broken gate rather than a missing dependency. A dist answers none of those three. NixOS as a build engine, roswell with an explicit system-package manifest, and a documented host contract are the candidates; they are not mutually exclusive. One production deployment is already live while this is undecided.
The edge serves a name set, not a configured name
Direction from fade, 2026-08-16. valis must be able to hold potentially many thousands of domains in large deployments, and every domain a node holds is potentially a live edge, because any of them can point at the node's external interface for routing. ⇒ The served name set is therefore a property of what the node holds, not something an operator restates in configuration.
⛔ VALIS_EDGE_DOMAIN is the weakness, and one string answers three unrelated
questions. Read in valis src/main.lisp and src/apply/delivery.lisp: it selects which
credential the edge loads; it is a precondition for :443 coming up at all, since an unset
value leaves the public edge dark before a certificate is even looked for; and the
delivery-apply readiness confirmation reads it directly as the zone to confirm the node
against after a restart. That third question has no answer at all once the node holds a
set. It is also singular by construction: the resident sources one shared credential cell
at boot.
What is already right, and what is not
✅ Custody scales. Custody on the live host is keyed per domain on disk, so a second order has somewhere of its own to land rather than a slot to take over.
⚠ Selection does not. boomer already accepts an SNI callback, returning a certificate chain and key for the hostname a client asked for, and it can also refuse a name outright with an unrecognized-name alert, which is the answer a node needs for the names it does not hold. mercer's server-TLS wrapper does not forward it, so the capability exists upstream and does not come through the path valis uses. The seam carries a contract version; whether forwarding the callback disturbs it is open.
⛔ And the edge has no path to what the node serves. Zone state is keyed by zone identifier in operator state; nothing exposes "the names this node answers for" to the edge. Until something does, a name set cannot be derived and can only be restated.
What large numbers change, beyond correctness
A design that works for two names can fail at scale in ways that do not look like defects:
- The issuing CA rate limits per registrable domain, so standing up many names is a pacing problem rather than a loop. ⛔ Never widen a certificate to cover more names instead: a SAN-packed certificate publishes the whole name list to certificate transparency permanently, correlating every domain on it.
- Renewal becomes scheduling. A watcher that renews one credential is a different thing from one that keeps thousands current without a stampede.
- Cost must remain measurable per domain, not in aggregate, or there is no way to say how many domains one node can carry and still pay for itself.
- One cell sourced at start is fine for one name and wrong for thousands, which puts the timing of selection in question rather than settling it.
⇒ Nothing here is designed yet, and the first question is not SNI. It is where the served name set comes from.
dsmr carries the operations and delivery axis
Ruled by fade, 2026-08-16. dsmr is a constellation member, not tooling that happens to live nearby, and its role grows as the work turns toward operations and continuous delivery. It is registered in the responsibility map as a third kind of member: supply, beside the constellation rather than carved out of it.
What it owns today is the constellation's dependency closure, what every repository here is
to be built from, pinned to exact commits and published as a Quicklisp distribution. That is
the mechanism the constellation has not had for pinning what it builds against, and it is
the reason the ordinary case can track main while a release records what it was built
from.
⚠ The sequencing gate on this has expired. The custom distribution and qlot were
deferred behind the public go-live, which has happened. The deferral named its own
condition, and that condition has been met: the work is no longer held.
⛔ It closes the dependency half only. The open item above, What a production deployment is, and what sequesters a build, states plainly that a dist answers neither the C toolchain, nor the C libraries, nor the host configuration a build leans on. Nothing here changes that.
Node machinery belongs in the Lisp image
Ruled by fade, 2026-08-16. All node machinery should exist in the Lisp image unless there is a very good reason for it not to, because the nodes must be able to control their environment. A node that reaches its own state through an external command is a node whose state model is that command's, not its own.
The standing residue this names first: namespace creation, entry and teardown shell out
to ip(8). Creation, entry and deletion are syscalls – unshare(CLONE_NEWNET),
setns, netlink RTM_NEWLINK – and are reachable from the image. The precedent is
already here: whistler loads and attaches eBPF objects in Common Lisp, which is why a
deploy node needs no BPF toolchain at all.
The motivating defect, and the reason this is not a matter of taste: a resident could
fail to start after any ungraceful death because /run/netns/<name> was a plain file
rather than a namespace, and the liveness test was probe-file. That is a filesystem
question standing in for a liveness question, and it exists only because ip netns
models namespaces as bind mounts over files in a directory and that state model leaks
through. Holding the namespace descriptor in the image makes liveness a property of a
descriptor the node owns rather than of a path it inspects.
Not scoped to any current branch. The attach-proof work in flight lands first and still
shells to ip(8); this is the direction it should be brought under afterwards.
The discriminator: which side of the node boundary the act sits on
Refined by fade, 2026-08-16, and it is what keeps the ruling above from over-applying. The rule is not "shell is wrong." Standing up host OS services is a perfectly normal and correct use of shell code. What requires Lisp is anything a node must manipulate from valis.
| Preparing the ground a node runs on | shell is right | scripts/pg-dev.sh stands up a dev PostgreSQL; valis consumes a DSN and never touches the container |
| Acting through a node at runtime, under its authority | Lisp | namespace create, enter and teardown; placing publications into /pub |
The test to apply: does valis itself have to do this, while running, as part of being
a node? If a person or a build host prepares something before valis starts, shell is
fine. If the node reaches its own state or an owner's through it, that path belongs in
the image. A namespace is not host provisioning, which is why ip(8) falls on the
Lisp side: the resident creates it, enters it, and must know whether it is live.
Publishing a tree into /pub is the second instance, and its reason is different
Called out by fade, 2026-08-16, the same day as the ruling above. The first driver
that placed the manual into /pub was written as a Python build script that shells
out to the publish verb once per page. It works and it is what put the
documentation site up, so it is not being torn out to reach that aim.
The reason it is the wrong shape is not the reason ip(8) is. That one is about a
node controlling its own environment. This one is about whose authority the act runs
under: a stranger operating their own valis must be able to publish their own words
within their own instance's authority, and that makes tree publishing an
owner-facing capability of the node, not scaffolding on a build host. Filing it as
site tooling mistook our first use of a general capability for the capability itself,
in the same way :443 is an existence proof for the http server rather than the point
of it.
What follows from that, for whoever picks this up:
- The act belongs in the image, reachable by an owner who has no build host, no
python3, and no checkout. A capability a stranger cannot reach is not a capability they have. - It should open one owner session and publish through it, rather than one session per page.
- It must call
valid-slug-prather than restate it. The Python driver copied that charset into a regular expression, which puts a fail-closed traversal gate in two languages with nothing keeping them honest. - The link rewriting is a separate matter and can stay a build step: turning
ARCHITECTURE.htmlinto/architectureis post-processing an Org export, and it sits besidesplit-api.py, which already post-processes the same export.
⇒ The split to carry forward: rewriting names is a build concern, placing publications is a node concern. Bundling them is what hid the second inside the first.
The precedent was already in the tree. tools/api-doc.lisp, contributed by Glenn
Thompson on 2026-06-08, walks valis's own packages through sb-introspect and writes
docs/API.org. The documentation pipeline has carried the right division since then:
| code to docs | tools/api-doc.lisp |
Lisp, because it introspects the loaded image |
| org to HTML | site.el, split-api.py |
text transformation, and non-Lisp is fine |
Placing publications is the first kind of work, not the second: it acts on a running
node under an owner's authority. Nothing needed inventing here; the shape was already
demonstrated, and consulting tools/ would have settled it.
🄯 Brian O'Reilly <fade@deepsky.com>, 2026