Scaling out

A single sithbitd process on SQLite is the zero-config default and the right shape for one operator on one host (see Running a mail server for getting that far). To run multiple instances of the mail services (SMTP/IMAP/POP listeners, account-api, spooler workers) behind a load balancer or an orchestrator, every box below must be ticked — each one is an invariant the single-process default provides for free. The instances need not be identical, either: the same section toggles split a fleet by role — see Role-split topologies below.

The checklist

1. A shared store. [store] kind = "postgres", "aws", or "azure" (postgres is also the Google Cloud shape, against Cloud SQL). SQLite is one process per store, full stop: its write pool is a single connection over a local file, and its blob/watch defaults are process-local. kind = "turso" follows the same one-process rule: it is a local libSQL file, optionally an embedded replica that syncs to a remote Turso/libSQL primary. The replica keeps a synced-to-cloud copy for durability and local-speed reads, but reads still hit the local file (which lags the primary by up to sync_interval_secs), so it does not give the cross-instance lease/queue consistency postgres/aws/azure do — treat turso like SQLite for scaling, not as a shared store.

   kind = "cloudflare" (D1 + Queues + Workers KV + R2) is a true shared store: Cloudflare Queues give cross-instance queue coordination, keyed leases are strict single-statement SQL on D1’s leases table, and IMAP-uid allocation (uidnext) is a server-side atomic UPDATE … RETURNING — D1 executes each statement atomically on its per-database SQLite writer, so N daemons delivering into the same mailbox over one D1 database allocate distinct uids. The historical one-writer delivery caveat (uid allocation was once serialized only by an in-process mutex) was removed 2026-07-19; D1’s lack of a multi-statement transaction no longer constrains scaling, because nothing counter-critical spans statements anymore.

2. A shared blob store. [store.blobs] must point at S3 (which covers GCS via its S3-interop endpoint) or Azure Blob. The local-directory backend only works multi-instance on a shared volume, which is discouraged.

3. The same key files everywhere. credential.key (the mail-password seal), the account API’s JWT key, and any DKIM signing key must be identical on every replica — distribute them as secrets, and back them up: losing credential.key orphans every stored password.

4. IDLE polling on IMAP instances. [imap] watch_poll_seconds = N (e.g. 2–5). Delivery push is in-process; an instance that didn’t do the delivering only learns of new mail by polling. Instances that combine the spooler and IMAP still push their own deliveries instantly. An instance running IMAP with both SMTP roles off adopts 5 s automatically when the setting is left at its 0 default — see Role-split topologies.

5. PROXY protocol or source-IP preservation at the balancer. DNSBL checks and per-client connection limits key on the peer address. Behind an L4 balancer that rewrites sources, either preserve client IPs (e.g. Kubernetes externalTrafficPolicy: Local) or enable proxy_protocol = true in each listener’s server section and the balancer. Enabling it requires a proxy_trusted CIDR allowlist naming the balancer — startup refuses the switch without one, since the preamble is trivially spoofable by anything else that reaches the port. Never enable it on a listener that clients can reach directly.

6. Replica-aware limits. max_connections/max_per_peer are enforced per process; the fleet-wide effective cap is the per-instance value times the replica count. Those slots recycle themselves under a flood of connections that go quiet: limits.handshake_timeout_secs (30 s) bounds how long a peer can hold one without finishing TLS, and limits.write_timeout_secs (60 s) bounds one that stops reading altogether, so an exhausted listener recovers on its own instead of waiting for the kernel to give up on the sockets. Both are per-listener settings in the [*.server] section.

What already just works

• Job queues (chain/relay/DSN/delete) claim atomically under concurrent workers on all backends; jobs are at-least-once and every handler tolerates duplicates.

• Per-recipient SendMail ordering is serialized across instances by a store lease (send/{wallet}), so concurrent spoolers cannot double-send or double-spend stamps on one mailbox.

• POP maildrop exclusivity is a store lease with self-expiry (pop/{wallet}, a 15-minute TTL) — a crashed session on one instance cannot wedge the maildrop for the rest, because the next login steals the lapsed lease. That stealing cuts both ways, which is what renewal is for: a session renews its lease before every mutation, and a renew that comes back lost stops that deletion instead of expunging mail this instance no longer owns. Two daemons can therefore serve one wallet’s POP without either one deleting the other’s messages.

  A lapsed lease is not automatically a lost maildrop, though, and since 2026-08-07 it is not treated as one. When the renew refuses, the session gets one re-acquire, and keeps the maildrop only if two conditions both hold. First, the key was free: acquisition wins an absent or expired lease and nothing else, so the store handing one back is the proof that nobody else held one — there is no separate “who owns this?” read, and therefore no window between asking and acting. Second, the listing has not moved: the mailbox must still hold the same messages, at the same sizes, in the same order, that this session opened on, and a store that cannot answer the question counts as moved. The lease is taken before the listing is re-read, so nothing can slip in between; and if the listing gate refuses, the just-taken lease is handed straight back rather than squatting on a maildrop the session has disowned.

  What that buys is the quiet case, which on a single-instance server is the only case: a session that idled past the TTL on a maildrop nobody else touched now lands its QUIT-time deletions and signs off +OK. A genuinely stolen maildrop, a lapsed one whose contents moved underneath it, and any store that cannot answer are all still refused, exactly as before — see Known seams.

  Renewal and acquisition are one atomic conditional write each on all six backends, and the behavioral contract is covered by the shared conformance suite — but with the same provenance caveat the role-split topologies carry: they are executed in every build on SQLite, Turso and D1, while the DynamoDB, Azure Tables and Postgres implementations are exercised only when their test endpoints are configured. The roster of those gate variables lives in mail_store/README.md’s Tests section (outside this book) rather than being restated here — a test in that crate scans its sources and fails if a gate variable is missing from that section, so the crate’s list is the one that cannot fall behind. On those three backends, read the lease round trips as reviewed against the contract rather than as proven in the default build.

  The client is told, because POP3 (RFC 1939) has no untagged channel to warn on and the final reply is the whole signal: a session that lost the maildrop answers QUIT with -ERR [SYS/TEMP] some deleted messages were not removed and the server closes, instead of the usual +OK POP3 server signing off (N messages left). A client that would have dropped its local copies on +OK keeps them. A session that marked nothing for deletion still signs off +OK whatever the lease is doing — nothing was mutated, so nothing was lost.

  Re-proving is policy, not a background timer: nothing renews on a clock. The first mutation of a session always renews, and later ones renew only when the standing proof has gone stale — the coded threshold is half the TTL (7.5 minutes), chosen so a renew’s fresh full TTL leaves slack rather than landing on the expiry second. A read-only session — LIST, RETR, QUIT with nothing marked — never renews at all, and never needs to.

• The reconciler may run on every instance; duplicate re-enqueues are absorbed by the chain worker’s state guards.

• account-api replicas share nonces and credentials through the store; any replica can answer any request.

Known seams

• \Recent (IMAP) is best-effort across instances: two concurrent SELECTs of one mailbox on different instances may both see a message as recent.
• IDLE latency on a poll-fed instance is bounded by watch_poll_seconds, not instant.
• An IMAP session’s selected-mailbox snapshot is taken at SELECT: an idler woken by a cross-process delivery gets its untagged EXISTS, but FETCHing the new message takes a re-SELECT first (NOOP-driven refresh is deferred work).
• A POP session that idles past the TTL on a busy maildrop still loses its deletions. The re-acquire recovers a lapsed lease only while the mailbox stood still. A session that spends more than the 15-minute TTL without mutating anything — a client left sitting in an open POP session, reading and then deleting at the end — has nothing left to renew by the time it marks a message, and if the maildrop no longer lists what that session opened on, its QUIT-time deletions all fail with -ERR [SYS/TEMP] some deleted messages were not removed. New mail arriving is enough: delivery into INBOX does not take the maildrop lease, so on a mailbox that receives anything during the lapse the listing has moved and the session refuses, single instance or fleet. It is the safe direction to fail — the messages are intact, the client keeps its copies, and no session is ever told +OK for a deletion that did not land — but it is user-visible, and the re-acquire is not a promise that a long enough session always keeps its maildrop.
• The account API’s plaintext summary cache can be shared; its other two caches are per replica. By default each account-api process holds its own plaintext summary cache, sealed-summary cache and reading-secret stash — nothing is shared through the [store] — so every replica behind the load balancer warms its own. Since v0.110.0 [cache] kind = "redis" points every replica at one Redis server for the plaintext cache, fail-open (a down server means slower, never wrong) — see A shared backend for a fleet. The sealed-summary cache and the reading-secret stash stay per replica by design: they hold decrypted content and credentials that must die with the session. Sizing follows the split: session_summary_capacity, max_cached_sessions and max_session_secrets are always sized by the concurrent sessions one replica sees, never the fleet total, and so is summary_capacity under the default kind = "local" — under "redis" that key is unused, and redis_ttl_secs (plus an optional server-side maxmemory-policy allkeys-lru) bounds the shared cache instead. The farm starting values (summary_capacity = 16384 or 65536, max_cached_sessions = 32 or more) are on the configuration page; decide the real ones from one replica’s sithbit.api.cache.* series with Tuning the account-API caches.
• Cloudflare leases are strict (since 2026-07-19). Lease acquisition (send/{wallet} SendMail serialization, pop/{wallet} maildrop locks) is the same single-statement compare-and-swap upsert the SQLite/Turso stores run, executed on D1’s leases table — atomic server-side, no TTL floor, expiry a plain integer comparison. The original Workers KV lease (read-then-write, no CAS, ~60 s minimum TTL, eventually-consistent expiry) is retired; every backend’s leases are now strictly atomic. The daemon’s in-process write mutex remains purely a REST-contention reducer, not a correctness dependency.

Role-split topologies

The checklist reads as if every replica were identical, but nothing requires that: each sithbitd listener and the background workers are independent config toggles, so a fleet can split by role — inbound MX edges, an authenticated-submission edge, an IMAP/POP pickup tier, and headless workers — all over one store. The split is the many-instance cloud-store story, so every checklist item applies unchanged; in particular item 1: a role split is a multi-process deployment, and SQLite stays one process per store, full stop. This is a sithbitd story — the standalone smtp-server/imap-server/pop-server binaries remain dev shells, not the production split.

Five toggles produce the roles:

The defaults match the first row ([smtp], [imap], [pop], and [spooler] on; [submission] off), so every preset below writes only the lines that differ — the usual store/TLS/hostname settings from the Configuration reference come on top.

[spooler] enabled = false skips all the background workers as one unit: relay, DSN, the chain pin/send + delete pipeline, the auto-settle sweeper, the reconciler, and the repin migration. Mail is still accepted and spooled — the jobs sit in the shared queues until a worker-enabled sibling drains them. Two things deliberately stay outside the switch: the DMARC RUA/RUF reporting workers keep their own section switches, and the embedded IPFS swarm runs whenever it is configured — DHT participation is the node’s job, not a spooler worker.

# MX edge — accept inbound mail and spool it; a worker sibling drains it.
# [grpc] alone gives this edge RCPT-time postage verification (and
# at-rest sealing key reads) without an [ipfs] provider it never uses —
# the verification-only posture; the pipeline runs on the worker tier.
[grpc]
endpoint = "http://mail-grpc:50051"
[imap]
enabled = false
[pop]
enabled = false
[spooler]
enabled = false

# Submission edge — authenticated client sends only.
[smtp]
enabled = false
[submission]
enabled = true
[imap]
enabled = false
[pop]
enabled = false
[spooler]
enabled = false

# Pickup tier — IMAP + POP readers.
[smtp]
enabled = false
[spooler]
enabled = false
# [imap]
# watch_poll_seconds = 5   # auto-adopted on an IMAP-only instance; see below

# Workers — no listeners, all the background workers (the [spooler]
# default). The chain pipeline runs where the workers run, so the
# [grpc] + [ipfs] sections belong on this instance; the SMTP edges
# carry [grpc] alone, for verification only.
[smtp]
enabled = false
[imap]
enabled = false
[pop]
enabled = false

mail-grpc itself is not a fleet member: it stays a single private-network service the worker tier points at, and a many-instance fleet can safely share one gateway because the store lease serializes each wallet’s chain writes. The reasoning is recorded in the gateway topology design note.

IDLE on a split-out pickup tier

Delivery push is in-process, and nothing delivers on a pickup instance — so its IDLE wakes come only from store polling: the IMAP backend polls the mailbox change counter every watch_poll_seconds and pushes the untagged EXISTS to idlers. Stated plainly:

• New-mail latency is the poll interval, not instant. An idler on a pickup instance learns of a delivery up to watch_poll_seconds after the worker tier lands it.
• Leaving the setting at its 0 default (“trust in-process push”) would leave idlers asleep forever on an instance where nothing delivers, so sithbitd applies a safety rider: IMAP on + both SMTP roles off + watch_poll_seconds = 0 auto-adopts 5 seconds, with an info log saying so. An explicit value is always the operator’s choice, and 0 keeps meaning in-process push whenever an SMTP role is co-resident.
• The known seams above bind with full force here: \Recent is best-effort across instances, and a woken idler re-SELECTs before FETCHing the new message.

Which stores support which split

The store rules are the checklist’s, mapped onto roles. On postgres/aws/azure/cloudflare, any role may run N-wide — queues, leases, and counters are all cross-instance. sqlite and turso allow no split at all — one process per store.

Both split topologies are proven in-tree. An always-on test walks one message across three role instances — real SMTP into an MX edge, chain pin + SendMail on a workers instance, poll-fed IDLE wake then FETCH and POP RETR on a pickup instance — over one SQLite store (mail_spooler’s one_message_crosses_the_role_split_topology; each instance gets its own store handle inside one test process, since real SQLite deployments stay single-process). The same walk runs as a true multi-writer split on postgres, each role booting its own store stack from [store] kind = "postgres", gated on a configured postgres test endpoint (one_message_crosses_the_role_split_topology_on_postgres); the gate variable is named, with the rest of the store-backend roster, in mail_store/README.md’s Tests section (outside this book).

Adding a storage backend

Six backends (SQLite, Postgres, DynamoDB+SQS, Azure Tables/Queues, Turso/libSQL, and Cloudflare D1/Queues/KV/R2) share one behavioral contract, and the plumbing is deliberately small. A new backend touches exactly six places, all but a one-line forwarding entry in mail_store:

1. Cargo.toml — a cargo feature gating the backend’s SDK deps. Declare it in mail_store and keep all complete; each of the five store-consuming binaries (mail-spooler, account-api, ipfs-daemon, ipfs-gateway, mail-console) forwards it in its own [features] table (see Slim-build features);
2. config.rs — a StoreKind variant plus its [store.<kind>] settings struct (never feature-gated: configs parse in every build);
3. a backend module implementing the four repo traits (AccountRepo, MailRepo, JobQueue, KeyedLease) — blobs are orthogonal and stay behind AnyBlobStore;
4. stores.rs — a <Kind>Stores alias with an open constructor (plus its BackendDisabled stub alias), one arm in the with_backend! macro (the workspace’s single backend dispatch point; sithbitd and account-api both route through it), and the enabled/disabled __with_backend_<kind> helper pair;
5. lib.rs exports, feature-gated;
6. tests.rs — an env-gated conformance registration deriving from the canonical test list (skips must be named, with a reason), feature-gated.

The contracts to honor are written where they bind: the counter- allocation rules (atomic, monotonic, gap-tolerant) on the MailRepo trait doc with the three known implementation strategies; the job identity-vs-claim-token split on JobQueue; and the two frozen composite-key codecs in mail_store::keys (pick unit_sep if the store allows control bytes in keys, percent if not — never invent a third). The conformance suite proves all of it against a live instance before the backend ships.

A backend that creates its own cloud resources requests provider-managed encryption at rest when it does so — key configuration is optional, never required. The AWS backend enables server-side encryption on the DynamoDB tables (AWS-owned key) and SSE-SQS on the queues it creates, or a customer-managed KMS key for both when [store.aws] kms_master_key_id is set (see Running a mail server for detail); Azure Storage/Tables and Cosmos are always encrypted at rest by the platform, so no code is needed there.

IPFS: the shared-bucket cluster

The self-hosted IPFS node scales by the same principle as the stores: the bucket is the truth, the nodes are stateless. N embedded nodes (or sithbit-ipfsd daemons) point at one S3/GCS/Azure bucket ([ipfs.blobs] / ipfsd’s [blobs]) and enable [ipfs.cluster] / [cluster] — that’s the whole join procedure: membership heartbeats live in the bucket next to the blocks and pin manifests, so a node needs nothing but the bucket credentials. There is no gossip transport, no bootstrap list, no consensus.

What the cluster coordinates:

• Any-node pin/unpin. Pin manifests are last-write-wins objects in the bucket; every node sees every pin (the reprovide sweep re-reads them), and any node can serve any pinned block over bitswap or GET /ipfs/{cid} — the data has exactly one billed copy, in the bucket.
• Partitioned DHT announces. With [swarm] provide = true, live members split the reprovide keyspace by rendezvous hashing — each root is announced by exactly one member. A member that misses heartbeats for member_ttl_secs is dead; survivors notice at their next heartbeat tick and immediately resweep, taking over its share (remote DHT records carry a ~24 h TTL, so a dead node’s announces stay resolvable while the takeover lands).
• GC. The sweep deletes blocks no manifest references, but only once they are at least gc_grace_secs old — a pin writes its blocks before its manifest, so in-flight pins are never collected. The grace is a floor, not an exact age. The sweep subtracts the block’s recorded write time from the sweeping node’s own clock, and both are whole seconds, so a grace of N makes a block eligible for the next sweep after at least N and at most N + 1 real seconds where one host supplies both — and only a sweep deletes it, up to gc_interval_secs later. On the shared bucket a cluster runs over they are not one clock at all: the write time is the object store’s LastModified, so skew between a node and the store moves the boundary in either direction — a store stamping ahead of the node lengthens the margin, while a node whose clock runs fast, or a store stamping behind, shortens it and can erase it outright. Plan for the shortening direction: keep the default’s slack rather than tuning the value down against a pin’s measured upload time. Sweeps are idempotent; several nodes sweeping concurrently is safe, just redundant. A configured gc_grace_secs = 0 is legal — it is what this project’s own tests use to make GC immediately observable — and means a block becomes eligible for deletion after roughly one second; nothing rejects it, but the daemon logs a startup warning so a 0 reaching a production config isn’t silent.

Failure economics: a dead node costs nothing but its share of DHT announces until a survivor’s next heartbeat tick. Try it: docker compose -f docker-compose.cluster.yml up -d boots two daemons over one minio bucket, and docker/cluster-smoke.sh pins on node 1, kills it, and fetches through node 2.