Scaling out

A single sithbitd process on SQLite is the zero-config default and the right shape for one operator on one host (see Running a mail server for getting that far). To run multiple instances of the mail services (SMTP/IMAP/POP listeners, account-api, spooler workers) behind a load balancer or an orchestrator, every box below must be ticked — each one is an invariant the single-process default provides for free.

The checklist

1. A shared store. [store] kind = "postgres", "aws", or "azure". SQLite is one process per store, full stop: its write pool is a single connection over a local file, and its blob/watch defaults are process-local.
2. A shared blob store. [store.blobs] must point at S3 or Azure Blob. The local-directory backend only works multi-instance on a shared volume, which is discouraged.
3. The same key files everywhere. credential.key (the mail-password seal), the account API’s JWT key, and any DKIM signing key must be identical on every replica — distribute them as secrets, and back them up: losing credential.key orphans every stored password.
4. IDLE polling on IMAP instances. [imap] watch_poll_seconds = N (e.g. 2–5). Delivery push is in-process; an instance that didn’t do the delivering only learns of new mail by polling. Instances that combine the spooler and IMAP still push their own deliveries instantly.
5. PROXY protocol or source-IP preservation at the balancer. DNSBL checks and per-client connection limits key on the peer address. Behind an L4 balancer that rewrites sources, either preserve client IPs (e.g. Kubernetes externalTrafficPolicy: Local) or enable proxy_protocol = true in each listener’s server section and the balancer. Never enable it on a listener that clients can reach directly — the preamble is trivially spoofable.
6. Replica-aware limits. max_connections/max_per_peer are enforced per process; the fleet-wide effective cap is the per-instance value times the replica count.

What already just works

• Job queues (chain/relay/DSN/delete) claim atomically under concurrent workers on all backends; jobs are at-least-once and every handler tolerates duplicates.
• Per-recipient SendMail ordering is serialized across instances by a store lease (send/{wallet}), so concurrent spoolers cannot double-send or double-spend stamps on one mailbox.
• POP maildrop exclusivity is a store lease with self-expiry — a crashed session on one instance cannot wedge the maildrop for the rest.
• The reconciler may run on every instance; duplicate re-enqueues are absorbed by the chain worker’s state guards.
• account-api replicas share nonces and credentials through the store; any replica can answer any request.

Known seams

• \Recent (IMAP) is best-effort across instances: two concurrent SELECTs of one mailbox on different instances may both see a message as recent.
• IDLE latency on a poll-fed instance is bounded by watch_poll_seconds, not instant.

Adding a storage backend

Four backends (SQLite, Postgres, DynamoDB+SQS, Azure Tables/Queues) share one behavioral contract, and the plumbing is deliberately small. A fifth backend touches exactly five places, all in mail_store:

1. config.rs — a StoreKind variant plus its [store.<kind>] settings struct;
2. a backend module implementing the four repo traits (AccountRepo, MailRepo, JobQueue, KeyedLease) — blobs are orthogonal and stay behind AnyBlobStore;
3. stores.rs — a <Kind>Stores alias with an open constructor and one arm in the with_backend! macro (the workspace’s single backend dispatch point; sithbitd and account-api both route through it, so no binary changes);
4. lib.rs exports;
5. tests/mod.rs — an env-gated conformance registration deriving from the canonical test list (skips must be named, with a reason).

The contracts to honor are written where they bind: the counter- allocation rules (atomic, monotonic, gap-tolerant) on the MailRepo trait doc with the three known implementation strategies; the job identity-vs-claim-token split on JobQueue; and the two frozen composite-key codecs in mail_store::keys (pick unit_sep if the store allows control bytes in keys, percent if not — never invent a third). The conformance suite proves all of it against a live instance before the backend ships.

IPFS: the shared-bucket cluster

The self-hosted IPFS node scales by the same principle as the stores: the bucket is the truth, the nodes are stateless. N embedded nodes (or sithbit-ipfsd daemons) point at one S3/Azure bucket ([ipfs.blobs] / ipfsd’s [blobs]) and enable [ipfs.cluster] / [cluster] — that’s the whole join procedure: membership heartbeats live in the bucket next to the blocks and pin manifests, so a node needs nothing but the bucket credentials. There is no gossip transport, no bootstrap list, no consensus.

What the cluster coordinates:

• Any-node pin/unpin. Pin manifests are last-write-wins objects in the bucket; every node sees every pin (the reprovide sweep re-reads them), and any node can serve any pinned block over bitswap or GET /ipfs/{cid} — the data has exactly one billed copy, in the bucket.
• Partitioned DHT announces. With [swarm] provide = true, live members split the reprovide keyspace by rendezvous hashing — each root is announced by exactly one member. A member that misses heartbeats for member_ttl_secs is dead; survivors notice at their next heartbeat tick and immediately resweep, taking over its share (remote DHT records carry a ~24 h TTL, so a dead node’s announces stay resolvable while the takeover lands).
• GC. The sweep deletes blocks no manifest references, but only once they are gc_grace_secs old — a pin writes its blocks before its manifest, so in-flight pins are never collected. Sweeps are idempotent; several nodes sweeping concurrently is safe, just redundant.

Failure economics: a dead node costs nothing but its share of DHT announces until a survivor’s next heartbeat tick. Try it: docker compose -f docker-compose.cluster.yml up -d boots two daemons over one minio bucket, and docker/cluster-smoke.sh pins on node 1, kills it, and fetches through node 2.