sithbitd: the mail daemon

Default port(s): SMTP 2525, IMAP 1430, POP 1100, health 8190. The submission listener is disabled by default and has no distinct default port (it would inherit SMTP’s 2525) — always set its bind_addr when enabling it. The docker-compose files rebind everything to the 2xxx convention (2525/2587/2143/2110) explicitly.

The combined mail daemon — SMTP MX and submission, IMAP, POP, and the spooler workers (chain pin/send, relay, DSN generation, reconciliation) all in one process. This is the production mail-handling binary; every deployment needs it.

When you need it: always — this is the core of a SithBit deployment. Run exactly one sithbitd per SQLite store (IMAP IDLE push and per-wallet SendMail ordering are in-process); a cloud store (aws/azure) lifts that limit, letting you run one per fleet member — see Scaling out.

Every role is a toggle: each listener section has its own enabled switch, and [spooler] enabled = false skips all the background workers — relay, DSN, chain pin/send + delete, auto-settle, reconciler, repin — for a listeners-only instance. Accepted mail is still spooled; a worker-enabled sibling over the same shared store drains the queues. The DMARC RUA/RUF reporting workers keep their own section switches, and the embedded IPFS swarm is unaffected. Presets and the role matrix are in Role-split topologies.

Quickstart:

cargo run -p mail-spooler --bin sithbitd

Config file sithbitd.toml, or point SITHBITD_CONFIG at an alternate path.

Note: an empty or missing config runs a loopback dev stack with the chain pipeline disabled — delivered mail stays in state received. This is the expected zero-config shape, not a bug.

Running as an OS service

sithbitd service installs the daemon under the host’s service manager — and every SithBit server binary supports the same subcommand: sithbitd, mail-grpc, domain-sithbit, account-api, sithbit-ipfsd, sithbit-gateway, and the three standalone dev/pilot protocol servers (pop-server, smtp-server, imap-server) each register under their binary name and record their own config env var (SITHBITD_CONFIG, MAIL_GRPC_CONFIG, POP_SERVER_CONFIG, …). Everything below reads the same for the other eight; only the names change. Both platforms record the current directory as the service’s working directory (config, .env files, and relative store paths resolve there) and, with --config, an absolute config path:

cd /srv/sithbit          # becomes the service's working directory
sithbitd service install --config sithbitd.toml

systemd (Linux). service install writes /etc/systemd/system/sithbitd.service (--unit-path <path> overrides the destination; --print renders the unit to stdout instead) and prints the activation step — it never touches systemd state itself:

systemctl daemon-reload && systemctl enable --now sithbitd

The generated unit restarts on failure and orders after network-online.target. Commented User= and AmbientCapabilities=CAP_NET_BIND_SERVICE lines are included for running unprivileged while still binding privileged ports — the standard low mail ports for sithbitd, the low web ports (80/443) for the binaries that plausibly face the internet (domain-sithbit, account-api, sithbit-gateway), and each standalone protocol server’s own ports only (110/995 for pop-server, 25/465/587 for smtp-server, 143/993 for imap-server); the fleet-internal mail-grpc and sithbit-ipfsd units carry only the commented User= line. sithbitd service uninstall removes the unit file (disable the service first).

Windows. service install, from an elevated prompt, registers an auto-start service named sithbitd with the service control manager; start it with Start-Service sithbitd. The registration launches this same binary with the internal service run verb, which re-anchors the recorded working directory before loading config (SCM services otherwise start in System32). sithbitd service uninstall stops the service and deletes the registration.

At-rest sealing (automatic)

On a chain-enabled deployment (a [grpc] gateway configured), delivered mail for password-less accounts is sealed at rest automatically — there is no switch. The per-account rule is the only gate: an account with a stored mail password keeps a readable copy (its CRAM-MD5/APOP logins could never unwrap one); a wallet-signature-only account gets its body envelope-sealed at spool time, with the recipient-facing IPFS copy and reply-linkage ids computed in the same pass (the stored body can never be re-parsed, and re-sealing later would change the pinned CID — the spool-time facts are canonical, so a reading key rotated between delivery and pinning takes effect from the next message). The chain-less dev stack has no gateway to resolve reading keys and stays all-plaintext.

There is still no switch over sealing, but there is one over what creates the exception. enable_stored_passwords = false — set on account-api (which refuses to store new passwords) and on the SMTP and POP listeners (which stop advertising CRAM-MD5) — makes the deployment wallet-signature-only, and therefore sealed at rest for every account. It does not invalidate passwords already stored: those accounts keep logging in, and keep getting readable copies, until each one is cleared with DELETE /v1/account/password. See the configuration reference.

Two operational consequences:

• Accepting mail for a password-less recipient asks the gateway for their published key at SMTP DATA time. A gateway outage tempfails the submission (451) — the same posture as the postage checks at RCPT — rather than silently downgrading anyone to plaintext.
• The wallet-signature AUTH on SMTP refuses a reading-secret suffix (base58(sig).base58(secret) is an IMAP/POP/webmail login shape): the submission path never decrypts, so a client shipping the secret there is leaking it, and the misconfiguration fails loudly instead.

Large-attachment IPFS offload (opt-in)

An attachment does not have to ride inside the message. With [spooler.offload] armed, sithbitd pulls each over-threshold part out at spool time, seals it under its own freshly generated key, pins the ciphertext through the same [ipfs] provider the chain workers use, and puts a small text/plain placeholder in its place — one carrying a gateway link whose #fragment is that key. Every copy of the message that follows — the stored one IMAP/POP serve, the sealed one pinned for the recipient’s wallet, and any relayed one — carries the link, not the file.

The feature is off by default, and the arming switches are the two size rules: with threshold_bytes and aggregate_bytes both at 0 — the default — no message is rewritten at all and delivered bytes are identical to a daemon built without the feature. Either one alone arms it. Keys and defaults live in [spooler.offload]; this section is the operational picture around them.

Before turning it on, read Offloaded attachments: the link is the credential. An offload link is a bearer credential and is deliberately weaker than the sealed-box path the message body takes. That trade is the whole decision here; the threshold is just a number.

When to turn it on

Turn it on when a handful of large attachments dominate what the deployment stores, pins, and relays: one 20 MB file inflates the stored blob, the pinned copy for every local recipient, and each relayed copy, where the offloaded form costs one pin total and a few hundred bytes per copy. Leave it off — or set the threshold high — when your users’ mail is mostly ordinary photos and PDFs, when the recipients are on clients you do not control and cannot expect to follow a link, or when the bearer-link posture above is not acceptable for the mail this server carries.

Two prerequisites are real, not advisory:

• The chain pipeline must be configured. The sink is built from the [ipfs] provider, and that provider is only constructed alongside [grpc] — a pinning-less daemon could not produce a fetchable link at all. On a chain-less dev stack (or with [grpc] but no [ipfs]) a non-zero threshold_bytes is simply inert: mail is delivered whole, with no error and no rewrite. The daemon says so at boot — a warning naming the threshold you set and stating that attachment offload is inert — because that combination is an operator’s setting doing nothing rather than a broken configuration. It is a warning, not a refusal: the boot proceeds, the sink is never armed, every over-threshold attachment delivers inline, and nothing is ever pinned. Read the startup log after arming the threshold rather than assuming the setting took.
• gateway_url must be the address recipients can reach. It is baked into the delivered message — so an internal or loopback value produces mail that only works inside your network, and correcting the setting later does not repair links already delivered. Treat the gateway’s public base URL as a long-lived commitment, the same way you treat an MX name.

Two size rules, and why the second exists

threshold_bytes asks “is this one file too big to store?” and is compared per part and only per part. That leaves a real gap: twenty 1 MB attachments under a 5 MiB threshold ride inline as a 20 MB message, each part innocent on its own. aggregate_bytes closes it by asking “is this whole message too big?”, capping the decoded attachment bytes one message may leave inline in total.

When a message is over its budget, the largest eligible parts are offloaded first, and the pass stops the moment what remains inline fits. That ordering is deliberate: it reaches the budget while turning the fewest attachments into links, so the recipient keeps as many inline files as the arithmetic allows. The two rules compose as a union — the threshold takes what it takes, and the budget tops the selection up from what is left.

Three properties are worth knowing before you set a budget:

• It counts attachments, not the message. The text and HTML bodies are never metered, because they can never be offloaded — a budget measured on something the feature cannot shrink would be unsatisfiable by construction. A body-heavy message can therefore still exceed the number you set.
• It is a target, not a guarantee. A part held inline by content_id counts toward the budget but can never be taken to satisfy it, so a message full of referenced inline images can sit over budget with nothing left to offload. The pass takes what it may and stops; refusing oversized mail is max_message_size’s job, not this one’s.
• It can arm the offload alone. Leaving threshold_bytes at 0 and setting only a budget is a valid policy — no individual file is too big, but no message may carry more than N bytes of attachment.

The threshold is a decoded size, not a wire size

Both size rules compare decoded bytes — the size a mail client shows next to the attachment — and both fire on strictly greater. The wire is bigger: base64 costs a third plus line breaks, so a 5 MiB decoded part is roughly 6.8 MiB of message on the wire.

That gap is the trap, because most of the numbers an operator has in hand are wire sizes: SMTP transaction logs, the SIZE a sending MTA advertises, max_message_size, and the stored blob. Reading those and setting threshold_bytes to match sets the bar about a third too high, and the attachments you meant to catch keep riding inline. Convert first: a 25 MiB wire ceiling admits only about 18 MiB of decoded attachment, so a threshold above that can never fire.

What is offloaded — and what never is

Candidates are attachments only, and three exclusions are deliberate:

• Body parts are never touched. The text/plain and text/html bodies are not attachments and are never candidates at any size.
• Containers are never offloaded — multipart/* and a nested message/rfc822. A container’s byte range covers its children, so offloading one would silently take an entire sub-message with it.
• A part carrying a Content-ID is held inline by default — at any size, which is how inline images survive: an HTML body referencing the part as cid:… would otherwise be left pointing at nothing. On the default a 40 MB inline image stays inline and the threshold will not save you from it. content_id is the setting that changes this; see below.

Inline parts: the content_id setting

Most mail clients stamp a Content-ID on every part they build, referenced or not. The default therefore holds back two quite different things under one rule: genuine inline images, and ordinary oversized attachments whose sending client merely labelled them. content_id separates them.

Under "all" the inline rendering is lost, and it cannot be otherwise. The offloaded file is sealed and its key rides in the link’s #fragment, which a browser never sends to a server — so pointing an <img src> at the gateway would fetch ciphertext and render a broken image in every client. The rewrite therefore replaces the whole element with a link naming the file, its size and its type, the same three facts the text placeholder carries. That is a real downgrade for the recipient, and it is the honest one: the alternative is a broken image.

A reference this cannot rewrite holds its part inline. Only a whole <img …> element has an unambiguous replacement. A part reached through a CSS url(cid:…), a background= attribute, a plain-text body, or an element whose tag cannot be identified is left exactly where it is, and that message delivers as it does today. The same applies if any cid: reference in the message resolves to no part at all — percent-encoded, or simply pointing at something absent: the scan cannot then prove any part is unreferenced, so under both "orphaned" and "all" every one of them stays inline. Refusing is always safe; guessing at a rewrite is not.

The threshold is still per part. content_id decides which parts are eligible, never how big one has to be. A message whose parts are each under the threshold is untouched however many of them there are and however large the total — that is a property of the offload as a whole, not of this setting. Bounding an oversized message is what max_message_size and max_wallet_bytes do, by refusing it rather than shrinking it.

Two more properties worth knowing: a message that does not parse is delivered untouched rather than failed, and when nothing exceeds the threshold the delivered bytes are bit-for-bit what they would have been with the feature off — only the parts actually offloaded are spliced, and every other byte, boundary, and encoding is copied through verbatim. The rewrite happens before DKIM signing, so the signature covers the message as delivered.

What the recipient sees

The placeholder is a plain text/plain part naming the file, its type and size, and the link, plus the one caveat that matters (“anyone you share the whole link with can read the attachment”) — readable in any client, including one that knows nothing about SithBit. Alongside the prose it carries machine-readable headers, for clients that would rather render the original attachment than a paragraph:

X-SithBit-Offload-Url holds the whole link, key and all — so it travels through every relay, filter, and archive the message passes, exactly like the link in the body. It is not a second secret, but it is a second place the credential is written down.

X-SithBit-Offload-Cid carries the cid without the key, and is not a third copy of the credential. It exists because the URL names one operator’s gateway_url forever, while a cid is permanent: a client that understands the offload, or a tool re-pointing old mail at a new gateway, needs the content address rather than a hostname that may be gone. This is the same reasoning behind the client-side offload envelope carrying cid + key rather than a URL.

When a pin fails

A seal or pin failure tempfails the whole submission (451); the sending MTA retries and the attachment is still intact. There is deliberately no fallback to delivering it inline, which would defeat the threshold in precisely the case the threshold exists for.

When an offloaded pin is released

Settling or expunging a message unpins that message’s copy. Expunging also releases the offloaded attachments that message was carrying — but only the ones no other copy still references, and only when the submission stayed inside this server. A storage-quota refusal releases them too, by a different route described below. Three properties are worth having straight before you plan storage around the feature.

Release is refcounted, not per-copy. Offload runs once per submission, ahead of the fan-out: one attachment is pinned once, and every copy that carries its link holds a reference — each local recipient’s copy, the sender’s Sent copy, and every further copy an IMAP COPY makes (a drag between folders in a mail client). Expunging a copy drops that copy’s reference; only the last reference standing hands the pin to the unpin worker, and it does so exactly once. So a user deleting their copy of a message no longer implies the attachment is gone: it is gone for them, and stays fetchable for every other reader whose copy still exists.

Relayed submissions are exempt, permanently. If a submission had any remote recipient, its offloaded pins are flagged at delivery and no local expunge ever releases them — not even the last local copy’s. The attachment’s URL has left this store’s reach by then, so releasing the pin would break a link on a server this one cannot see. There is therefore no automatic cleanup for relayed offload pins: their retention is the pinning provider’s policy or an operator sweep’s job, and the handle for that sweep is the offload/ object-name prefix every offloaded pin carries. That is a deliberate trade, not a gap waiting on a worker.

Two operational notes follow from how release is plumbed:

• No new queue to provision. The release job rides the existing chain_delete queue — the same worker, the same provider client, and the same “already gone is fine” posture as the message-body teardown it arrives beside. Turning offload on needs no queue provisioning changes on any store backend, cloud ones included.
• The pin is released by the coordinates the spooler recorded, not by a derived name: the object name (offload/<uuid>) and the cid the provider returned both travel with the job, because providers split on which one addresses a pin — Filebase unpins by name, Pinata by cid. An offload pin belongs to the submission’s attachment rather than to any one recipient’s mailbox, so there is no wallet and blob key to derive a name from the way there is for a message’s own pin.

A message that offloaded nothing costs one empty lookup when it is expunged and enqueues nothing at all, so a deployment that never arms the threshold cannot tell this machinery is there. See the threat-model section for what all of this means for confidentiality.

See the Configuration reference for the full key/default table.

A quota refusal releases unrefcounted, and regardless of relay. The two properties above describe the expunge route, where a pin is handed over only by the last reference standing and a relayed submission is exempt permanently. A submission refused for max_wallet_bytes never got that far: it was delivered nowhere, so no copy and no pin row exist, and there is no reference for a count to reach zero. Its pins are released directly, whether or not the message had a remote recipient. Nothing still linked can be caught by this — every offload seals its attachment under a freshly minted key and pins it under a fresh name, so two submissions carrying the same file never share a pin.

Without it a full mailbox would leak an attachment on every delivery attempt: a quota refusal is deterministic, so the sending server retries the same message for its whole retry window and each attempt would pin the same payload again.

An over-quota recipient refuses the whole envelope

The storage cap answers per transaction, not per recipient. A message addressed to five local wallets, one of them full, is refused to all five with 452 4.2.2 Mailbox full, and the sending server re-sends to all five on its next attempt. The spool reports one outcome per submission and has no per-recipient verdict to give, so a cap tight enough to trip routinely will hold up mail for recipients who had room. Size it for the account you mean to bound, not for the median message.

The refusal is transient on every path, so nothing bounces while the recipient prunes. The cap is also advisory rather than exact: usage is read and the write happens separately, so two deliveries to one wallet in flight together both see the pre-write total and both land. Overshoot is bounded by how many writes are concurrent times their size, not by the number you set — size a hard ceiling with that headroom in mind.