Monitoring and backups

What to back up

The store — the store volume in the compose files, or wherever [store] points — is the only state that matters, and its pieces have very different values:

Two SQLite copy rules: a live database is only complete with its -wal and -shm sidecar files, and the distroless images have no shell — so from a container, docker cp all three files out (a copy missing the WAL reads as empty or stale). For a consistent snapshot prefer stopping the service first, or run sqlite3 sithbit.db ".backup ..." from the host against a mounted volume.

Cloud stores (kind = "aws" / "azure", or "postgres" on a managed instance like Cloud SQL) move this problem to the provider: durability comes from DynamoDB/S3/Azure Storage/the managed database (with GCS behind the s3 blob kind on Google Cloud), and only the key files above still need your own backups.

Logs

Every binary — sithbitd, account-api, domain-sithbit, and mail-grpc — logs structured tracing lines to stdout — docker logs <service> in the compose stacks. The default level is info; filter with RUST_LOG using target=level directives:

RUST_LOG=info,mail_spooler=debug,sqlx=warn

The same RUST_LOG filter also shapes what the OTLP export (below) sends — it sits in front of both the console and the exporter.

Telemetry export (OTLP)

Every binary can push traces and metrics to an OpenTelemetry collector over OTLP/gRPC. Export is off by default and enabled per service by the presence of the [observability.otlp] config section (defaults shown commented in every example config):

[observability.otlp]
# endpoint = "http://127.0.0.1:4317"
# metrics_interval_seconds = 60

mail-grpc is env-configured like the rest of its settings: OTLP_ENDPOINT (absent or empty = no export) and OTLP_METRICS_INTERVAL_SECONDS.

The design is push only — no Prometheus scrape endpoint. Prometheus users run an OTel collector with a Prometheus exporter and point the services at it. For a working dev example, the docker-compose.otel.yml overlay boots a collector with the debug exporter and turns every service’s export on:

docker compose -f docker-compose.yml -f docker-compose.otel.yml up -d
docker compose logs -f otel-collector    # spans + metrics print here

Two gotchas:

• The standard OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / …_METRICS_ENDPOINT env vars override the configured endpoint inside the exporter — leave them unset for the config file to be authoritative.
• Traces ride the RUST_LOG filter: a target silenced for logging is also not exported.

The metrics (all under the sithbit. prefix, labeled as noted):

Tuning the account-API caches

The account API’s four bounded in-memory caches are sized by the [cache] section of account_api.toml — see [cache] — the summary-cache sizes for what each key bounds, the startup validations, and the farm starting values. The defaults fit a small self-contained deployment; the ten sithbit.api.cache.* / sithbit.api.session_secrets.* series above exist so a bigger deployment can decide its own values from its own traffic instead of taking ours. Read them one cache at a time: filter on cache="summary" to size summary_capacity, and on cache="session_summary" (with session_drops beside it) to size the sealed cache’s two bounds.

Hit rate is derived, not exported. hit rate = hits / (hits + misses), over a window. Both counters are cumulative since process start, so take each one’s increase over the window you are judging (a PromQL increase(), or the collector’s delta temporality) rather than dividing lifetime totals, which never forget the cold start.

Four states, one action each:

Which key “the key” is. For cache="summary" the action changes summary_capacity — under the default kind = "local". Under kind = "redis" the four-state table does not apply to cache="summary": entries, evictions and capacity all read 0 (the shared server does not report them to one client), hits and misses are still tallied per process, and the bound to move is redis_ttl_secs or the server’s maxmemory, judged from Redis’s own INFO rather than from these series (the shared backend). For cache="session_summary", capacity is the ceiling and session_drops tells the two bounds apart: session_drops climbing means the ceiling is dropping whole sessions — raise max_cached_sessions; evictions climbing while session_drops stays flat means individual sessions are hitting their own quota — raise session_summary_capacity (rare: the default already holds a full keyed-search window with headroom). For the reading-secret stash, sithbit.api.session_secrets.evictions above zero means the bound is logging users out — raise max_session_secrets.

A sizing estimate, from the measured grid. Under poor locality — uniform random access across the working set, the pessimistic case the grid on summary_cache.rs’s DEFAULT_CAPACITY was measured under — the hit rate tracks capacity over working set: hit rate ≈ capacity / working set. So a hit rate H observed at capacity C implies a working set of roughly C / H, and C / H is an upper bound on the capacity that reaches ~100%: a 45% hit rate at the default summary_capacity = 4096 says ~9 100; 15% at the same capacity says ~27 000. Real mail access skews hard toward recent messages and beats that bound, so step up (the next power of two, say), let the counters run, and re-measure — rather than jumping straight to it.

Sitting just below the working set is the worst place to be: a poor hit rate and an eviction on nearly every insert. Once the working set fits, evictions stop altogether. So raising summary_capacity past the working set makes the cache cheaper in CPU, not dearer; the memory is what you pay. Watch evictions fall to zero after a raise — if it does not, the raise was not enough.

Each eviction itself is cheap, and its cost does not grow with the size you choose. Eviction takes its victim from a stamp-ordered index, so it examines one entry whether the capacity is 4 096 or 65 536. This was not always so: eviction used to rank every resident entry — O(cap) — while holding the one lock every request to that cache contends for, which made a large capacity punishing precisely when it was evicting. Sizing this value is now a memory question and a hit-rate question, and not a lock-contention one.

Memory. Budget ~0.5–1 KB per cached summary — an estimate, not a measurement: a parsed summary holds the rendered addresses, subject, message-id, references and a 120-character snippet, plus its key. So capacity × 1 KB is a safe ceiling: summary_capacity = 16384 is ~16 MB, 65536 is ~64 MB, and the sealed cache’s ceiling max_cached_sessions × session_summary_capacity converts the same way.

Per replica, never the fleet. Every series here is per process, and each replica holds its own caches — nothing is shared through the [store]. A dashboard summing N replicas’ entries is showing N private caches, not one shared one, and a hit rate derived from summed counters is a fleet average that no single replica sees. Derive the hit rate and pick the size from one replica’s series, then set that same value on every replica; the trap is spelled out beside the farm starting values in the configuration reference.

The job queues

All background work rides durable job queues: chain (encrypt → IPFS pin → on-chain SendMail), relay (outbound SMTP), chain_delete (expunge teardown), dsn (DSN status notifications), and — only while an [ipfs.repin] migration is configured — repin. Jobs are at-least-once with a visibility timeout; failures retry with backoff, and a job that keeps failing is buried to a dead-letter queue with a reason. The two numbers worth watching are depth (backlog) and age of the oldest job (a stuck consumer).

SQLite — the jobs and dead_jobs tables:

-- depth and oldest job per queue (timestamps are unix seconds)
SELECT queue, COUNT(*), MIN(created_at) FROM jobs GROUP BY queue;
-- poison jobs, with why they died
SELECT queue, reason, died_at, payload FROM dead_jobs;

AWS — SQS queues named <queue_prefix>-chain, -relay, -chain-delete, -dsn, and -dead; watch the standard ApproximateNumberOfMessagesVisible / ApproximateAgeOfOldestMessage CloudWatch metrics.

Azure — Storage queues under the same <queue_prefix>-* names, with -dead for buried jobs; watch approximate message counts.

A buried job’s payload is self-describing JSON (a type field plus the job’s parameters). The interactive way to inspect and re-drive dead jobs is the sithbit-console TUI (its Queues tab lists depths and dead letters with confirmed requeue/discard keys — see the console tutorial); underneath it is an account-api admin call (a wallet on its admin_wallets allowlist): GET /v1/admin/dead-jobs lists buried jobs with their queue, reason, and payload, and POST /v1/admin/dead-jobs/requeue (echo a listed entry back) fixes it onto its source queue with attempts reset — POST /v1/admin/dead-jobs/discard deletes it instead. Queue depths are GET /v1/admin/queues. On the cloud backends a listing claims each returned entry for five minutes (the id is a claim token, like a worker’s receipt handle), so requeue/discard within that window; a lapsed entry simply lists again later. Without the admin API, the manual fallback still works: fix the cause and re-insert the payload (SQLite: copy the row back into jobs with attempts = 0, visible_at = now; SQS/Azure: send the body to the source queue).

Buried jobs do not pile up forever: the worker role runs an hourly prune that discards dead-letter entries buried longer ago than [spooler] dead_retention_days (default 30; 0 disables the prune entirely — see Configuration). Every backend stamps the bury time into the dead message itself, so entries age correctly across restarts; a residual entry with no readable date (e.g. one buried by an older build) counts as older than any cutoff and is pruned, not spared. Requeue or discard a poison job you care about within the retention window.

The console also carries a balances pane (press b on a wallet): its native SOL balance, its mailbox’s default stamp price and received mail count, and the prepaid stamps each other loaded wallet holds toward it. Unlike every other pane, this one reads chain state directly — the mailbox/frombox figures come gRPC-direct from mail-grpc’s GetMailbox/GetFrombox, and the SOL balance from a Solana JSON-RPC getBalance — not through the account API. It is a read-only spot check; its two endpoints (gateway_endpoint, rpc_url) are configured alongside the console’s api_url (see Configuration).

Outbound quotas and suspension

Every authenticated sender carries rolling hour/day counters of the external (relayed foreign-domain) recipients it has been accepted for — local, on-chain-stamped mail never counts — enforced against the age-ramped allowances configured in [smtp.quota] / [submission.quota] and the account API’s twin [quota]. Alongside them rides a per-account suspend flag, honored by every server whether or not quotas are enabled.

Both are administered over the account API (a wallet on its admin_wallets allowlist, like every /v1/admin route):

• GET /v1/admin/accounts/{wallet}/quota — the wallet’s rolling usage and the allowances actually in force at its current age: {last_hour, last_day, suspended, account_age_weeks, quota_enabled, hourly_allowance, daily_allowance}. 404s for a wallet with no account row (i.e. one that never logged in — counters alone don’t create an account).
• PUT /v1/admin/accounts/{wallet}/suspend with body {"suspended": true} (or false to lift it) — 204 on success, 404 for a missing account.

What a refused sender sees on the wire, per surface — the reference for debugging “why can’t this account send/collect, or change its credentials”:

What an authenticated SMTP session pins, and for how long. The mid-session row above is the visible edge of a deliberate split. An authenticated submission session resolves the sender’s identity exactly once, at AUTH — the wallet its counters and suspend flag are keyed on, which for an alias login is whichever wallet that alias pointed at in that moment — and every later decision in the session asks about that wallet. Nothing else is carried forward: both verdicts are asked afresh, suspension at AUTH and again at every MAIL, rolling usage at every external RCPT, so a suspension or an exhausted allowance that lands mid-session bites on the next message rather than at the next login. The one operator-visible surprise is the flip side of that pin: re-point an alias at a different wallet while a session is open and the open session does not follow it. Its mail keeps being charged to the wallet it authenticated as, and it is that wallet’s suspend flag — not the new wallet’s — that stops it, so suspending the new wallet leaves the session sending while suspending the old one refuses it at the next MAIL. The window closes when the session authenticates again; since a second AUTH on an already-authenticated session is refused (503, RFC 4954 §4), that means the client’s next connection in practice, so expect a re-pointed alias to move a live sender one connection late rather than immediately. Compose has no such window at all — POST /v1/mail/send reads the suspend flag and the counters on every request.

The account-disabled replies are deliberately distinguishable from a bad-credentials refusal, and deliberately safe to disclose: every one of them is issued only after the presented credentials verified, so only the account holder ever learns the account is suspended — a stranger probing passwords still sees the ordinary bad-credentials refusal.

Quota refusals surface in telemetry as sithbit.rcpt.refusals{reason="custom_452"} — a growing count means senders are hitting their allowances, which is the control working, not an outage.

The compose 429 has a series of its own. That custom_452 line covers the SMTP surface only; a compose refused by the same quota is counted by sithbit.api.compose_quota.refusals (above), which meters exactly one thing — a POST /v1/mail/send turned away by the quota math — and carries no labels, so it is alertable as it stands, with no filter to write and get right. The blanket sithbit.api.refusals counter sees that same refusal as {status=429, route="/v1/mail/send"}, and that is where a panel here goes wrong: one refused compose increments both instruments, so never add the two together — a summed “compose refusals” line reads double. Keep them apart rather than picking one, because they answer different questions. The blanket series on that route is not quota-only — any other 429 that route answers lands in it too, so a rise there is not by itself a quota event — while the dedicated series is quota-only by construction. That asymmetry is the whole reason it exists. It also reads differently when enforcement is off: with [quota] enabled = false the quota math never runs, so the dedicated series sits flat at zero — that flat line means “enforcement off”, not “nobody is over quota” — while the blanket counter keeps metering whatever else refuses on that route.

Both account-mutation refusals are metered. sithbit.api.refusals (above) samples once per refused request, so 428s and 429s ride the same OTLP export as everything else — no reverse-proxy or ingress access log is needed to count them. The account API still logs nothing per request (only internal errors reach the log), so the counter, not the log, is where these live. Two things to know before building the panel. The whole gated surface is three route values — /v1/account/password, /v1/account/auth-epoch, and /v1/account/pin-provider — because the label is the route template and carries no HTTP method: the five gated method/path pairs collapse onto three series per status, and a refused PUT /v1/account/password is indistinguishable from a refused DELETE of it. And the counter covers the API’s entire surface, so filter on those three routes unless you want unrouted 404s and malformed-JSON 400s in the same line.

Alert on the two statuses separately, because they mean opposite things. A 429 spike is the budget working — one wallet being hammered, or a client retry loop; the Retry-After it carries is the whole remedy and no operator action follows. A 428 spike means clients cannot sign: a disconnected wallet extension, a client build that never fetches a challenge, or challenges expiring before they are spent — the step-up nonce lives 300 seconds, stamped on the issuing replica’s clock and judged on the consuming one, so skew between API replicas eats into that window.

The two interleave for one wallet, and that is a shape worth recognizing rather than a second fault: the budget is charged in front of the gate, so a bare (proof-less) attempt spends a slot on its way to its 428, and a client that retries it blindly burns the wallet’s whole window and finishes on a 429. That ordering is visible in the metric, and it is the one way a dashboard misleads: once the window is spent the limiter refuses in front of the handler holding the gate, so the 429 series climbs while the 428 series goes quiet. Under sustained abuse of a gated route the two replace each other rather than rising together — sum both statuses over the three routes for an honest “sensitive-mutation refusals” signal, and keep the split for diagnosis. Isolated 428s are not worth a page — a client that tries the mutation first and steps up on the cue produces one per successful change, which is the flow working. What deserves the alert is 428s for a wallet that are never followed by a success.

Complaint handling is deliberately manual for now: on an abuse report, suspend the wallet via the admin API above and lift the flag once resolved — suspension stops SMTP, IMAP, POP, and compose in one switch. Automated ARF (abuse-report) ingestion is deliberately deferred.

Chain states

Every delivered message copy tracks its progress to the chain in messages.chain_state:

SELECT chain_state, COUNT(*) FROM messages GROUP BY chain_state;

A copy sitting in a non-terminal state (received, pinned) for more than 15 minutes is picked up by the reconciler, which sweeps every 5 minutes and re-enqueues a chain job for it — duplicates are harmless by design. A growing received count on a chain-enabled deployment therefore means the pipeline itself is unhealthy: check the chain queue depth, the dead-letter queue, and connectivity to mail-grpc and IPFS.

Liveness

Every binary serves two HTTP health endpoints on a loopback health listener (the [health] section in every binary’s TOML config; enabled = false disables). Default ports, one per binary so a dev host can run them all:

• GET /healthz — 200 while the process serves (liveness).
• GET /readyz — 200 once startup finished (listeners bound); 503 lists what’s still waiting (readiness).

Because the runtime images are distroless (no shell, no curl), every binary also takes a --health-probe flag: it loads the same config, GETs its own /readyz, and exits 0/1. The compose files use exactly that as their healthcheck: (see docker-compose.yml), and docker/smoke.sh waits for the services to report healthy. One caveat: mail-grpc runs host networking in the chain profile, so its probe port 8193 lives on the host — move it with [health] bind_addr (or MAIL_GRPC_HEALTH__BIND_ADDR) if something else holds it.

Also useful:

• SMTP/IMAP/POP answer with a protocol banner on connect — docker/smoke.sh scripts exactly this for the dev stack.
• mail-grpc’s ListAliases RPC answers UNAVAILABLE (“still backfilling”) until the alias index has completed its first sync — a finer-grained readiness signal for the index than /readyz, which only tracks the gRPC listener (the sithbit.alias_index.staleness_seconds metric covers ongoing sync health).
• mail-grpc’s /readyz also carries a fee_payer flag: it goes not-ready when the signing wallet’s balance falls below fee_payer_floor_lamports, which is also when the write RPCs start answering UNAVAILABLE. Alert on it — a gateway that cannot pay stops the whole fleet’s chain writes, and the fix (fund the wallet) is entirely operational. The flag is ready-by-definition when the floor is disabled.