Monitoring and backups

What to back up

The store — the store volume in the compose files, or wherever [store] points — is the only state that matters, and its pieces have very different values:

Two SQLite copy rules: a live database is only complete with its -wal and -shm sidecar files, and the distroless images have no shell — so from a container, docker cp all three files out (a copy missing the WAL reads as empty or stale). For a consistent snapshot prefer stopping the service first, or run sqlite3 sithbit.db ".backup ..." from the host against a mounted volume.

Cloud stores (kind = "aws" / "azure") move this problem to the provider: durability comes from DynamoDB/S3/Azure Storage, and only the key files above still need your own backups.

Logs

Every binary — sithbitd, account-api, domain-sithbit, and mail-grpc — logs structured tracing lines to stdout — docker logs <service> in the compose stacks. The default level is info; filter with RUST_LOG using target=level directives:

RUST_LOG=info,mail_spooler=debug,sqlx=warn

The same RUST_LOG filter also shapes what the OTLP export (below) sends — it sits in front of both the console and the exporter.

Telemetry export (OTLP)

Every binary can push traces and metrics to an OpenTelemetry collector over OTLP/gRPC. Export is off by default and enabled per service by the presence of the [observability.otlp] config section (defaults shown commented in every example config):

[observability.otlp]
# endpoint = "http://127.0.0.1:4317"
# metrics_interval_seconds = 60

mail-grpc is env-configured like the rest of its settings: OTLP_ENDPOINT (absent or empty = no export) and OTLP_METRICS_INTERVAL_SECONDS.

The design is push only — no Prometheus scrape endpoint. Prometheus users run an OTel collector with a Prometheus exporter and point the services at it. For a working dev example, the docker-compose.otel.yml overlay boots a collector with the debug exporter and turns every service’s export on:

docker compose -f docker-compose.yml -f docker-compose.otel.yml up -d
docker compose logs -f otel-collector    # spans + metrics print here

Two gotchas:

• The standard OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_ENDPOINT / …_METRICS_ENDPOINT env vars override the configured endpoint inside the exporter — leave them unset for the config file to be authoritative.
• Traces ride the RUST_LOG filter: a target silenced for logging is also not exported.

The metrics (all under the sithbit. prefix, labeled as noted):

The job queues

All background work rides durable queues: chain (encrypt → IPFS pin → on-chain SendMail), relay (outbound SMTP), chain_delete (expunge teardown), dsn (status notifications), and — only while an [ipfs.repin] migration is configured — repin. Jobs are at-least-once with a visibility timeout; failures retry with backoff, and a job that keeps failing is buried to a dead-letter queue with a reason. The two numbers worth watching are depth (backlog) and age of the oldest job (a stuck consumer).

SQLite — the jobs and dead_jobs tables:

-- depth and oldest job per queue (timestamps are unix seconds)
SELECT queue, COUNT(*), MIN(created_at) FROM jobs GROUP BY queue;
-- poison jobs, with why they died
SELECT queue, reason, died_at, payload FROM dead_jobs;

AWS — SQS queues named <queue_prefix>-chain, -relay, -chain-delete, -dsn, and -dead; watch the standard ApproximateNumberOfMessagesVisible / ApproximateAgeOfOldestMessage CloudWatch metrics.

Azure — Storage queues under the same <queue_prefix>-* names, with -dead for buried jobs; watch approximate message counts.

A buried job’s payload is self-describing JSON (a type field plus the job’s parameters). The interactive way to inspect and re-drive dead jobs is the sithbit-console TUI (its Queues tab lists depths and dead letters with confirmed requeue/discard keys — see Configuration); underneath it is an account-api admin call (a wallet on its admin_wallets allowlist): GET /v1/admin/dead-jobs lists buried jobs with their queue, reason, and payload, and POST /v1/admin/dead-jobs/requeue (echo a listed entry back) fixes it onto its source queue with attempts reset — POST /v1/admin/dead-jobs/discard deletes it instead. Queue depths are GET /v1/admin/queues. On the cloud backends a listing claims each returned entry for five minutes (the id is a claim token, like a worker’s receipt handle), so requeue/discard within that window; a lapsed entry simply lists again later. Without the admin API, the manual fallback still works: fix the cause and re-insert the payload (SQLite: copy the row back into jobs with attempts = 0, visible_at = now; SQS/Azure: send the body to the source queue).

Chain states

Every delivered message copy tracks its progress to the chain in messages.chain_state:

SELECT chain_state, COUNT(*) FROM messages GROUP BY chain_state;

A copy sitting in a non-terminal state (received, pinned) for more than 15 minutes is picked up by the reconciler, which sweeps every 5 minutes and re-enqueues a chain job for it — duplicates are harmless by design. A growing received count on a chain-enabled deployment therefore means the pipeline itself is unhealthy: check the chain queue depth, the dead-letter queue, and connectivity to mail-grpc and IPFS.

Liveness

Every binary serves two HTTP health endpoints on a loopback health listener ([health] in the TOML configs; HEALTH_BIND for mail-grpc, empty disables). Default ports, one per binary so a dev host can run them all:

• GET /healthz — 200 while the process serves (liveness).
• GET /readyz — 200 once startup finished (listeners bound); 503 lists what’s still waiting (readiness).

Because the runtime images are distroless (no shell, no curl), every binary also takes a --health-probe flag: it loads the same config, GETs its own /readyz, and exits 0/1. The compose files use exactly that as their healthcheck: (see docker-compose.yml), and docker/smoke.sh waits for the services to report healthy. One caveat: mail-grpc runs host networking in the chain profile, so its probe port 8193 lives on the host — move it with HEALTH_BIND if something else holds it.

Also useful:

• SMTP/IMAP/POP answer with a protocol banner on connect — docker/smoke.sh scripts exactly this for the dev stack.
• mail-grpc’s ListAliases RPC answers UNAVAILABLE (“still backfilling”) until the alias index has completed its first sync — a finer-grained readiness signal for the index than /readyz, which only tracks the gRPC listener (the sithbit.alias_index.staleness_seconds metric covers ongoing sync health).