Backups and restore

Two things hold data: Postgres and the object store. Back up both, plus the few secrets that decrypt stored credentials. Everything else can be recreated.

What holds state

Postgres

For a managed database, use your provider's backups and point-in-time recovery. For the bundled StatefulSet, take logical dumps:

kubectl exec -n critlab critlab-postgres-0 -- \
  pg_dump -U critlab -d critlab -Fc > critlab-$(date +%F).dump

For a Compose evaluation install:

docker compose exec -T postgres pg_dump -U critlab -d critlab -Fc > critlab-$(date +%F).dump

The bundled chart does not archive WAL, so dumps are your recovery points; run them on a schedule.

Continuous archiving with Compose (optional)

The production Compose file, infra/compose/production.yml, can archive WAL and take scheduled backups of its bundled Postgres to your own S3-compatible bucket with pgBackRest. It is the setup critlab's cloud runs, offered as an opt-in profile. The Helm chart does not include it, so the paragraph above still holds for Kubernetes.

• The image. Postgres is built from infra/docker/postgres.Dockerfile: the official Postgres 16 image plus pgBackRest 2.60.0.
• Turn it on. In infra/compose/.env set COMPOSE_PROFILES=backup and the repository: BACKUP_S3_ENDPOINT (a host, no scheme), BACKUP_S3_BUCKET, BACKUP_S3_REGION and BACKUP_REPO_PATH. The bucket must be reachable over TLS: pgBackRest does not speak plain HTTP to an S3 endpoint. The key pair goes in /etc/critlab/r2-backups.env (R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY; any S3 key pair with read and write on that bucket) and the repository passphrase in /etc/critlab/pgbackrest-cipher.env (PGBACKREST_REPO1_CIPHER_PASS). Neither file is in the repository, and Compose reads them as the user running it, so run it with sudo if you made them root-only.
• Where the secrets go. Only to the three backup services of the profile (wal-push, backup-init, backup). Postgres itself never holds them: its archive_command copies each WAL file into a spool volume and wal-push pushes the spool to the bucket, so neither SQL run as a superuser nor code running as the postgres user can read the bucket's key or the passphrase. Postgres refuses to start if either is put into its environment.
• Turn it off. Set BACKUP_ARCHIVING=off and leave backup out of COMPOSE_PROFILES; Postgres then runs exactly as the official image does. With archiving on (the default in this file) the profile is required: Postgres reports unhealthy, and the app does not start, until wal-push runs, and wal-push refuses to start until the target and both secrets are set, so a missing file is never a silent gap.
• What it does. Every WAL segment goes to the bucket (at least once a minute while the database has writes), a backup runs daily at 03:00 UTC (a full when the newest full is a week old, a differential otherwise), and old backups expire with the WAL they need: 14 days of point-in-time recovery, in infra/pgbackrest/pgbackrest.conf (edit it and rebuild the image to change it; the 30-day deletion promise is critlab's cloud's, your retention is yours). The repository is encrypted with the passphrase.
• When the bucket is unreachable, WAL that waits in the spool beyond 8 GiB (BACKUP_ARCHIVE_PUSH_QUEUE_MAX) is dropped rather than filling the disk and stopping Postgres; point-in-time recovery then resumes from the next backup, and the backup service takes that backup by itself as soon as the bucket is reachable again. Watch wal-push's health (unhealthy when the oldest waiting file is 10 minutes old) and read the spool's record of drops (docker compose exec wal-push cat /var/spool/critlab-wal/dropped) or the Postgres log for dropped WAL file, or turn on the alerts below. pg_stat_archiver only tells you that WAL reached the spool.
• Alerts (optional). The backup service can watch the backups and tell you when one fails. Every 5 minutes it checks that wal-push is draining the spool, that no WAL was dropped after the newest backup started, that the repository's newest WAL segment has caught up with the last one Postgres archived, that the newest full backup is under 8 days old and the newest backup of any kind under 26 hours, that the last expire ran within 26 hours, and that the disk has room for a full spool. When a check starts failing it sends one mail through Resend (set BACKUP_ALERT_TO and BACKUP_ALERT_FROM in infra/compose/.env, and put BACKUP_ALERT_RESEND_API_KEY in a root-owned file, /etc/critlab/backup-alerts.env, that only the backup service reads); it sends another only when a new check joins the failure, every 6 hours while one still fails, and a "recovered" mail when all pass again. If you set BACKUP_SENTRY_DSN it also checks in to three Sentry Cron Monitors (the check, the backup and the expire), so a stopped host is noticed too. A mail holds conditions and numbers, never data. Both channels are off until you set them, and Resend is the only mail path. The conditions, thresholds and the setup are in docs/ops/backups.md in the repository ("E3 as built").
• Keep the passphrase off the server. A repository without it restores nothing.
• Restore. critlab-restore-drill, inside the same image, restores into an empty data directory, replays WAL to a point you choose, starts Postgres with archiving off and checks the result. Finish every restore that way, in a one-off container that holds the secrets, and only then start the postgres service on the promoted data directory: the service refuses a data directory that is still in recovery, because its restore_command would need the secrets. The commands, the rollout steps and what was measured are in docs/ops/backups.md in the repository ("E2 as built").
• The app's database role. The same Compose file connects web and sandbox as critlab_app, a role that is not a superuser and has only SELECT, INSERT, UPDATE and DELETE on critlab's tables; the one-off db-roles service creates it on every up from POSTGRES_APP_PASSWORD, and only migrate uses the owner's POSTGRES_PASSWORD. Set both in infra/compose/.env (openssl rand -base64 48 | tr -d '/+=').

Object storage

Objects are write-once and named by content, and critlab relies on no bucket versioning, so a plain sync is a complete backup. rclone works against every S3-compatible store:

rclone sync critlab-s3:critlab backup-s3:critlab-backup

Order and consistency

Postgres rows point at objects; objects never point at rows. So:

1. Back up Postgres first, then the bucket. A bucket copy taken after the dump contains everything the dump references, and possibly a few newer files, which is harmless.
2. Keep bucket backups at least as long as database backups. Files no version references any more are deleted by a garbage-collection sweep after a grace period. If you restore a week-old database, the bucket must still have the files that database points at.

Restore

1. Stop writes: scale web, sandbox and realtime to zero.

kubectl scale deploy -n critlab critlab-web critlab-sandbox critlab-realtime --replicas=0

2. Restore the database into an empty database of the same name:

kubectl exec -i -n critlab critlab-postgres-0 -- \
  pg_restore -U critlab -d critlab --clean --if-exists --no-owner < critlab-2026-09-28.dump

3. Restore the bucket (rclone sync backup-s3:critlab-backup critlab-s3:critlab).
4. Make sure the secrets match the ones the backup was written with.
5. If the restored database is from an older release than the images you run, run the migration job (a helm upgrade does it) before scaling back up.
6. Scale web, sandbox and realtime back up.

The licence key lives in the deployment, not the database, so restoring a backup never brings back an expired or older licence.

Deletions after a restore

A restore to an earlier point also brings back whatever was deleted since: a prototype, a comment, an uploaded context document, a workspace or an account that a person deleted on purpose. To make that undoable, every such permanent delete writes one row of ids and timestamps (no names, no content) to the deletion_ledger table, which keeps rows for 35 days. A restore rolls that table back too, so the ledger is only useful if it was copied somewhere the restore does not reach: set the DELETION_LEDGER_EXPORT_* variables and web copies new rows to a bucket of your choosing every hour (its own bucket, with a key that can read and write that bucket only; keep it apart from the bucket your database backups go to, so the app holds no key to the backups) (see Scheduled jobs). With the export set, pnpm --filter @critlab/web ops:reapply-deletions in a source checkout applies the deletions made since the restore point to the restored database; it is not part of the images. The step-by-step is docs/ops/deletion-ledger.md in the repository.