# Backups and restore

> What holds state, what is stateless, and how to back up and restore Postgres and object storage.

Source: https://critlab.app/docs/self-hosting/backups

Two things hold data: Postgres and the object store. Back up both, plus the few secrets that decrypt stored credentials. Everything else can be recreated.

## What holds state

| Component | Holds | Back up? |
| --- | --- | --- |
| Postgres | Workspaces, members, prototypes, versions and their file manifests, comments, transcripts, audit log, encrypted integration and bucket credentials | **Yes** |
| Object storage | Package files (by content hash), voice notes, screenshots, thumbnails | **Yes** |
| Secrets | `AUTH_SECRET`, `SANDBOX_TOKEN_SECRET`, `STORAGE_SECRET`, `INTEGRATIONS_SECRET`, SMTP and S3 credentials, the licence key | **Yes**, in your secret manager |
| Redis | Transient pub/sub traffic between web and realtime | No |
| web, sandbox, realtime | Nothing (in-memory caches only) | No |

> **Keep the encryption secrets with the database**
>
> Integration tokens are encrypted with `INTEGRATIONS_SECRET` and workspace bucket credentials with `STORAGE_SECRET`, each falling back to `AUTH_SECRET` when unset. A database restored without the secrets it was written with cannot decrypt them: every integration must be reconnected and every workspace bucket re-entered. Changing `AUTH_SECRET` also signs everyone out.

## Postgres

For a managed database, use your provider's backups and point-in-time recovery. For the bundled StatefulSet, take logical dumps:

```bash
kubectl exec -n critlab critlab-postgres-0 -- \
  pg_dump -U critlab -d critlab -Fc > critlab-$(date +%F).dump
```

For a Compose evaluation install:

```bash
docker compose exec -T postgres pg_dump -U critlab -d critlab -Fc > critlab-$(date +%F).dump
```

The bundled chart does not archive WAL, so dumps are your recovery points; run them on a schedule.

## Continuous archiving with Compose (optional)

The production Compose file, `infra/compose/production.yml`, can archive WAL and take scheduled backups of its bundled Postgres to **your own S3-compatible bucket** with [pgBackRest](https://pgbackrest.org). It is the setup critlab's cloud runs, offered as an opt-in profile. The Helm chart does not include it, so the paragraph above still holds for Kubernetes.

- **The image.** Postgres is built from `infra/docker/postgres.Dockerfile`: the official Postgres 16 image plus pgBackRest 2.60.0.
- **Turn it on.** In `infra/compose/.env` set `COMPOSE_PROFILES=backup` and the repository: `BACKUP_S3_ENDPOINT` (a host, no scheme), `BACKUP_S3_BUCKET`, `BACKUP_S3_REGION` and `BACKUP_REPO_PATH`. The bucket must be reachable over **TLS**: pgBackRest does not speak plain HTTP to an S3 endpoint. The key pair goes in `/etc/critlab/r2-backups.env` (`R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY`; any S3 key pair with read and write on that bucket) and the repository passphrase in `/etc/critlab/pgbackrest-cipher.env` (`PGBACKREST_REPO1_CIPHER_PASS`). Neither file is in the repository, and Compose reads them as the user running it, so run it with `sudo` if you made them root-only.
- **Where the secrets go.** Only to the three backup services of the profile (`wal-push`, `backup-init`, `backup`). Postgres itself never holds them: its `archive_command` copies each WAL file into a spool volume and `wal-push` pushes the spool to the bucket, so neither SQL run as a superuser nor code running as the postgres user can read the bucket's key or the passphrase. Postgres refuses to start if either is put into its environment.
- **Turn it off.** Set `BACKUP_ARCHIVING=off` and leave `backup` out of `COMPOSE_PROFILES`; Postgres then runs exactly as the official image does. With archiving **on** (the default in this file) the profile is required: Postgres reports unhealthy, and the app does not start, until `wal-push` runs, and `wal-push` refuses to start until the target and both secrets are set, so a missing file is never a silent gap.
- **What it does.** Every WAL segment goes to the bucket (at least once a minute while the database has writes), a backup runs daily at 03:00 UTC (a full when the newest full is a week old, a differential otherwise), and old backups expire with the WAL they need: 14 days of point-in-time recovery, in `infra/pgbackrest/pgbackrest.conf` (edit it and rebuild the image to change it; the 30-day deletion promise is critlab's cloud's, your retention is yours). The repository is encrypted with the passphrase.
- **When the bucket is unreachable,** WAL that waits in the spool beyond 8 GiB (`BACKUP_ARCHIVE_PUSH_QUEUE_MAX`) is dropped rather than filling the disk and stopping Postgres; point-in-time recovery then resumes from the next backup, and the `backup` service takes that backup by itself as soon as the bucket is reachable again. Watch `wal-push`'s health (unhealthy when the oldest waiting file is 10 minutes old) and read the spool's record of drops (`docker compose exec wal-push cat /var/spool/critlab-wal/dropped`) or the Postgres log for `dropped WAL file`, or turn on the alerts below. `pg_stat_archiver` only tells you that WAL reached the spool.
- **Alerts (optional).** The `backup` service can watch the backups and tell you when one fails. Every 5 minutes it checks that `wal-push` is draining the spool, that no WAL was dropped after the newest backup started, that the repository's newest WAL segment has caught up with the last one Postgres archived, that the newest full backup is under 8 days old and the newest backup of any kind under 26 hours, that the last expire ran within 26 hours, and that the disk has room for a full spool. When a check starts failing it sends one mail through [Resend](https://resend.com) (set `BACKUP_ALERT_TO` and `BACKUP_ALERT_FROM` in `infra/compose/.env`, and put `BACKUP_ALERT_RESEND_API_KEY` in a root-owned file, `/etc/critlab/backup-alerts.env`, that only the `backup` service reads); it sends another only when a new check joins the failure, every 6 hours while one still fails, and a "recovered" mail when all pass again. If you set `BACKUP_SENTRY_DSN` it also checks in to three [Sentry Cron Monitors](https://docs.sentry.io/product/crons/) (the check, the backup and the expire), so a stopped host is noticed too. A mail holds conditions and numbers, never data. Both channels are off until you set them, and Resend is the only mail path. The conditions, thresholds and the setup are in `docs/ops/backups.md` in the repository ("E3 as built").
- **Keep the passphrase off the server.** A repository without it restores nothing.
- **Restore.** `critlab-restore-drill`, inside the same image, restores into an empty data directory, replays WAL to a point you choose, starts Postgres with archiving off and checks the result. Finish every restore that way, in a one-off container that holds the secrets, and only then start the `postgres` service on the promoted data directory: the service refuses a data directory that is still in recovery, because its `restore_command` would need the secrets. The commands, the rollout steps and what was measured are in `docs/ops/backups.md` in the repository ("E2 as built").
- **The app's database role.** The same Compose file connects web and sandbox as `critlab_app`, a role that is not a superuser and has only `SELECT`, `INSERT`, `UPDATE` and `DELETE` on critlab's tables; the one-off `db-roles` service creates it on every `up` from `POSTGRES_APP_PASSWORD`, and only `migrate` uses the owner's `POSTGRES_PASSWORD`. Set both in `infra/compose/.env` (`openssl rand -base64 48 | tr -d '/+='`).

## Object storage

Objects are write-once and named by content, and critlab relies on no bucket versioning, so a plain sync is a complete backup. `rclone` works against every S3-compatible store:

```bash
rclone sync critlab-s3:critlab backup-s3:critlab-backup
```

## Order and consistency

Postgres rows point at objects; objects never point at rows. So:

1. **Back up Postgres first, then the bucket.** A bucket copy taken after the dump contains everything the dump references, and possibly a few newer files, which is harmless.
2. **Keep bucket backups at least as long as database backups.** Files no version references any more are deleted by a garbage-collection sweep after a grace period. If you restore a week-old database, the bucket must still have the files that database points at.

## Restore

1. Stop writes: scale web, sandbox and realtime to zero.

```bash
kubectl scale deploy -n critlab critlab-web critlab-sandbox critlab-realtime --replicas=0
```

2. Restore the database into an empty database of the same name:

```bash
kubectl exec -i -n critlab critlab-postgres-0 -- \
  pg_restore -U critlab -d critlab --clean --if-exists --no-owner < critlab-2026-09-28.dump
```

3. Restore the bucket (`rclone sync backup-s3:critlab-backup critlab-s3:critlab`).
4. Make sure the secrets match the ones the backup was written with.
5. If the restored database is from an older release than the images you run, run the migration job (a `helm upgrade` does it) before scaling back up.
6. Scale web, sandbox and realtime back up.

The licence key lives in the deployment, not the database, so restoring a backup never brings back an expired or older licence.

## Deletions after a restore

A restore to an earlier point also brings back whatever was deleted since: a prototype, a comment, an uploaded context document, a workspace or an account that a person deleted on purpose. To make that undoable, every such permanent delete writes one row of ids and timestamps (no names, no content) to the `deletion_ledger` table, which keeps rows for 35 days. A restore rolls that table back too, so the ledger is only useful if it was copied somewhere the restore does not reach: set the `DELETION_LEDGER_EXPORT_*` variables and web copies new rows to a bucket of your choosing every hour (its own bucket, with a key that can read and write that bucket only; keep it apart from the bucket your database backups go to, so the app holds no key to the backups) (see [Scheduled jobs](/docs/self-hosting/monitoring)). With the export set, `pnpm --filter @critlab/web ops:reapply-deletions` in a source checkout applies the deletions made since the restore point to the restored database; it is not part of the images. The step-by-step is `docs/ops/deletion-ledger.md` in the repository.

> **Open questions**
>
> - **Workspace-owned buckets.** Files a workspace stores in its own bucket are that workspace's to back up; critlab never deletes from them. Whether operators need tooling to verify such a bucket against a restored database is open.
> - **A tested restore drill for the Helm chart.** The Compose profile has a scripted restore drill (`critlab-restore-drill`, with a timed, recorded procedure in `docs/ops/backups.md`); the steps above for the chart follow from the schema and have no scripted, verified restore yet.