# Health checks, monitoring and logs

> Health endpoints for each service, what to alert on, and where logs go.

Source: https://critlab.app/docs/self-hosting/monitoring

Each service answers a health endpoint, logs to standard output, and keeps nothing on local disk. This page lists what to probe, what to alert on and where to look when something breaks.

## Health endpoints

| Service | Endpoint | Healthy response |
| --- | --- | --- |
| web | `GET /api/health` on port 3000 | `200 {"ok":true}` |
| sandbox | `GET /health` on port 4001 | `200 {"ok":true}` |
| realtime | `GET /health` on port 4010 | `200 {"ok":true,"connections":<open sockets>}` |

The Helm chart wires these as liveness and readiness probes, and the web image has a matching Docker `HEALTHCHECK`. They report that the process is up and serving; they do not check Postgres, Redis or storage. For an end-to-end check, open a share link: that exercises web, Postgres, the sandbox, storage and your wildcard certificate in one request.

`/api/health` is public on the app host by design. realtime's `/health` is public too and says nothing beyond that the process is up and how many sockets it holds. The operator view (room sizes, whether `redis` is connected, and the follow-mode stream counters) is `GET /internal/stats`: it answers only a request that carries the header `x-critlab-internal: <secret>`, where the secret is `INTERNAL_LOOKUP_SECRET`, else `DOMAIN_LOOKUP_SECRET`, else `AUTH_SECRET`, the same order web and the sandbox use. With no secret configured, or a wrong or missing header, it answers the same `404` as any unknown path. The Helm chart passes `INTERNAL_LOOKUP_SECRET` to realtime; outside the chart, set one of the three on the realtime container. Room names are prototype identifiers, so keep both paths on the internal network: the chart exposes only `/ws` on the app host.

## Error reporting

Error reporting is **off unless you set `SENTRY_DSN`**. Unset (the default), nothing initialises and no request leaves any pod. Set it to the DSN of a Sentry project, or of a self-hosted [GlitchTip](https://glitchtip.com) (same DSN format, no other change), and web, sandbox and realtime report unhandled errors there. In the Helm chart it is `config.sentryDsn` (and `config.sentryEnvironment`); with `networkPolicy.enabled`, also open egress to the DSN's host for the sandbox and realtime pods (`networkPolicy.sandbox.extraEgress`, `networkPolicy.realtime.extraEgress`).

A report holds an exception type, stack frames and the route template (`/p/[slug]`, never a URL). It never holds the error message, comment text, URLs with ids or tokens, cookies, request bodies, headers, breadcrumbs, screenshots, IP addresses or an email. Browser errors are sent to web's own `/api/diagnostics` route, which forwards only to your DSN, so no page's Content Security Policy names an outside host. There is no session replay and no performance tracing.

## Product analytics

Product analytics are **off on a self-hosted install** (`BILLING_PROVIDER` unset or `none`) and on for critlab cloud. Set `ANALYTICS=on` (Helm: `config.analytics: "on"`) to record them in your own Postgres, `ANALYTICS=off` to force them off. It is read for each request, so a restart applies it; there is no per-user opt-out.

Nothing leaves your database. When on, web records ten server-side events (a sign-up, a workspace created, an agent's first request, a version published, a thread opened or resolved, a thread bundle read through an API key, an invitation sent, a beta limit reached) as one row each in `product_events`: the event name, the time, the workspace id and either a member's opaque user id or an agent's API key id. It never records a reviewer (a link or password visitor), an IP address, a user agent, a comment, a URL or any free text, and it sets no cookie. The overlay, the staging snippet and the sandbox record nothing. Deleting an account clears the user id on that person's events, and deleting a workspace clears the workspace id; the rows stay as counts with nothing to link them to anyone. The data retention job keeps raw rows 13 months and then folds them into daily totals (`product_event_daily`: day, event, member or agent, nothing else).

After switching it on for an install that already has accounts, run the backfill once, from the repository's `apps/web` directory with `DATABASE_URL` pointing at your database: `pnpm analytics:backfill` is a dry run that prints what it would add, and `pnpm analytics:backfill --confirm` writes. It derives the sign-up, workspace, first-agent-request and first-publish events from rows that already exist, adds nothing twice, and refuses the local development database unless you pass `--allow-dev-database`.

## Scheduled jobs

web runs its own maintenance jobs. **You do not schedule anything:** every web process starts a timer for each job when it boots, the Helm chart needs no CronJob and Docker Compose needs no sidecar.

| Job | How often | What it does |
| --- | --- | --- |
| Data retention | hourly | Deletes what the [Privacy Policy](/legal/privacy) says is not kept: audit events older than 13 months (change with `RETENTION_AUDIT_MONTHS`; `0` keeps them forever), sessions 30 days after they expire, expired sign-in codes, cached link titles older than 7 days (6 hours for a failed fetch), and the read marks a visitor without an account left on a thread once nothing has touched them for 180 days (a visitor who keeps reading keeps theirs). Two housekeeping rules are not in the policy's table: a custom share domain that was claimed but never passed DNS verification is removed 30 days after its last check (its creation, if it was never checked), and a row of the deletion ledger (ids and timestamps of what was deleted, kept for [restores](/docs/self-hosting/backups)) is removed 35 days after the delete. A product analytics event older than 13 months is not deleted outright: it is counted into the daily totals first and then removed, so the totals keep their history while the rows that name a user, a key and a workspace go. |
| Notification digest | hourly | Sends the digest emails people who chose "digest" get instead of one email per event. Without it only instant notification emails go out. |
| Asset sweep | hourly | Deletes the files no version references any more. Every delete already starts one sweep; this one drains whatever that left queued. |
| Review batch poll | every minute | Collects the results of an AI review batch. It does nothing, and takes no lock, unless a batch is outstanding, which cannot happen while AI review is off. |
| Canvas backlog | every minute | Starts the next [canvas](/docs/agents/canvas) build that is owed: one a full queue could not hold, or whose process stopped mid-build (a build untouched for 6 minutes is put back in the queue, and failed after three such attempts). It does nothing, and takes no lock, unless a build is owed, and nothing at all with `CANVAS_CRAWL=off`. Builds also start by themselves: each one that ends claims the next. |
| Custom domain re-check | hourly | Re-reads the DNS that proves a [custom share domain](/docs/self-hosting/domains-tls) is still the workspace's: the TXT ownership token and the traffic record of every active domain whose last check is 6 hours old (20 per run, 4 at a time, 2 minutes at most; the next run takes what is left). It uses DNS only and never calls the TLS provider. A DNS answer that the record is gone or someone else's drops the domain to `failed`, so share links, routing and the frame-ancestors list stop trusting it, and writes one audit row in the workspace's log. DNS not answering (a timeout, `SERVFAIL`) changes nothing except the time of the attempt. It does nothing, and takes no lock, unless a domain is due. |
| Deletion ledger export | hourly | Copies the new rows of the deletion ledger to a bucket of its own (not the one your database backups go to) as JSON Lines, so that after a restore of Postgres to an earlier point the deletions made since can be applied again. It does nothing, and takes no lock, unless a `DELETION_LEDGER_EXPORT_*` variable is set; with only some of them set it logs which are missing. |
| Custom hostname release | hourly | Retries releasing the custom share hostnames of a deleted workspace at the TLS provider, when the release right after the deletion did not finish. Each hostname waits longer between tries, from an hour to a day, and after eight tries the job logs what to delete by hand at the provider. It does nothing, and takes no lock, unless a release is due. |
| Legal notice delivery | every minute | Sends critlab's own legal notices (a subprocessor change notice to workspace owners), queued when critlab's operators send one, one email per person, retried three times with a growing wait before it is marked failed. On a self-hosted install it does nothing at all: it runs only where critlab's own legal documents are shown, and takes no lock unless a notice is queued. |

How it behaves:

- **Never on boot.** The first run of each job is 3 to 8 minutes after the process starts, chosen separately per job, so a rollout does not make every replica hit Postgres, the mailer and storage at once. Each later run is the interval give or take 10%.
- **One runner at a time, across replicas.** Before a run, a process takes a Postgres advisory lock for that job (`pg_try_advisory_xact_lock`, held in a short-lived transaction on a connection of its own). If another replica holds it, the run is skipped and tried again at the next tick. Within one process a job never overlaps itself.
- **Bounded and safe to repeat.** The retention sweep deletes in small batches and skips rows another request is using; a second run finds nothing left. If a run stops on its cap with more due (`capped` in its log line), the next one comes after about five minutes rather than a full hour, so a first run over a big backlog finishes in an afternoon. To see what a first run would remove from an existing database, start web with `RETENTION_DRY_RUN=1`: the log says how many rows each rule would delete and none are deleted. `RETENTION_DISABLED=1` turns retention off while the other jobs keep running.
- **Failures are log lines.** A job that fails logs `[scheduler] job failed` with the error's class and code and never its message or a row, is reported to error reporting when `SENTRY_DSN` is set, and tries again at its next tick. A finished run logs `[scheduler] job finished` with counts only.
- **Connections.** While a job runs, a web process holds a few extra Postgres connections for seconds: one per running job for its lock, plus the job's own queries, each closed after 30 seconds idle. Count that in when you size `max_connections` for many replicas.
- **Postgres setting to know.** The lock lives as long as its transaction, and the transaction sits open while the job runs (seconds, at most a few minutes). If you set `idle_in_transaction_session_timeout` on Postgres, keep it above five minutes: a shorter one ends the transaction early, the lock goes, and a second replica may repeat the same job. Every job is safe to repeat, so the cost is a repeated sweep and never lost data.

### Turning it off, or using your own cron

Set `SCHEDULER=off` (Helm: `config.scheduler: "off"`) and web starts nothing. Do this if you would rather drive the jobs from your own cron, or to pause them. The same jobs are available as routes, which keep working whether the scheduler is on or off:

| Route | Job |
| --- | --- |
| `POST /api/internal/retention` | Data retention |
| `POST /api/internal/notifications/digest` | Notification digest |
| `POST /api/internal/assets/gc` | Asset sweep |
| `POST /api/internal/review-batches/poll` | Review batch poll |

Five jobs have no route and run only while the scheduler is on: the canvas backlog, the custom domain re-check, the deletion ledger export, the custom hostname release and the legal notice delivery. With `SCHEDULER=off`, custom domains are not re-checked, the ledger is not exported and a deleted workspace's hostnames are released only by the deletion itself.

Call each hourly (the poll every minute), from inside the deployment network, with the header `x-critlab-internal: <INTERNAL_LOOKUP_SECRET>` (the `INTERNAL_LOOKUP_SECRET` in your Secret). A wrong or missing secret gets an identical `404`; never expose these routes to the internet. A Kubernetes CronJob is enough. Adapt the Service and Secret names to your release:

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: critlab-retention
spec:
  schedule: "17 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
            - name: call
              image: curlimages/curl:8.10.1
              env:
                - name: INTERNAL_LOOKUP_SECRET
                  valueFrom:
                    secretKeyRef: { name: <your-secret>, key: INTERNAL_LOOKUP_SECRET }
              command:
                - sh
                - -c
                - >-
                  curl -fsS -X POST -H "x-critlab-internal: $INTERNAL_LOOKUP_SECRET"
                  http://<web-service>:3000/api/internal/retention
```

Leaving the scheduler on and also calling the routes is harmless, just redundant: every job is safe to repeat and to overlap.

## What to alert on

| Signal | Why |
| --- | --- |
| The `migrate` Job failing during an upgrade | The upgrade stops and the old version keeps running until you fix it. |
| 5xx rate on the app host | Most failures surface here first. |
| Sandbox 404s rising sharply | Usually a certificate, DNS or `SANDBOX_*` mismatch between web and sandbox. |
| Wildcard certificate expiry for the sandbox zone | When it lapses, every prototype stops loading while the app still works. |
| Postgres connections | Each web and sandbox process opens a pool of up to `DATABASE_POOL_MAX` (10) connections; replicas multiply it. |
| Disk usage of Postgres and object storage volumes | The chart's bundled defaults are 10Gi and 20Gi. |
| Mail send errors in web's log | Nobody can sign in with an email code if mail is down. |
| `[scheduler] job failed` in web's log | A maintenance job, retention above all, did not run. The Privacy Policy's retention periods are only kept while it does. |
| Licence expiry | A banner shows in the grace period; the platform admin sees the state in Settings → License. `GET /api/license` needs that admin's browser session, so it is not an automation endpoint. |

## Logs

All three services write plain-text logs to stdout and stderr; collect them with your cluster's log pipeline (`kubectl logs deploy/critlab-web`, `deploy/critlab-sandbox`, `deploy/critlab-realtime`). Lines are prefixed by origin, for example `[@critlab/web]`, `[@critlab/sandbox]` or `[critlab:mail]`.

Worth knowing:

- With the console mail transport, sign-in codes and invitation links are printed to web's log. Restrict log access, or configure SMTP.
- If writing an audit event fails, web logs the whole event with `console.error` rather than failing the user's action. Alert on those lines if the audit log matters to you.
- The chart runs the containers with a read-only root filesystem. The one file the app may write is the licence at `LICENSE_KEY_FILE`, and only when that path is a writable mount.

The audit log itself lives in Postgres, not in log files; see [Audit log](/docs/enterprise/audit-log).

> **Open questions**
>
> - **Metrics.** There is no Prometheus (or other) metrics endpoint yet. Until there is, monitor through the ingress, the health endpoints and the database.
> - **Structured logs.** Logs are plain text, not JSON. A log format and level switch are not decided.
> - **Dependency-aware readiness.** Whether readiness should fail when Postgres or storage is unreachable (so a broken pod leaves the load balancer) is not decided.