# Runbooks

Symptom, command and verification for the operational procedures the template supports.

Each runbook names a symptom, the commands to run and how to confirm the result. The reasons live in the linked decisions; this page lists only the steps. Run database commands against the environment you mean, and never against a database you need without a backup.

## Database restored or moved to another host [#database-restored-or-moved-to-another-host]

**Symptom.** After a `pg_restore` or a provider move, tenant requests see no rows, fail with permission errors, or `db:apply-pending` reports RLS problems. A restore with `--no-owner --no-acl` loses grants, and loses policies when the `app_tenant` role is missing.

<Steps>
<Step title="Create the role if the host cannot">
If the target connection role cannot create roles, have an administrator create `app_tenant` and grant it once:

```sql
CREATE ROLE app_tenant NOLOGIN NOINHERIT NOBYPASSRLS;
GRANT app_tenant TO <connection role> WITH INHERIT FALSE, SET TRUE;
```
</Step>
<Step title="Restore and apply">
Restore the dump, then run the pending migrations against the target. The command ends with the RLS verification and repairs its idempotent part:

```sh
pnpm --filter @template-cloudflare/db db:apply-pending
```
</Step>
<Step title="Verify and repair">
Check, and repair what is left if needed:

```sh
pnpm --filter @template-cloudflare/db db:verify-rls
pnpm --filter @template-cloudflare/db db:verify-rls --repair
```
</Step>
</Steps>

**Verify.** `db:verify-rls` exits 0, and `select pg_has_role(current_user, 'app_tenant', 'SET')` is true on the runtime connection. If the runtime role differs from the migration role, add it to `app.session_roles`; `db:verify-rls` says so. Reason: [ADR-0009](/docs/adr/0009-row-level-security).

## Tenant statements fail after a database or driver change [#tenant-statements-fail-after-a-database-or-driver-change]

**Symptom.** Tenant queries error or time out after a PostgreSQL, Hyperdrive or `pg` change, while system paths (Better Auth, cron) work.

<Steps>
<Step title="Set the kill switch">
Turn off statement pipelining without a code change. Set the Worker variable or secret `DB_TENANT_PIPELINE=0`. Tenant statements then use explicit `BEGIN`, settings, statement and `COMMIT`.
</Step>
<Step title="Retry">
Redeploy or wait for the variable to apply, and retry the failing request.
</Step>
<Step title="Prove both modes locally">
Reproduce locally with both modes before turning pipelining back on:

```sh
pnpm --filter @template-cloudflare/db test:rls-matrix
```
</Step>
</Steps>

**Verify.** The failing request succeeds with the switch set. `test:rls-matrix` passes in both modes before you remove the variable.

## Hyperdrive query caching is on [#hyperdrive-query-caching-is-on]

**Symptom.** The production deploy stops at the Hyperdrive check, or a Hyperdrive was created without `pnpm setup`. With caching on, reads are keyed by query text and parameters and can cross the tenant boundary.

```sh
node packages/project-setup-cli/src/hyperdrive.mjs check
pnpm setup --hyperdrive-id <id>
pnpm doctor --connect
```

**Verify.** The check exits 0 and `pnpm doctor --connect` reports caching disabled. Reason: [ADR-0009](/docs/adr/0009-row-level-security).

## Prove tenant isolation on a deploy [#prove-tenant-isolation-on-a-deploy]

**When.** Before a production deploy that touches tenancy, the database driver or Hyperdrive.

```sh
node scripts/rls-preview-proof.mjs <previewUrl>
```

Run it against the pull request's preview: the `tenancy.probe` procedure exists only where `TENANCY_PROBE=1`, which is set for previews and refused in the production configuration. When sign-up needs email verification, pass two verified accounts with `RLS_PROOF_USER_A=email:password` and `RLS_PROOF_USER_B=email:password`.

**Verify.** Exit 0 means no leak. Exit 1 is a leak or a broken guarantee, so do not deploy. Exit 2 means setup failed (for example, verification required).

## Take production offline [#take-production-offline]

**Symptom.** Production must stop serving traffic now.

Run the **Emergency shutdown (wrangler worker)** workflow with `action=shutdown` and `confirm=yes`. Run it again with `action=restore` to bring it back.

<Callout title="The custom domain stays online">
The workflow toggles the Worker's `workers.dev` subdomain only. Production is served from a custom domain (`routes` in `apps/web/wrangler.jsonc`), which the workflow does not block. To stop traffic on the custom domain, also disable the route or add a WAF rule that blocks it in the Cloudflare dashboard, and remove it afterwards.
</Callout>

**Verify.** Request the custom domain and the `workers.dev` URL. Both must refuse traffic during the shutdown, and both must serve again after the restore.

## Roll back a Worker deploy [#roll-back-a-worker-deploy]

Follow [Deployment and rollback](/docs/deploy#migrations-and-rollback). A Worker rollback does not reverse migrations, restore deleted records or undo external deliveries.

## After an incident [#after-an-incident]

Write a short note in the issue or pull request that closed it, with three headings:

- **What is back**: the service, the data and the time it returned.
- **What is lost**: rows, deliveries or files that could not be recovered, with evidence.
- **Verdict**: recovered, partly recovered or not recovered. Add follow-up issues.

## Key files [#key-files]

- [`.github/workflows/emergency-shutdown.yml`](https://github.com/CarlosZiegler/template-cloudflare/blob/main/.github/workflows/emergency-shutdown.yml)
- [`packages/db/scripts/verify-rls.mjs`](https://github.com/CarlosZiegler/template-cloudflare/blob/main/packages/db/scripts/verify-rls.mjs)
- [`scripts/rls-preview-proof.mjs`](https://github.com/CarlosZiegler/template-cloudflare/blob/main/scripts/rls-preview-proof.mjs)
- [`packages/project-setup-cli/src/hyperdrive.mjs`](https://github.com/CarlosZiegler/template-cloudflare/blob/main/packages/project-setup-cli/src/hyperdrive.mjs)