Deployment runbook

The rules (who deploys what, when) are in environments and releases; the targets (sizing, backups, monitoring, restore) in deployment and operations. This page is the how: what lives in the repository (OPS-02) and the steps done by hand in GitHub and Dokploy. hoco.dz and the host names are placeholders until the domain is confirmed (D-12).

What runs whereCopy Anchor Link

Images and tagsCopy Anchor Link

All under ghcr.io/anisbetehi/.

Dokploy apps follow the moving tag of their environment (hoco-server:staging, hoco-dashboard:production…); a deploy webhook makes Dokploy pull it again. Rollback pins the previous immutable tag instead (rollback).

GitHub setupCopy Anchor Link

EnvironmentsCopy Anchor Link

Settings → Environments:

Every webhook secret is optional: the deploy job skips a missing one with a notice, so the images can be built before Dokploy exists. The deploy job waits for approval on preprod and production, then deploys api, waits until GET /v1/system/version reports the new commit and /ready answers (10 minutes at most), then the worker, back-office, reseller web and backup runner.

Repository variablesCopy Anchor Link

Settings → Secrets and variables → Actions → Variables (not secrets: they end up in the browser bundle):

They are repository variables, not environment variables, so the images build before the approval and the approval gates only the deploy. The API URL is the origin, without /v1.

Repository secretCopy Anchor Link

The registry needs no secret: the workflows push with GITHUB_TOKEN (packages: write).

Branch protectionCopy Anchor Link

On dev, master and release/* (repository rules), require these checks:

The Image build check (…) jobs run only when docker/** or the lockfile changes, so they are not required (a required check that does not run blocks the PR). Tags v* are restricted to the repository owner (Settings → Rules → Rulesets → Tag rules).

DokployCopy Anchor Link

Two VPS (sizing): VPS 2 runs the projects hoco-staging and hoco-preprod, VPS 1 the project hoco-production (and hoco-docs, restricted). On each: Ubuntu 24.04, Dokploy installed, SSH by key only, firewall open on 22, 80 and 443 only (and the Dokploy panel behind its own domain with HTTPS).

Registry. Settings → Registry: add ghcr.io with Anis's GitHub username and a fine-grained or classic token with read:packages only. Every app below uses it.

Services per environmentCopy Anchor Link

Apps per environmentCopy Anchor Link

Every app: provider Docker, registry ghcr.io, the image and tag below, container port 3000 unless noted, environment from the env group. <env> is staging, preprod or production; the production apps drop the suffix (hoco-api, hoco-worker, hoco-dashboard, hoco-web).

For each app, Deployments → Webhook URL is the value of its DOKPLOY_*_WEBHOOK secret in the matching GitHub environment.

Health checks and zero-downtime updates (Advanced → Swarm settings), so a failed migration or a crash keeps the previous version serving:

// hoco-api-<env> — Health Check
{ "Test": ["CMD", "node", "-e", "fetch('http://localhost:3000/ready').then(r=>process.exit(r.ok?0:1),()=>process.exit(1))"],
  "Interval": 10000000000, "Timeout": 5000000000, "StartPeriod": 120000000000, "Retries": 3 }
// hoco-worker-<env> — the same with http://localhost:3001/health
// every app — Update Config (Rollback Config: the same without FailureAction)
{ "Parallelism": 1, "Delay": 10000000000, "FailureAction": "rollback", "Order": "start-first", "Monitor": 30000000000 }

The Next apps and the docs can use / as their health URL on port 3000.

Env groupsCopy Anchor Link

One group per environment: the project environment of hoco-staging, hoco-preprod and hoco-production (Project → Environment), referenced from each app as NAME=${{project.NAME}}. Every value is distinct per environment; production values are generated in Dokploy and kept in the password vault (secrets). The names come from packages/env (validated at boot: a missing or malformed value stops the process).

hoco-api and hoco-worker (packages/env/src/server.ts):

hoco-dashboard and hoco-web (packages/env/src/dashboard.ts, web.ts): NEXT_PUBLIC_API_URL and STORAGE_PUBLIC_URL are build arguments, set as repository variables, not in Dokploy. At run time they need only SESSION_COOKIE_NAME, the API's value, read by the sign-in redirect (NODE_ENV=production and PORT=3000 come from the image).

Backup runnerCopy Anchor Link

hoco-backup: DATABASE_URL (the same database; a read-only role is enough), BACKUP_S3_ENDPOINT, BACKUP_S3_REGION, BACKUP_S3_ACCESS_KEY, BACKUP_S3_SECRET_KEY, BACKUP_S3_BUCKET, BACKUP_PREFIX (staging, preprod, production), BACKUP_AGE_RECIPIENTS, BACKUP_KEEP_DAILY, BACKUP_KEEP_WEEKLY, BACKUP_KEEP_MONTHLY, BACKUP_KEEP_PREDEPLOY, optional BACKUP_HEARTBEAT_URL; for the bucket copy, the S3_* variables of the environment and BACKUP_KEEP_PRIVATE_DAYS, BACKUP_KEEP_PUBLIC_DAYS.

Domains and TLSCopy Anchor Link

• DNS at the registrar: A (and AAAA) records to VPS 2 for the -staging and -preprod hosts, to VPS 1 for production and docs; CAA 0 issue "letsencrypt.org" on the apex (domains).
• In Dokploy, each domain: HTTPS on, certificate Let's Encrypt. Traefik redirects HTTP to HTTPS and renews the certificates.
• HSTS is sent by the apps themselves (helmet on the API, the security headers of the Next apps), so Traefik needs no extra middleware.

Migrations and rollbackCopy Anchor Link

• Migrations are the pre-start step of api. docker/start.sh runs prisma migrate deploy before the server listens; with start-first updates and the /ready health check, a failed migration leaves the new task unhealthy, Swarm rolls back, and the previous version keeps serving. The worker never migrates. Migrations are additive (expand then contract).
• Before a production deploy with a migration, run the backup app's Schedules → predeploy → Run now (hoco-backup predeploy), then approve the production environment.
• Rollback (10 minutes): in Dokploy, set the image of each affected app to the previous immutable tag — hoco-server:<previous sha> or :X.Y.(Z-1) for api and worker together, hoco-dashboard:<env>-<previous sha>… — and deploy. Return to the moving tag (:<env>) once the fix is released. A database rollback is a restore (restore runbook), never a down-migration.

BackupsCopy Anchor Link

The hoco-backup-<env> app runs docker/backup.sh from Dokploy Schedules (type Application, command run in the container). Dokploy's cron is UTC; Algiers is UTC+1 all year.

Retention per environment (data policy, backups):

• Target: a bucket at another provider or account than the VPS, with credentials that can write and delete in it only; one prefix per environment. Object lock if the provider has it.
• Encryption keys: age-keygen -o hoco-backup.key on an offline machine, once for Anis and once for HOCO's designated person; both public keys (age1…) go in BACKUP_AGE_RECIPIENTS, the private keys never reach a server. Restore: age -d -i hoco-backup.key -o hoco.dump <file>.dump.age, then the restore runbook.
• Not covered by the script: continuous WAL archiving and the weekly base backup for point-in-time recovery in production — set up with the PostgreSQL service (for example WAL-G to the same target) in SRV-28, together with the timed restore drill on the VPS. Dokploy's built-in database backups are not used: they are not encrypted.

Docs siteCopy Anchor Link

docs.yml builds hoco-docs on every change to apps/docs on dev and calls DOKPLOY_DOCS_WEBHOOK. In Dokploy: app hoco-docs, image hoco-docs:latest, port 3000, domain docs.hoco.dz with Let's Encrypt, and Security → Basic auth with one user per reader (restricted access). It needs no environment variable.

Mock APICopy Anchor Link

mock-api-staging serves examples generated from the committed OpenAPI document of dev (--dynamic, CORS on) — the same as pnpm mock. It redeploys with staging, so it always matches the contract on dev.

Setup checklist for AnisCopy Anchor Link

GitHub:

• Environments staging, preprod (required reviewer, tags v*-rc.*) and production (required reviewer, tags v*) (environments).
• The six repository variables *_NEXT_PUBLIC_API_URL and *_STORAGE_PUBLIC_URL (variables).
• Branch protection on dev, master, release/* with the three required checks; tag rule on v* (branch protection).
• After the first images.yml run: the GHCR packages stay private; link them to the repository (Package settings → Manage Actions access) if the push is refused.

Infrastructure:

• VPS 1 and VPS 2 provisioned and hardened; Dokploy installed on both; the panel on HTTPS.
• DNS records and CAA (domains).
• Registry ghcr.io with a read:packages token in Dokploy.
• Projects hoco-staging, hoco-preprod, hoco-production; in each, PostgreSQL 17, Redis 7 and MinIO with their buckets, versioning and CORS (services).
• Project environment (env group) of each project filled from the tables; secrets generated per environment and stored in the vault.
• Apps created with images, domains, Let's Encrypt, health checks and update config (apps); each webhook URL copied into the matching GitHub environment secret.
• Pre-production: reference seed and the anonymised recette data, never the demo seed with APP_ENV=production; staging: reference and demo seeds (pnpm db:seed:demo run once from a one-off container of the server image).
• Backup target bucket and credentials; two age key pairs (Anis, HOCO); hoco-backup-<env> app and its three schedules; Dokploy email notifications on failed schedules; check after the first three nights that the files are there and pruned (OPS-02 acceptance).
• Uptime monitor outside VPS 1: /health and /ready of every API, / of admin, app and docs, every minute (monitoring).
• hoco-docs with basic auth; mock-api-staging.

Verification (OPS-02 tests):

• A no-op PR merged into dev shows its commit at https://api-staging.hoco.dz/v1/system/version within 10 minutes.
• A tag v0.1.0-rc.1 on release/0.1.0 waits for approval, then deploys pre-production.
• Rollback drill on staging: pin the previous hoco-server:<sha> on api and worker, check /v1/system/version, return to :staging.
• A broken migration on a throwaway branch deployed to staging leaves the previous version serving.