nats — NATS Tower + single-node NATS JetStream server + S3 (*.barbara8.de)

Service	URL	Stack (Dockhand env 3 = UM890)
NATS Tower (UI/API)	https://nats-tower.barbara8.de	nats-tower (host port 18440)
NATS server (websocket, public)	wss://nats.barbara8.de	nats (host port 18443 → container 4443)
NATS server (plain, Tailscale-only)	nats://100.96.0.81:18442	nats (host port 18442 → container 4222)
S3 endpoint (nats-s3)	https://s3.barbara8.de	nats (host port 18444 → container 5222)

• NATS Tower (https://nats-tower.com) manages the operator: it holds the operator keypair and signs accounts/users. There is one installation, created by hand and pointed at the public websocket URL (wss://nats.barbara8.de). No BOOTSTRAP_URL is set — Tower would match a bootstrap URL against the operator record and silently re-create a throwaway IP-based operator on every start whenever it isn't found.
• NATS server is a single node with JetStream, initialized with the operator JWT generated by Tower (see docker/nats/config/operator.conf, operator.jwt — the operator JWT is public by design; the operator/SYS seeds never leave Tower).
• Accounts use a full (disk-backed) account resolver: Tower publishes accounts at runtime (system user → $SYS.REQ.CLAIMS.UPDATE) and the server stores every account JWT as a file in /etc/nats/jwt — one file per account, named <account-pubkey>.jwt. That directory is the natsjwt named volume (seeded from the image with the SYS account JWT), so accounts survive server restarts — no re-publishing in Tower needed.
• Current identity: account NATS, user nick (creds below).
• S3 endpoint: nats-s3 — an S3 API-compatible gateway that stores buckets/objects in NATS JetStream Object Stores (one stream per bucket, OBJ_<bucket>), running as a service in the nats stack at https://s3.barbara8.de. Any S3 client (aws cli, SDKs) works with the access key/secret below. S3 buckets are NATS object streams — visible/manageable via the nats CLI and Tower (nats stream ls), and there is no separate S3 data volume: the jsdata volume is the only state. The gateway exposes /healthz (200 when connected to NATS, 503 otherwise).
  • The gateway talks to NATS as dedicated user nats-s3 (account NATS); its creds file lives in the read-only s3creds named volume (seeded out-of-band, gitignored secrets/nats-s3.creds).

Credentials

Tower login (admin): nick.simon.kratzke@gmail.com (password kept as the TOWER_ADMIN_PASSWORD Dockhand stack secret — not in git).

NATS client creds for user nick (account NATS): kept in secrets/nats.creds (gitignored). Re-issue any time from the Tower UI: Accounts → NATS → Users → nick → View credentials.

S3 credentials: access key + secret key are the S3_ACCESS_KEY / S3_SECRET_KEY Dockhand stack secrets on the nats stack (seeded once out-of-band, not in git; never sent by CI). They are SigV4 keys for the nats-s3 gateway, not NATS credentials. The gateway's NATS identity is user nats-s3 (JWT + seed), kept in gitignored secrets/nats-s3.creds and mounted from the s3creds volume.

Connecting

# nats CLI (mise: see .mise.toml)
nats context add barbara8 -s wss://nats.barbara8.de --creds secrets/nats.creds
nats context select barbara8
nats account info
nats stream ls

# any NATS client
nats URL:      wss://nats.barbara8.de
auth:          user JWT + seed (or the .creds file above)

JetStream is enabled for account NATS (unlimited memory/disk by Tower's default limits), so stream/consumer operations work as-is.

S3 endpoint (works with any S3 client; region us-east-1, path style):

Client compatibility note (verified 2026-09-20): nats-s3 v0.4.1 requires a Content-Length on PUT and does not support S3 flexible checksums (trailer-based). Recent AWS SDKs/CLI (botocore ≥ 1.38-ish, aws-sdk-go-v2, JS v3) enable default request checksums and, over HTTPS only, switch to aws-chunked + Transfer-Encoding: chunked without Content-Length — the gateway rejects those as MissingFields. Fix: disable default checksums per client, e.g. env AWS_REQUEST_CHECKSUM_CALCULATION=when_required (AWS CLI / boto3 Config(request_checksum_calculation="when_required")) — or talk to the endpoint over plain HTTP (Tailscale http://100.96.0.81:18444), where the SDKs keep header-based checksums + Content-Length. Older SDKs without default flexible checksums work as-is.

export AWS_REQUEST_CHECKSUM_CALCULATION=when_required   # see note above

# aws cli (v1 or v2)
aws --endpoint-url https://s3.barbara8.de --region us-east-1 s3 ls
aws --endpoint-url https://s3.barbara8.de --region us-east-1 s3 mb s3://my-bucket
aws --endpoint-url https://s3.barbara8.de --region us-east-1 s3 cp file.txt s3://my-bucket/file.txt

# mc (works with any S3v4 endpoint)
mc alias set barbara8 https://s3.barbara8.de "$S3_ACCESS_KEY" "$S3_SECRET_KEY" --api S3v4
mc ls barbara8/

# gateway health (via Tailscale or through Caddy)
curl -s http://100.96.0.81:18444/healthz   # -> 200 when connected to NATS

Repo layout

docker/
  nats-tower/docker-compose.yml   # tower stack (source of truth)
  nats/docker-compose.yml         # server stack (source of truth)
  nats/Dockerfile                 # custom server image (upstream + operator config)
  nats/config/                    # baked into the server image at /etc/nats
    nats-server.conf              # ports, websocket, jetstream, full resolver
    operator.conf                 # operator + system_account (Tower-generated)
    operator.jwt                  # operator JWT (public)
    jwt/<SYS-pubkey>.jwt          # seed for the full-resolver dir (SYS account)
  deploy.ts                       # two-step Dockhand deployer (bun)
.forgejo/workflows/ci.yml         # validate + build image + deploy on push to main

Images (all on the org registry — no external registries at deploy time)

Image	Source
code.schwanke.it/home/nats/nats-tower:v0.6.0	mirror of ghcr.io/nats-tower/nats-tower:v0.6.0
code.schwanke.it/home/nats/nats:2.14.2-alpine	mirror of Docker Hub nats:2.14.2-alpine
code.schwanke.it/home/nats/nats-server:<sha>	built here (Dockerfile) on every push to main; bakes the operator config + the jwt/ resolver seed dir
code.schwanke.it/home/nats/nats-s3:v0.4.1	mirror of ghcr.io/wpnpeiris/nats-s3:v0.4.1

The compose files reference the images via ${TOWER_IMAGE} / ${NATS_IMAGE} / ${NATS_S3_IMAGE} Dockhand stack env vars (tag-agnostic source of truth); CI pins them per commit.

Deploy

CI (.forgejo/workflows/ci.yml, on push to main):

1. validates the compose files,
2. builds + pushes nats-server:<sha> (operator config from the repo),
3. runs docker/deploy.ts (bun) which upserts both stacks (nats-tower, nats) with start:false and deploys them (pull:true) on Dockhand env 3 — the standard two-step flow; CI sends only the non-secret image refs, so the isSecret TOWER_ADMIN_PASSWORD / S3_ACCESS_KEY / S3_SECRET_KEY stay untouched.

First-time bootstrap (already done, one-time, out-of-band) created the stacks with their stack secrets (TOWER_ADMIN_PASSWORD for nats-tower, S3_ACCESS_KEY + S3_SECRET_KEY for nats) before the first compose up. It also seeded the s3creds volume with the gateway's NATS creds file (secrets/nats-s3.creds, gitignored) and set it owned by uid/gid 1000 (the app user the gateway runs as).

Operations

• Tower data: sqlite in the pbdata volume (/pb_data in the container). Back that up (it is the operator keypair). Losing it loses the operator. The NATS stack's jsdata volume (JetStream — including all S3 bucket objects, which are object-store streams) and natsjwt (account JWTs) are the other stateful bits worth backing up.
• Server restart: docker restart nats-nats-1 on UM890 (or re-deploy the stack). Accounts are loaded from the natsjwt volume automatically — no action needed.
• Resolving accounts: the full resolver reads /etc/nats/jwt (type: full, allow_delete: false, limit: 1000). Account JWTs arrive there when Tower publishes them; the SYS seed ships in the image.
• New account/user: Tower UI (Accounts / Users per installation).
  • S3 buckets: just JetStream object streams — nats stream ls shows OBJ_<bucket>; delete a bucket via the S3 API or nats stream delete OBJ_<bucket>.
• Regenerated operator config (e.g. after a Tower re-bootstrap): Tower UI → installation → Key → "Copy as NATS Config" → update docker/nats/config/operator.conf, operator.jwt, config/jwt/<SYS-pubkey>.jwt → push to main (CI rebuilds + redeploys).
• Routes: rendered by the caddy-ingress-controller from the schwanke.it/ingress-* labels in the compose files.