Skip to content

CLAUDE.md: upgrade runbook from the 26.10.05 dev cutover #5

Merged
Peter Marquardt (wwwutz) merged 1 commit into from
Oct 5, 2026
Merged

CLAUDE.md: upgrade runbook from the 26.10.05 dev cutover #5

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 38 additions & 9 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,19 +158,48 @@ Older deployments built before this scheme existed (their `backend/media` is sti

#### Cutting over to a new release (verified procedure)

1. `./buildall.sh` for the new `BUILD_TAG` - produces a fully built but not-yet-started `${PROJECT}` tree, with its own empty `${PROJECT}/media` and no `pgdata` yet.
2. Stop the *old* release's full stack: `cd <old PROJECT> && ./startstop.sh stop`. Postgres needs a clean shutdown before `pgdata` is copied - copying a live data directory risks a torn/inconsistent snapshot mid-WAL-write.
3. Copy state into the new release:
0. Bump `BUILD_TAG` in **`build.local`** too, not just `build.profile`: `build.local` sets its own `BUILD_TAG` (e.g. `26.10.05-dev`), and since `build.profile` sources it, that value wins. If you forget, `PROJECT` still points at the *live* tree and `buildall.sh` rebuilds into it.
1. `./buildall.sh` for the new `BUILD_TAG`. This produces a fully built but not-yet-started `${PROJECT}` tree, with its own empty `${PROJECT}/media` and no `pgdata` yet. If you started the new tree's Postgres before cutover (e.g. to run `./test-parkour2.sh`), `startstop.sh` will have `initdb`'d a fresh empty `pgdata` there: stop it and `mv pgdata pgdata.empty-initdb` first, or the rsync below mixes two clusters.
2. Take a baseline from the old release: row counts of a few key models, the newest request, and the Index Pairs admin changelist rendered in-process via `RequestFactory` (no session written).
3. Stop the *old* release's full stack: `cd <old PROJECT> && ./startstop.sh stop`. Postgres needs a clean shutdown before `pgdata` is copied, because copying a live data directory risks a torn/inconsistent snapshot mid-WAL-write. Check that `pgdata/postmaster.pid` is gone. (Don't check with `pgrep -f <tag>`: it matches your own shell command line.)
4. Copy state into the new release:
```bash
rsync -a <old PROJECT>/pgdata/ <new PROJECT>/pgdata/
rsync -a <old PROJECT>/<old BUILD_parkour2>/backend/media/ <new PROJECT>/media/
rsync -a <old PROJECT>/media/ <new PROJECT>/media/
```
4. Flip the pointer: `ln -sfn <new BUILD_TAG> ${PROJECT_ROOT}/prod`
5. Start the new stack: `cd <new PROJECT> && ./startstop.sh start`. The copied `pgdata` still has the *old* release's absolute paths baked into `postgresql.conf` - no manual fix needed, `srv_postgresql_start` (see `startstop.build.sh`) self-heals `unix_socket_directories`/`log_directory`/`cluster_name` to the new instance's own paths on first start.
6. Verify before treating the old release as decommissioned: site responds (`curl`), gunicorn's startup log shows migrations applied cleanly (`No migrations to apply` is expected/fine if the new release's schema already matched what was in the copied DB), a known DB row round-trips correctly, and a real file under the new `${PROJECT}/media` is servable over HTTP through the new nginx.
5. Flip the pointer: `ln -sfn <new BUILD_TAG> ${PROJECT_ROOT}/prod`
6. Start **only** Postgres (`./startstop.sh start postgresql`) and preview with `python manage.py migrate --plan` before gunicorn runs it for real. The copied `pgdata` still has the *old* release's absolute paths baked into `postgresql.conf`. No manual fix is needed: `srv_postgresql_start` (see `startstop.build.sh`) self-heals `unix_socket_directories`/`log_directory`/`cluster_name` to the new instance's own paths on first start.
7. Start the rest: `./startstop.sh start gunicorn` (runs `migrate`), then `./startstop.sh start nginx`.
8. Verify before treating the old release as decommissioned:
- the site responds (`curl`);
- gunicorn's startup log shows migrations applied cleanly (`No migrations to apply` is fine if the schema already matched);
- `makemigrations --check` and `migrate --plan` report nothing;
- the baseline from step 2 is unchanged;
- a real file under the new `${PROJECT}/media` is servable over HTTP.

Afterward, the old release's `pgdata`/`media` are a frozen, point-in-time rollback snapshot - confirm nothing still references them (no config, `lsof +D` on both dirs, no cron/systemd) before treating that confirmation as final; safe to leave indefinitely (they're tiny) or delete once confidence in the new release is established.
After a version jump with migrations, also compare `pg_dump --schema-only` of the migrated DB against a scratch DB (`createdb parkour2_fresh`, `DATABASE_URL=...parkour2_fresh... manage.py migrate`). Normalise the 8-hex-digit Django constraint/index hash suffixes and ignore trailing commas, since column order differs. This is the only check that catches migrations upstream edited after they were applied (see below).

**Rollback** is cheap as long as the old release's `pgdata` is untouched: stop the new stack, `ln -sfn <old BUILD_TAG> ${PROJECT_ROOT}/prod`, start the old stack. Afterward, the old release's `pgdata`/`media` are a frozen, point-in-time rollback snapshot. Confirm nothing still references them (no config, `lsof +D` on both dirs, no cron/systemd) before treating that confirmation as final. They're safe to leave indefinitely (they're tiny) or to delete once confidence in the new release is established.

#### Upstream migration-history hazards (seen at 26.10.05)

In September 2026 upstream `2d4b3ded`/`490cc398` rewrote migrations that existing databases had already applied:

- **Renamed migrations.** This only hits DBs that ran `develop` between 26.07.28 and 2026-09-11. `migrate` fails with `relation "common_attachmentfiletype" already exists`. Fix: `migrate --fake` of `common.0022_attachmentfiletype_invoicingtemplate_and_more`, `common.0023_attachmentfiletype_seed_defaults`, `common.0024_transliterate_umlaut_user_names`, `invoicing.0006_delete_invoicingreport` and `request.0015_filerequest_file_type`; then run `migrate` normally. Verified operation-by-operation against the old files; used for dev `26.08.31-dev` -> `26.10.05-dev`.
- **`library.0012_alter_library_name` / `sample.0017_alter_sample_name` edited in place.** In 26.10.05 they sanitise names to `[A-Za-z0-9_-]` (max 99 chars, no trailing non-alphanumeric) and shrink `name` to `varchar(99)`. DBs that applied the **26.07.14-26.07.28** versions silently skip both. Fix: sanitise (see `fix_invalid_names` in those files), then `ALTER TABLE library_library/sample_sample ALTER COLUMN name TYPE varchar(99)`.
- **From 26.04.23 (no `library.0012`/`sample.0017` yet), neither problem occurs.** Nothing applied is renamed, and both migrations run in their new form. But that means **sanitising will rename real library/sample names**, so list the affected names before migrating:
```sql
select 'library', id, name from library_library where name !~ '^[A-Za-z0-9_-]{1,99}$' or name ~ '[^A-Za-z0-9]$'
union all
select 'sample', id, name from sample_sample where name !~ '^[A-Za-z0-9_-]{1,99}$' or name ~ '[^A-Za-z0-9]$';
```

Reported upstream (draft: `/project/parkour-dev/git/issue-migration-history-rewrite.md`).

### Regression tests against the built checkout

`./test-parkour2.sh [manage.py test args]` runs `tests/parkour2/<app>/test_*.py` against `${PROJECT}/${BUILD_parkour2}`. It copies each file into its app for the run and removes it on exit, so `git pull` stays clean. It needs that tree's Postgres running, and Django creates/drops its own `test_parkour2` DB. Current content: the orphaned-IndexPair admin 500, fixed upstream by maxplanck-ie/parkour2#370 (open as of 2026-10-05). Once that's in the pinned release, the test should pass.

## Source download strategy

Sources for compiled components (SQLite, Python, Node, PostgreSQL, nginx) are downloaded from the private mirror at `beehive.molgen.mpg.de` (MD5 hash in the URL path). `uv` is downloaded directly from GitHub Releases (no beehive mirror for it).
Sources for compiled components (SQLite, Python, Node, PostgreSQL, nginx) are downloaded from the private mirror at `beehive.molgen.mpg.de` (MD5 hash in the URL path). `uv` is mirrored there too (fetch it from GitHub Releases, then mirror it like the others).