From 874e4d1b2e5ac2bc95cca3b8c8c4051b5da1b80a Mon Sep 17 00:00:00 2001 From: "Project parkour-dev @ molgen" Date: Mon, 5 Oct 2026 12:59:41 +0200 Subject: [PATCH] CLAUDE.md: upgrade runbook from the 26.10.05 dev cutover Extend the cutover procedure with what the 26.08.31-dev -> 26.10.05-dev upgrade needed: bump BUILD_TAG in build.local, move a test-initdb'd pgdata aside, baseline + migrate --plan preview, schema diff against a from-scratch migrate, and rollback. Document upstream's September migration-history rewrite (renamed and edited-in-place migrations), which path is affected (develop Aug-Sep, 26.07.x releases, 26.04.23), and test-parkour2.sh. Fix the stale note that uv isn't on beehive. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 47 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 38 insertions(+), 9 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f855b60..5d5011e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -158,19 +158,48 @@ Older deployments built before this scheme existed (their `backend/media` is sti #### Cutting over to a new release (verified procedure) -1. `./buildall.sh` for the new `BUILD_TAG` - produces a fully built but not-yet-started `${PROJECT}` tree, with its own empty `${PROJECT}/media` and no `pgdata` yet. -2. Stop the *old* release's full stack: `cd && ./startstop.sh stop`. Postgres needs a clean shutdown before `pgdata` is copied - copying a live data directory risks a torn/inconsistent snapshot mid-WAL-write. -3. Copy state into the new release: +0. Bump `BUILD_TAG` in **`build.local`** too, not just `build.profile`: `build.local` sets its own `BUILD_TAG` (e.g. `26.10.05-dev`), and since `build.profile` sources it, that value wins. If you forget, `PROJECT` still points at the *live* tree and `buildall.sh` rebuilds into it. +1. `./buildall.sh` for the new `BUILD_TAG`. This produces a fully built but not-yet-started `${PROJECT}` tree, with its own empty `${PROJECT}/media` and no `pgdata` yet. If you started the new tree's Postgres before cutover (e.g. to run `./test-parkour2.sh`), `startstop.sh` will have `initdb`'d a fresh empty `pgdata` there: stop it and `mv pgdata pgdata.empty-initdb` first, or the rsync below mixes two clusters. +2. Take a baseline from the old release: row counts of a few key models, the newest request, and the Index Pairs admin changelist rendered in-process via `RequestFactory` (no session written). +3. Stop the *old* release's full stack: `cd && ./startstop.sh stop`. Postgres needs a clean shutdown before `pgdata` is copied, because copying a live data directory risks a torn/inconsistent snapshot mid-WAL-write. Check that `pgdata/postmaster.pid` is gone. (Don't check with `pgrep -f `: it matches your own shell command line.) +4. Copy state into the new release: ```bash rsync -a /pgdata/ /pgdata/ - rsync -a //backend/media/ /media/ + rsync -a /media/ /media/ ``` -4. Flip the pointer: `ln -sfn ${PROJECT_ROOT}/prod` -5. Start the new stack: `cd && ./startstop.sh start`. The copied `pgdata` still has the *old* release's absolute paths baked into `postgresql.conf` - no manual fix needed, `srv_postgresql_start` (see `startstop.build.sh`) self-heals `unix_socket_directories`/`log_directory`/`cluster_name` to the new instance's own paths on first start. -6. Verify before treating the old release as decommissioned: site responds (`curl`), gunicorn's startup log shows migrations applied cleanly (`No migrations to apply` is expected/fine if the new release's schema already matched what was in the copied DB), a known DB row round-trips correctly, and a real file under the new `${PROJECT}/media` is servable over HTTP through the new nginx. +5. Flip the pointer: `ln -sfn ${PROJECT_ROOT}/prod` +6. Start **only** Postgres (`./startstop.sh start postgresql`) and preview with `python manage.py migrate --plan` before gunicorn runs it for real. The copied `pgdata` still has the *old* release's absolute paths baked into `postgresql.conf`. No manual fix is needed: `srv_postgresql_start` (see `startstop.build.sh`) self-heals `unix_socket_directories`/`log_directory`/`cluster_name` to the new instance's own paths on first start. +7. Start the rest: `./startstop.sh start gunicorn` (runs `migrate`), then `./startstop.sh start nginx`. +8. Verify before treating the old release as decommissioned: + - the site responds (`curl`); + - gunicorn's startup log shows migrations applied cleanly (`No migrations to apply` is fine if the schema already matched); + - `makemigrations --check` and `migrate --plan` report nothing; + - the baseline from step 2 is unchanged; + - a real file under the new `${PROJECT}/media` is servable over HTTP. -Afterward, the old release's `pgdata`/`media` are a frozen, point-in-time rollback snapshot - confirm nothing still references them (no config, `lsof +D` on both dirs, no cron/systemd) before treating that confirmation as final; safe to leave indefinitely (they're tiny) or delete once confidence in the new release is established. + After a version jump with migrations, also compare `pg_dump --schema-only` of the migrated DB against a scratch DB (`createdb parkour2_fresh`, `DATABASE_URL=...parkour2_fresh... manage.py migrate`). Normalise the 8-hex-digit Django constraint/index hash suffixes and ignore trailing commas, since column order differs. This is the only check that catches migrations upstream edited after they were applied (see below). + +**Rollback** is cheap as long as the old release's `pgdata` is untouched: stop the new stack, `ln -sfn ${PROJECT_ROOT}/prod`, start the old stack. Afterward, the old release's `pgdata`/`media` are a frozen, point-in-time rollback snapshot. Confirm nothing still references them (no config, `lsof +D` on both dirs, no cron/systemd) before treating that confirmation as final. They're safe to leave indefinitely (they're tiny) or to delete once confidence in the new release is established. + +#### Upstream migration-history hazards (seen at 26.10.05) + +In September 2026 upstream `2d4b3ded`/`490cc398` rewrote migrations that existing databases had already applied: + +- **Renamed migrations.** This only hits DBs that ran `develop` between 26.07.28 and 2026-09-11. `migrate` fails with `relation "common_attachmentfiletype" already exists`. Fix: `migrate --fake` of `common.0022_attachmentfiletype_invoicingtemplate_and_more`, `common.0023_attachmentfiletype_seed_defaults`, `common.0024_transliterate_umlaut_user_names`, `invoicing.0006_delete_invoicingreport` and `request.0015_filerequest_file_type`; then run `migrate` normally. Verified operation-by-operation against the old files; used for dev `26.08.31-dev` -> `26.10.05-dev`. +- **`library.0012_alter_library_name` / `sample.0017_alter_sample_name` edited in place.** In 26.10.05 they sanitise names to `[A-Za-z0-9_-]` (max 99 chars, no trailing non-alphanumeric) and shrink `name` to `varchar(99)`. DBs that applied the **26.07.14-26.07.28** versions silently skip both. Fix: sanitise (see `fix_invalid_names` in those files), then `ALTER TABLE library_library/sample_sample ALTER COLUMN name TYPE varchar(99)`. +- **From 26.04.23 (no `library.0012`/`sample.0017` yet), neither problem occurs.** Nothing applied is renamed, and both migrations run in their new form. But that means **sanitising will rename real library/sample names**, so list the affected names before migrating: + ```sql + select 'library', id, name from library_library where name !~ '^[A-Za-z0-9_-]{1,99}$' or name ~ '[^A-Za-z0-9]$' + union all + select 'sample', id, name from sample_sample where name !~ '^[A-Za-z0-9_-]{1,99}$' or name ~ '[^A-Za-z0-9]$'; + ``` + +Reported upstream (draft: `/project/parkour-dev/git/issue-migration-history-rewrite.md`). + +### Regression tests against the built checkout + +`./test-parkour2.sh [manage.py test args]` runs `tests/parkour2//test_*.py` against `${PROJECT}/${BUILD_parkour2}`. It copies each file into its app for the run and removes it on exit, so `git pull` stays clean. It needs that tree's Postgres running, and Django creates/drops its own `test_parkour2` DB. Current content: the orphaned-IndexPair admin 500, fixed upstream by maxplanck-ie/parkour2#370 (open as of 2026-10-05). Once that's in the pinned release, the test should pass. ## Source download strategy -Sources for compiled components (SQLite, Python, Node, PostgreSQL, nginx) are downloaded from the private mirror at `beehive.molgen.mpg.de` (MD5 hash in the URL path). `uv` is downloaded directly from GitHub Releases (no beehive mirror for it). +Sources for compiled components (SQLite, Python, Node, PostgreSQL, nginx) are downloaded from the private mirror at `beehive.molgen.mpg.de` (MD5 hash in the URL path). `uv` is mirrored there too (fetch it from GitHub Releases, then mirror it like the others).