Update 3.3.1

CLAUDE.md

@@ -42,7 +42,9 @@ There is no test suite. Lint shell scripts with `shellcheck --rcfile=shellcheckr
 | `${PROJECT}/profile` | Generated runtime env sourced by `startstop.sh`; chains all component `profile` files |
 | `${PROJECT}/paperless-ngx/paperless.conf` | Generated Django/paperless config |
 
-To update a component version: change `BUILD_<name>` and `BUILD_<name>_SRCURL` in `build.profile`, delete its `.built` tag, re-run `buildall.sh`.
+To update a component version: change `BUILD_<name>` and `BUILD_<name>_SRCURL` in `build.profile`, delete its `.built` tag, re-run `buildall.sh`. New tarballs must be mirrored first with `md5repo.sh <upstream-url>` (only available on the dev host; production hosts just pull the resolved URLs).
+
+`.built` tags live in `${BUILD_DIR}/<pkg-version>/` and are **not** per `BUILD_TAG`. When `BUILD_TAG` changes (new `PREFIX`), components whose version did not change would be skipped and missing from the new prefix — remove all tags first: `find build/ -mindepth 2 -maxdepth 2 -name .built -delete`.
 
 ## Runtime architecture
 
@@ -71,9 +73,9 @@ PID/PGID files live in `$DEVSHM` (`/dev/shm/${USER}/${BUILD_TAG}`). Logs are in
 
 ## Key patches applied during build
 
-- **`nginx.chicken.patch`**: Renames nginx as "chicken" in HTTP headers, error pages, and version strings. Applied in `nginx.build.sh`. The Server response header advertises `chicken/1.32.0` instead of `nginx/...`.
+- **`nginx.chicken.patch`**: Renames nginx as "chicken" in HTTP headers, error pages, and version strings. Applied in `nginx.build.sh` (only if not already applied; a rejected hunk fails the build, and after install `nginx -v` must report `chicken/`). The Server response header says `chicken`. The patch carries the version lines of `src/core/nginx.h` as context, so check that it still applies on every nginx bump. The same patch is shared with immich/toss-baremetal.
 
-- **`paperless-ngx.asgi.patch`**: Backports a fix for [Django ticket #36399](https://code.djangoproject.com/ticket/36399) into paperless-ngx's `asgi.py`. Has an `assert django.VERSION < (6,0)` guard so it will fail loudly when Django 6 ships and the upstream fix lands. Applied in `paperless-ngx.build.sh`.
+The former `paperless-ngx.asgi.patch` (backport of [Django ticket #36399](https://code.djangoproject.com/ticket/36399), multiple `Cookie` headers joined with `,`) was removed: it had been disabled since 2.16.3, and the bug cannot trigger here because nginx merges HTTP/2/3 cookie headers into a single `Cookie` header and talks HTTP/1.1 to granian. Django 5.2.x still has the bug, so only revisit this if granian is ever exposed directly.
 
 ## systemd
 
@@ -98,12 +100,12 @@ The build order in `buildall.sh` respects these dependencies (see `README.dot` f
 
 ## Source download strategy
 
-Sources are downloaded from a private mirror at `beehive.molgen.mpg.de` (MD5 hash in the URL path) rather than upstream directly. Git-based components (imagemagick, pngquant, unpaper, tesseract, jbig2enc, jbig2dec, qpdf) are cloned from GitHub, then cached as `.tar` archives in `${BUILD_DIR}` to speed up rebuilds.
+Sources are downloaded from a private mirror at `beehive.molgen.mpg.de` (MD5 hash in the URL path) rather than upstream directly. Git-based components (imagemagick, pngquant, unpaper, tesseract, jbig2enc, jbig2dec, qpdf) are cloned from GitHub, then cached as `.tar` archives in `${BUILD_DIR}` to speed up rebuilds. Tagged ones clone `--branch <tag>`; the snapshot ones (jbig2enc, pngquant, unpaper) use a `git describe`-style pin like `unpaper-7.0.0-127-g5bcef8a` and the build script checks out the `<sha>` after `-g`, so to bump one put the new `git describe --tags` output into `BUILD_<name>`.
 
 ## Utility scripts
 
-- `checkversions.sh` — checks installed versions against upstream releases for all components; sources `${PROJECT}/profile` to find installed binaries
-- `cleanall.sh` — removes all build directories (not nginx) and logs; run from `${BUILD_DIR}`
+- Version checks use the shared `../baremetals/checkversions.sh .` (run from this repo; compares `build.profile` pins against upstream). The old local `checkversions.sh` was removed.
+- `cleanall.sh` — removes all component build trees (and with them their `.built` tags) and logs under `${BUILD_DIR}`, except nginx; cached source tarballs are kept. Run from the repo directory
 - `runifchanged.sh` — wraps `inotifywait` to re-execute a script when its file is saved (development helper)
 
 ## Management commands
@@ -119,7 +121,7 @@ After build, wrapper scripts for paperless Django management commands are instal
 
 ## Data migration between versions
 
-Set `BUILD_TAG` in `build.profile` to the new version and rebuild. Then:
+Set `BUILD_TAG` in `build.profile` to the new version, remove all `.built` tags (see above) and rebuild. Then (see `UPGRADE.md` for the 2.20.x → 3.x roadmap):
 
 ```bash
 OLD=~/paperless-server/2.8.6

README.dot

@@ -7,7 +7,6 @@ digraph depends {
   FF [ label="ffmpeg" ]
   PN [ label="pngquant" ]
   UN [ label="unpaper" ]
-  ZB [ label="zbar" ]
   PL [ label="paperless" ]
   LE [ label="leptonica" ]
   TE [ label="tesseract" ]
@@ -39,7 +38,7 @@ digraph depends {
 
   { SQ, PY } -> NG
 
-  { PY, SQ, NO, FF, PN, ZB, UN, LE, TE, JD, JE, NG } -> PL
+  { PY, SQ, NO, FF, PN, UN, LE, TE, JD, JE, NG } -> PL
 
 }
 

UPGRADE.md

@@ -0,0 +1,204 @@
+# Upgrade roadmap: 2.20.x → 2.20.15 → 3.3.1
+
+This runbook is for production hosts. Everything they need is already in the repo:
+all source tarballs are pre-mirrored on `beehive.molgen.mpg.de`, with the URLs resolved in `build.profile`.
+You do **not** need `md5repo.sh`, `../baremetals` or any sibling repo.
+
+paperless-ngx only supports upgrading to v3 **from 2.20.15**
+([upstream migration guide](https://github.com/paperless-ngx/paperless-ngx/blob/v3.3.1/docs/migration-v3.md)).
+So every host goes through two hops. Each hop is a git tag:
+
+| Tag | `BUILD_TAG` | What it is |
+|---|---|---|
+| `paperless-2.20.15-4` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 |
+| `paperless-3.3.1-8`   | 3.3.1   | paperless 3.3.1 plus the build/config changes v3 needs |
+
+**Use only these two tags.** All older `paperless-2.20.15*` and `paperless-3.3.1*` tags are superseded:
+- libjpeg-turbo 3.2.0 fails to build on them.
+- On most of them, `buildall.sh` reports success even when a component failed.
+- nginx shows `Server: nginx` instead of chicken.
+- The consume-test PDF is malformed.
+- The `bin/<command>` wrappers drop all their arguments.
+
+Both hops were tested end to end on a fresh host with these two tags:
+- full build of each hop
+- start, consume with OCR and login
+- data migration 2.20.15 → 3.3.1, including the Tantivy reindex and search
+
+Do **not** skip hop 1. 2.20.15 must start once, so its DB migrations get applied, before you build 3.3.1.
+
+---
+
+## 0. Before you start (once per host)
+
+- **Network.** The host needs to reach `beehive.molgen.mpg.de`, `github.com`, PyPI, the npm registry and,
+  for v3 only, `download.pytorch.org`.
+- **Disk.** Each hop is a fresh install under `~/paperless-server/<BUILD_TAG>`, and `data/` + `media/` get *copied*.
+  You need room for roughly two times `media/` plus about 10 GB for the build (v3 pulls in torch, which is CPU-only but still large).
+- **CPU (v3).** NumPy needs SSE4.2: `grep -o -m1 sse4_2 /proc/cpuinfo` must print something.
+  If it prints nothing, add `PAPERLESS_TRAIN_TASK_CRON=disable` to the conf (see the migration guide).
+- **`build.local`.** Keep the one you already have. Check that it sets a fixed `PAPERLESS_SECRET_KEY`.
+  If it doesn't, copy the value out of the running install's `paperless-ngx/paperless.conf` into `build.local`.
+  Otherwise every rebuild generates a new key, which logs everyone out, and v3 refuses to start without a key.
+- **Leftovers in `consume/`.** List them: `find -L "$OLD/consume" -type f ! -path '*/.st*'`. v2 rejected duplicates and left
+  the files there, so they're retried on every start. v3 would *import* them; see the duplicates row in the table under hop 2.
+- **Backup.** With the old install running:
+  ```bash
+  OLD=~/paperless-server/2.20.10        # your current version
+  mkdir -p "$OLD/export"
+  cd "$OLD/paperless-ngx/src" && ( . "$OLD/profile"; python3 manage.py document_exporter "$OLD/export" )
+  ls "$OLD/export"                      # must contain manifest.json
+  ```
+  Call `manage.py` directly here. On installs built from older tags, `bin/document_exporter` drops its arguments
+  and exports into `paperless-ngx/src` instead.
+
+## Hop 1: 2.20.x → 2.20.15
+
+```bash
+cd <path>/paperless-baremetal
+git fetch --tags origin            # or whichever remote this host pulls from
+git checkout paperless-2.20.15-4
+
+# BUILD_TAG changed, so remove all .built tags. Otherwise components whose
+# version did not change get skipped and are missing from the new prefix.
+find build/ -mindepth 2 -maxdepth 2 -name .built -delete
+
+./buildall.sh                      # long; log in build/logs/build.log
+```
+
+Make sure every component built: `grep -i error build/logs/build.log`, and check that no `build/logs/<pkg>.log` ends in a failure.
+
+**Switch over:**
+```bash
+OLD=~/paperless-server/2.20.10
+NEW=~/paperless-server/2.20.15
+
+cd "$OLD" && ./startstop.sh stop
+rm -rf "$NEW/data" "$NEW/media"
+cp -ax "$OLD/data"  "$NEW/"
+cp -ax "$OLD/media" "$NEW/"
+# move any not-yet-consumed files
+cp -ax "$OLD/consume/." "$NEW/consume/" 2>/dev/null || :
+cd "$NEW" && ./startstop.sh start   # runs pending migrations automatically
+./startstop.sh status
+```
+
+If you use the systemd unit, `systemd.build.sh` printed the new `sudo install` / `systemctl` commands.
+Re-install the unit, because its path contains `BUILD_TAG`.
+
+**Encrypted documents.** v3 removed GPG support and the `decrypt_documents` command, so check now while 2.20.15 is running:
+```bash
+cd "$NEW/paperless-ngx/src" && . "$NEW/profile"
+./manage.py shell -c "from documents.models import Document; print(Document.objects.filter(storage_type='gpg').count())"
+```
+If this prints anything other than `0`, run `"$NEW/bin/decrypt_documents" --passphrase <old PAPERLESS_PASSPHRASE>` before hop 2.
+
+**Verify hop 1:**
+- Web login works; the UI footer or settings show 2.20.15.
+- Drop the repo's `paperless-online-test.pdf` into `$NEW/consume/`. It should appear as a document with OCR text.
+- `log/` has no tracebacks: `grep -il traceback "$NEW"/log/*`.
+- Leave it running for a bit. The old install stays untouched as your rollback.
+
+## Hop 2: 2.20.15 → 3.3.1
+
+Read the "Breaking changes" section below first, especially if you use custom pre/post-consume scripts, OIDC or PostgreSQL/MariaDB.
+
+```bash
+cd <path>/paperless-baremetal
+git fetch --tags origin
+git checkout paperless-3.3.1-8
+find build/ -mindepth 2 -maxdepth 2 -name .built -delete
+./buildall.sh
+```
+
+Then switch over exactly as in hop 1, with `OLD=~/paperless-server/2.20.15` and `NEW=~/paperless-server/3.3.1`.
+On first start the worker's migrate step also runs `document_index reindex --if-needed`.
+That **rebuilds the full-text index (Whoosh → Tantivy)** and can take a while on a large archive.
+Search results are incomplete until it finishes.
+
+**Verify hop 2:**
+- Web login works and shows 3.3.1. A `403` on login points at the proxy/IP settings; see below.
+- Search returns results once the reindex is done.
+- The consume test with `paperless-online-test.pdf` works.
+- A barcode/ASN test document still gets split/tagged, if you use barcodes. v3 uses zxing-cpp instead of zbar.
+- `grep -il traceback "$NEW"/log/*` finds nothing. Exceptions: consume errors for files that also failed on v2
+  (e.g. Office files without Tika, 16-bit TIFFs) and duplicate rejections.
+- Retrain the classifier once: `"$NEW/bin/document_create_classifier"`. v3 clears the task history, so the status
+  page warns "No classifier training tasks found". A manual run doesn't clear that warning, because the page only counts
+  *scheduled* training tasks. It goes away after the next scheduled run (hourly by default).
+
+### Breaking changes in v3 and how this repo handles them
+
+| Change | Handled by | Action on the host |
+|---|---|---|
+| `PAPERLESS_SECRET_KEY` required (`change-me` rejected) | conf always sets it | keep it fixed in `build.local` (step 0) |
+| GPG encryption removed, `decrypt_documents` gone | wrapper removed | decrypt during hop 1 if needed |
+| NLTK removed | NLTK download and `PAPERLESS_NLTK_DIR` removed | `~/paperless-server/*/nltk` can be deleted later |
+| pyzbar removed (zxing-cpp only) | zbar component removed from the build | none |
+| Search index Whoosh → Tantivy | `startstop.sh` runs `document_index reindex --if-needed` | wait for the reindex; saved views using `note:`/`custom_field:` are migrated, plain terms no longer match notes |
+| New LLM index | `startstop.sh` runs `document_llmindex migrate`; `bin/document_llmindex` wrapper added | none |
+| Login rate limit uses client IP | conf sets `PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER=X-Real-IP`; nginx sets this header and granian only listens on a unix socket | none |
+| Consumer: `CONSUMER_POLLING*`/`INOTIFY_DELAY` renamed/removed, ignore patterns are now regex | not used here | update them if you added them by hand |
+| Duplicates no longer rejected by default | `PAPERLESS_CONSUMER_DELETE_DUPLICATES` passes through from `build.local` (default `false` = v3 behaviour) | **check `consume/` for old leftover files** that v2 kept rejecting: v3 would import them as duplicates. Set `PAPERLESS_CONSUMER_DELETE_DUPLICATES=true` in `build.local` to reject them like v2; v3 then also *deletes* them from `consume/`, which a syncthing send-receive folder propagates to the sender |
+| `OCR_MODE=skip/skip_noarchive` and `OCR_SKIP_ARCHIVE_FILE` removed; new `ARCHIVE_FILE_GENERATION` defaults to `auto` (no archive for born-digital PDFs) | we use `OCR_MODE=force` (still valid) and set `ARCHIVE_FILE_GENERATION=always`, which keeps the v2 behaviour of always making a PDF/A archive | set `PAPERLESS_ARCHIVE_FILE_GENERATION=auto` in `build.local` if you want the v3 default; check OCR settings in the admin UI if you changed them there |
+| Pre/post-consume scripts no longer get `$1…$8` | — | switch custom scripts to the `DOCUMENT_*` env vars |
+| Postgres/MariaDB need explicit `PAPERLESS_DBENGINE` | SQLite is the default here | only if you configured an external DB |
+| OIDC may need `token_auth_method` | — | only if you use OIDC and get `invalid_client` |
+| Task history cleared | — | none |
+| `torch` (CPU) is now a hard dependency | `paperless-ngx.build.sh` installs from the PyTorch CPU index, like upstream's Dockerfile | network access to `download.pytorch.org` |
+
+## Host variations (check before you start)
+
+Real installs differ from the plain `~/paperless-server/<version>` layout assumed above. Adapt the steps if your host has any of these:
+
+- **Different permissions on the old install** (e.g. `drwxrws---` for a shared project group). New builds use the
+  builder's umask; match the old install before switching: `chmod -R g+w,o-rwx "$NEW"` (doesn't follow the consume symlink).
+- **A shared `BUILD_DIR`** (the default `build`, or `BUILD_DIR=build` in `build.local`). The `.built` markers then belong to
+  whatever version built last. Run the `find build/ … -name .built -delete` step before **each** hop's build, not
+  just once. Otherwise the hop 2 build skips almost everything, and those components are missing from the 3.3.1 install.
+- **A per-version `BUILD_DIR` in `build.local`** (e.g. `BUILD_DIR=/dev/shm/${USER}/build-${BUILD_TAG}`). Every version
+  gets a fresh build dir, so you can skip the `find build/ … -name .built -delete` step. The repo's `build` symlink may still
+  point to an old, deleted build dir; that's harmless.
+- **A shared `consume/` symlink** (e.g. `consume -> ../consume`, fed by syncthing or scanners). The build creates `consume/`
+  as a real directory in the new install. Replace it with the same symlink, and don't run the `cp … consume/.` step,
+  or files land where nobody watches:
+  ```bash
+  rmdir "$NEW/consume" && ln -s "$(readlink "$OLD/consume")" "$NEW/consume"
+  ```
+- **A `prod` symlink plus a top-level `startstop.sh` wrapper** (`cd prod; ./startstop.sh "$@"`), which is what the boot
+  scripts call. Switch over by re-pointing the symlink, then start through the wrapper:
+  ```bash
+  ln -sfn <new-version> /path/to/prod
+  ```
+  Everything that goes through `prod` (e.g. a `media -> prod/media` link) follows automatically.
+- **Fetching with a read-only deploy key**, if the host's default ssh key has no GitHub access:
+  ```bash
+  GIT_SSH_COMMAND="ssh -i ~/.ssh/ghm_paperlessbm_ro -o IdentitiesOnly=yes" git fetch --tags origin
+  ```
+
+**Both hops in one maintenance window.** Building doesn't touch the running install: it uses its own prefix, build dir and
+`/dev/shm` paths, and it doesn't start any services. So you can build both versions ahead of time while the old one keeps
+serving:
+1. Check out the hop 1 tag and build.
+2. Check out the hop 2 tag and build.
+3. In the window: stop the old install, copy to 2.20.15, start it (this runs its migrations), verify, stop it.
+4. Copy to 3.3.1, start it, verify.
+
+## Rollback
+
+Each hop leaves the previous install untouched in its own directory. To roll back, stop the new install
+and start the old one: `cd $OLD && ./startstop.sh start`. Any documents added since the switch-over are
+only in the new install; re-import them from its `media/` or from an export.
+
+## Cleanup (after a week or so of stable operation)
+
+```bash
+rm -rf ~/paperless-server/2.20.10 ~/paperless-server/2.20.15
+```
+
+## Maintainer notes (dev host only)
+
+- Bumping a tarball means running `md5repo.sh <upstream-url>`, then putting the printed beehive URL into `build.profile`.
+- Check versions with `../baremetals/checkversions.sh .`
+- `./cleanall.sh` (from the repo dir) wipes all component build trees except nginx. It's an alternative to
+  the `find … -name .built -delete` step, but it also throws away the extracted sources.