From 80bf1eb1a0e0c0e1ad953798256684b1b1193f2e Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker Date: Tue, 13 Oct 2026 10:33:48 +0200 Subject: [PATCH 01/21] update to 2.20.15, bump deps to current bugfix level, add UPGRADE.md Phase 1 of the 2.20.x -> 3.3.1 upgrade (v3 only migrates from 2.20.15). Versions (siblings' pins where shared, else latest patch): paperless 2.20.15, sqlite 3.53.4, redis 7.4.11, python 3.12.15, node 24.21.0, nginx 1.31.6, ffmpeg 8.0.3, libjpegturbo 3.2.0, libtiff 4.7.2, ghostscript 10.07.1, imagemagick 7.1.2-33, tesseract 5.5.3. All tarballs pre-mirrored on beehive so production hosts need neither md5repo.sh nor the sibling repos. Script fixes ported from the sibling repos: - brace-group "test -d X || { mkdir X && tar ...; }" (tar re-extracted on every run due to && / || precedence) - python: fix malformed rpath "-Wl,-rpath -Wl,-L..." -> "-Wl,-rpath,..." Remove local checkversions.sh in favour of ../baremetals/checkversions.sh. CLAUDE.md: note that .built tags are not per BUILD_TAG, that the asgi patch is not actually applied, and point to UPGRADE.md. Co-Authored-By: Claude Opus 5.5 --- CLAUDE.md | 10 +- UPGRADE.md | 146 ++++++++++++++++++++ build.profile | 62 ++++----- checkversions.sh | 300 ------------------------------------------ ffmpeg.build.sh | 2 +- ghostscript.build.sh | 2 +- leptonica.build.sh | 2 +- libfontttf.build.sh | 2 +- libjpegturbo.build.sh | 2 +- libtiff.build.sh | 2 +- libwebp.build.sh | 2 +- nginx.build.sh | 2 +- node.build.sh | 2 +- python.build.sh | 4 +- redis.build.sh | 2 +- sqlite.build.sh | 2 +- zbar.build.sh | 2 +- 17 files changed, 197 insertions(+), 349 deletions(-) create mode 100644 UPGRADE.md delete mode 100755 checkversions.sh diff --git a/CLAUDE.md b/CLAUDE.md index f11af16..1e90d42 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -42,7 +42,9 @@ There is no test suite. Lint shell scripts with `shellcheck --rcfile=shellcheckr | `${PROJECT}/profile` | Generated runtime env sourced by `startstop.sh`; chains all component `profile` files | | `${PROJECT}/paperless-ngx/paperless.conf` | Generated Django/paperless config | -To update a component version: change `BUILD_` and `BUILD__SRCURL` in `build.profile`, delete its `.built` tag, re-run `buildall.sh`. +To update a component version: change `BUILD_` and `BUILD__SRCURL` in `build.profile`, delete its `.built` tag, re-run `buildall.sh`. New tarballs must be mirrored first with `md5repo.sh ` (only available on the dev host; production hosts just pull the resolved URLs). + +`.built` tags live in `${BUILD_DIR}//` and are **not** per `BUILD_TAG`. When `BUILD_TAG` changes (new `PREFIX`), components whose version did not change would be skipped and missing from the new prefix — remove all tags first: `find build/ -mindepth 2 -maxdepth 2 -name .built -delete`. ## Runtime architecture @@ -73,7 +75,7 @@ PID/PGID files live in `$DEVSHM` (`/dev/shm/${USER}/${BUILD_TAG}`). Logs are in - **`nginx.chicken.patch`**: Renames nginx as "chicken" in HTTP headers, error pages, and version strings. Applied in `nginx.build.sh`. The Server response header advertises `chicken/1.32.0` instead of `nginx/...`. -- **`paperless-ngx.asgi.patch`**: Backports a fix for [Django ticket #36399](https://code.djangoproject.com/ticket/36399) into paperless-ngx's `asgi.py`. Has an `assert django.VERSION < (6,0)` guard so it will fail loudly when Django 6 ships and the upstream fix lands. Applied in `paperless-ngx.build.sh`. +- **`paperless-ngx.asgi.patch`**: Backports a fix for [Django ticket #36399](https://code.djangoproject.com/ticket/36399) into paperless-ngx's `asgi.py`. Has an `assert django.VERSION < (6,0)` guard so it will fail loudly when Django 6 ships and the upstream fix lands. **Currently NOT applied** — no script references it, and the patch is broken as written (`XXXASGIHandler`, missing imports). Open issue; see `UPGRADE.md`. ## systemd @@ -102,7 +104,7 @@ Sources are downloaded from a private mirror at `beehive.molgen.mpg.de` (MD5 has ## Utility scripts -- `checkversions.sh` — checks installed versions against upstream releases for all components; sources `${PROJECT}/profile` to find installed binaries +- Version checks use the shared `../baremetals/checkversions.sh .` (run from this repo; compares `build.profile` pins against upstream). The old local `checkversions.sh` was removed. - `cleanall.sh` — removes all build directories (not nginx) and logs; run from `${BUILD_DIR}` - `runifchanged.sh` — wraps `inotifywait` to re-execute a script when its file is saved (development helper) @@ -119,7 +121,7 @@ After build, wrapper scripts for paperless Django management commands are instal ## Data migration between versions -Set `BUILD_TAG` in `build.profile` to the new version and rebuild. Then: +Set `BUILD_TAG` in `build.profile` to the new version, remove all `.built` tags (see above) and rebuild. Then (see `UPGRADE.md` for the 2.20.x → 3.x roadmap): ```bash OLD=~/paperless-server/2.8.6 diff --git a/UPGRADE.md b/UPGRADE.md new file mode 100644 index 0000000..c1074bd --- /dev/null +++ b/UPGRADE.md @@ -0,0 +1,146 @@ +# Upgrade roadmap: 2.20.x → 2.20.15 → 3.3.1 + +This runbook is for production hosts. Everything they need is already in the repo: +all source tarballs are pre-mirrored on `beehive.molgen.mpg.de`, with the URLs resolved in `build.profile`. +You do **not** need `md5repo.sh`, `../baremetals` or any sibling repo. + +paperless-ngx only supports upgrading to v3 **from 2.20.15** +([upstream migration guide](https://github.com/paperless-ngx/paperless-ngx/blob/v3.3.1/docs/migration-v3.md)). +So every host goes through two hops. Each hop is a git tag: + +| Tag | `BUILD_TAG` | What it is | +|---|---|---| +| `paperless-2.20.15` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 | +| `paperless-3.3.1` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | + +Do **not** skip hop 1. 2.20.15 must start once, so its DB migrations get applied, before you build 3.3.1. + +--- + +## 0. Before you start (once per host) + +- **Network.** The host needs to reach `beehive.molgen.mpg.de`, `github.com`, PyPI, the npm registry and, + for v3 only, `download.pytorch.org`. +- **Disk.** Each hop is a fresh install under `~/paperless-server/`, and `data/` + `media/` get *copied*. + You need room for roughly two times `media/` plus about 10 GB for the build (v3 pulls in torch, which is CPU-only but still large). +- **CPU (v3).** NumPy needs SSE4.2: `grep -o -m1 sse4_2 /proc/cpuinfo` must print something. + If it prints nothing, add `PAPERLESS_TRAIN_TASK_CRON=disable` to the conf (see the migration guide). +- **`build.local`.** Keep the one you already have. Check that it sets a fixed `PAPERLESS_SECRET_KEY`. + If it doesn't, copy the value out of the running install's `paperless-ngx/paperless.conf` into `build.local`. + Otherwise every rebuild generates a new key, which logs everyone out, and v3 refuses to start without a key. +- **Backup.** With the old install running: + ```bash + OLD=~/paperless-server/2.20.10 # your current version + mkdir -p "$OLD/export" + "$OLD/bin/document_exporter" "$OLD/export" + ``` + +## Hop 1: 2.20.x → 2.20.15 + +```bash +cd /paperless-baremetal +git fetch --tags origin # or whichever remote this host pulls from +git checkout paperless-2.20.15 + +# BUILD_TAG changed, so remove all .built tags. Otherwise components whose +# version did not change get skipped and are missing from the new prefix. +find build/ -mindepth 2 -maxdepth 2 -name .built -delete + +./buildall.sh # long; log in build/logs/build.log +``` + +Make sure every component built: `grep -i error build/logs/build.log`, and check that no `build/logs/.log` ends in a failure. + +**Switch over:** +```bash +OLD=~/paperless-server/2.20.10 +NEW=~/paperless-server/2.20.15 + +cd "$OLD" && ./startstop.sh stop +rm -rf "$NEW/data" "$NEW/media" +cp -ax "$OLD/data" "$NEW/" +cp -ax "$OLD/media" "$NEW/" +# move any not-yet-consumed files +cp -ax "$OLD/consume/." "$NEW/consume/" 2>/dev/null || : +cd "$NEW" && ./startstop.sh start # runs pending migrations automatically +./startstop.sh status +``` + +If you use the systemd unit, `systemd.build.sh` printed the new `sudo install` / `systemctl` commands. +Re-install the unit, because its path contains `BUILD_TAG`. + +**Encrypted documents.** v3 removed GPG support and the `decrypt_documents` command, so check now while 2.20.15 is running: +```bash +cd "$NEW/paperless-ngx/src" && . "$NEW/profile" +./manage.py shell -c "from documents.models import Document; print(Document.objects.filter(storage_type='gpg').count())" +``` +If this prints anything other than `0`, run `"$NEW/bin/decrypt_documents" --passphrase ` before hop 2. + +**Verify hop 1:** +- Web login works; the UI footer or settings show 2.20.15. +- Drop the repo's `paperless-online-test.pdf` into `$NEW/consume/`. It should appear as a document with OCR text. +- `log/` has no tracebacks: `grep -il traceback "$NEW"/log/*`. +- Leave it running for a bit. The old install stays untouched as your rollback. + +## Hop 2: 2.20.15 → 3.3.1 + +Read the "Breaking changes" section below first, especially if you use custom pre/post-consume scripts, OIDC or PostgreSQL/MariaDB. + +```bash +cd /paperless-baremetal +git fetch --tags origin +git checkout paperless-3.3.1 +find build/ -mindepth 2 -maxdepth 2 -name .built -delete +./buildall.sh +``` + +Then switch over exactly as in hop 1, with `OLD=~/paperless-server/2.20.15` and `NEW=~/paperless-server/3.3.1`. +On first start the worker's migrate step also runs `document_index reindex --if-needed`. +That **rebuilds the full-text index (Whoosh → Tantivy)** and can take a while on a large archive. +Search results are incomplete until it finishes. + +**Verify hop 2:** +- Web login works and shows 3.3.1. A `403` on login points at the proxy/IP settings; see below. +- Search returns results once the reindex is done. +- The consume test with `paperless-online-test.pdf` works. +- A barcode/ASN test document still gets split/tagged, if you use barcodes. v3 uses zxing-cpp instead of zbar. +- `grep -il traceback "$NEW"/log/*` finds nothing. + +### Breaking changes in v3 and how this repo handles them + +| Change | Handled by | Action on the host | +|---|---|---| +| `PAPERLESS_SECRET_KEY` required (`change-me` rejected) | conf always sets it | keep it fixed in `build.local` (step 0) | +| GPG encryption removed, `decrypt_documents` gone | wrapper removed | decrypt during hop 1 if needed | +| NLTK removed | NLTK download and `PAPERLESS_NLTK_DIR` removed | `~/paperless-server/*/nltk` can be deleted later | +| pyzbar removed (zxing-cpp only) | zbar component removed from the build | none | +| Search index Whoosh → Tantivy | `startstop.sh` runs `document_index reindex --if-needed` | wait for the reindex; saved views using `note:`/`custom_field:` are migrated, plain terms no longer match notes | +| New LLM index | `startstop.sh` runs `document_llmindex migrate`; `bin/document_llmindex` wrapper added | none | +| Login rate limit uses client IP | conf sets `PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER=X-Real-IP`; nginx sets this header and granian only listens on a unix socket | none | +| Consumer: `CONSUMER_POLLING*`/`INOTIFY_DELAY` renamed/removed, ignore patterns are now regex | not used here | update them if you added them by hand | +| Duplicates no longer rejected by default | not set | set `PAPERLESS_CONSUMER_DELETE_DUPLICATES=true` if you want the old behaviour | +| `OCR_MODE=skip/skip_noarchive` and `OCR_SKIP_ARCHIVE_FILE` removed | we use `OCR_MODE=force`, still valid | check OCR settings in the admin UI if you changed them there | +| Pre/post-consume scripts no longer get `$1…$8` | — | switch custom scripts to the `DOCUMENT_*` env vars | +| Postgres/MariaDB need explicit `PAPERLESS_DBENGINE` | SQLite is the default here | only if you configured an external DB | +| OIDC may need `token_auth_method` | — | only if you use OIDC and get `invalid_client` | +| Task history cleared | — | none | +| `torch` (CPU) is now a hard dependency | `paperless-ngx.build.sh` installs from the PyTorch CPU index, like upstream's Dockerfile | network access to `download.pytorch.org` | + +## Rollback + +Each hop leaves the previous install untouched in its own directory. To roll back, stop the new install +and start the old one: `cd $OLD && ./startstop.sh start`. Any documents added since the switch-over are +only in the new install; re-import them from its `media/` or from an export. + +## Cleanup (after a week or so of stable operation) + +```bash +rm -rf ~/paperless-server/2.20.10 ~/paperless-server/2.20.15 +``` + +## Maintainer notes (dev host only) + +- Bumping a tarball means running `md5repo.sh `, then putting the printed beehive URL into `build.profile`. +- Check versions with `../baremetals/checkversions.sh .` +- Open issue: `paperless-ngx.asgi.patch` is not applied by any script and is broken as written. + Either fix and apply it, or delete it. v3.3.1 still ships Django 5.2. diff --git a/build.profile b/build.profile index 1dfcb39..1ba0511 100644 --- a/build.profile +++ b/build.profile @@ -1,6 +1,6 @@ -BUILD_TAG=2.20.10 +BUILD_TAG=2.20.15 -BUILD_paperless=paperless-ngx-2.20.10 +BUILD_paperless=paperless-ngx-2.20.15 BUILD_DIR=build BUILD_SRC="${PWD}" @@ -16,26 +16,26 @@ TMPDIR=${TMPDIR:-/tmp/${USER}/${BUILD_TAG}} LOGDIR=${PROJECT}/log # https://sqlite.org -# wget "https://sqlite.org/2026/sqlite-autoconf-3510200.tar.gz" -BUILD_sqlite=sqlite-3.51.2 -BUILD_sqlite_SRCURL="https://beehive.molgen.mpg.de/49600a5739d382c648b1a317e4b57446/sqlite-autoconf-3510200.tar.gz" +# wget "https://sqlite.org/2026/sqlite-autoconf-3530400.tar.gz" +BUILD_sqlite=sqlite-3.53.4 +BUILD_sqlite_SRCURL="https://beehive.molgen.mpg.de/ee54db5a5b04e77d4883ce0c051a1eba/sqlite-autoconf-3530400.tar.gz" # https://download.redis.io/releases # community edition ! -# wget https://download.redis.io/releases/redis-7.4.8.tar.gz -BUILD_redis=redis-7.4.8 -BUILD_redis_SRCURL="https://beehive.molgen.mpg.de/f8c77d9fa16d3da8c9f745114f4e69e3/${BUILD_redis}.tar.gz" +# wget https://download.redis.io/releases/redis-7.4.11.tar.gz +BUILD_redis=redis-7.4.11 +BUILD_redis_SRCURL="https://beehive.molgen.mpg.de/6d538d9ce2e8c91f0a875886ad550b3c/${BUILD_redis}.tar.gz" -# wget "https://www.python.org/ftp/python/3.12.13/Python-3.12.13.tgz" -BUILD_python=Python-3.12.13 -BUILD_python_SRCURL="https://beehive.molgen.mpg.de/64e26b69406a9e828dce782172f0d99c/${BUILD_python}.tgz" +# wget "https://www.python.org/ftp/python/3.12.15/Python-3.12.15.tgz" +BUILD_python=Python-3.12.15 +BUILD_python_SRCURL="https://beehive.molgen.mpg.de/dbbf38277b3579d0a53aecfd7e9d6012/${BUILD_python}.tgz" -# wget "https://nodejs.org/download/release/v24.14.0/node-v24.14.0.tar.gz" -BUILD_node=node-24.14.0 -BUILD_node_SRCURL="https://beehive.molgen.mpg.de/74be2d0f33e2fc00d3d319885ac0c6e5/${BUILD_node/-/-v}.tar.gz" +# wget "https://nodejs.org/download/release/v24.21.0/node-v24.21.0.tar.gz" +BUILD_node=node-24.21.0 +BUILD_node_SRCURL="https://beehive.molgen.mpg.de/a0d4131e52d9055f8428439542018d44/${BUILD_node/-/-v}.tar.gz" -# SRCURL="https://ffmpeg.org//releases/${BUILD_ffmpeg}.tar.gz" -BUILD_ffmpeg=ffmpeg-8.0 -BUILD_ffmpeg_SRCURL="https://beehive.molgen.mpg.de/74bb5dbf6c2b35ab50aad7a939ef3ea2/${BUILD_ffmpeg}.tar.gz" +# wget "https://ffmpeg.org/releases/ffmpeg-8.0.3.tar.gz" +BUILD_ffmpeg=ffmpeg-8.0.3 +BUILD_ffmpeg_SRCURL="https://beehive.molgen.mpg.de/3057126c124c8515ff7122a6d41d609b/${BUILD_ffmpeg}.tar.gz" # SRCURL="https://github.com/mchehab/zbar/archive/refs/tags/0.23.93.tar.gz" BUILD_zbar=zbar-0.23.93 @@ -45,32 +45,32 @@ BUILD_zbar_SRCURL="https://beehive.molgen.mpg.de/3f69d17f6495de023b59b3539ce5e60 BUILD_leptonica=leptonica-1.87.0 BUILD_leptonica_SRCURL="https://beehive.molgen.mpg.de/d523d2c883b71850a46af2517d8915bf/${BUILD_leptonica}.tar.gz" -# wget "https://nginx.org/download/nginx-1.31.0.tar.gz" -BUILD_nginx=nginx-1.31.0 -BUILD_nginx_SRCURL="https://beehive.molgen.mpg.de/40b2a7057eb55965d42d5416636c27e2/${BUILD_nginx}.tar.gz" +# wget "https://nginx.org/download/nginx-1.31.6.tar.gz" +BUILD_nginx=nginx-1.31.6 +BUILD_nginx_SRCURL="https://beehive.molgen.mpg.de/6d59f919382fae46812c44ffbc87498a/${BUILD_nginx}.tar.gz" # SRCURL="https://github.com/liberationfonts/liberation-fonts/files/7261482/liberation-fonts-ttf-2.1.5.tar.gz" BUILD_libfontttf=liberation-fonts-ttf-2.1.5 BUILD_libfontttf_SRCURL="https://beehive.molgen.mpg.de/31b453e0b77bacde410a34a725b34f8a/${BUILD_libfontttf}.tar.gz" -# wget "https://download.osgeo.org/libtiff/tiff-4.7.1.tar.gz" -BUILD_libtiff=tiff-4.7.1 -BUILD_libtiff_SRCURL="https://beehive.molgen.mpg.de/f1044dd3b4466cc53464210148e08146/${BUILD_libtiff}.tar.gz" +# wget "https://download.osgeo.org/libtiff/tiff-4.7.2.tar.gz" +BUILD_libtiff=tiff-4.7.2 +BUILD_libtiff_SRCURL="https://beehive.molgen.mpg.de/a2a65ea450b8f469c667128848ffebfc/${BUILD_libtiff}.tar.gz" -# SRCURL="https://github.com/libjpeg-turbo/libjpeg-turbo/releases/download/3.1.3/libjpeg-turbo-3.1.3.tar.gz" -BUILD_libjpegturbo=libjpegturbo-3.1.3 -BUILD_libjpegturbo_SRCURL="https://beehive.molgen.mpg.de/d23f3be7e58ad79d297845e64807472c/libjpeg-turbo-3.1.3.tar.gz" +# SRCURL="https://github.com/libjpeg-turbo/libjpeg-turbo/releases/download/3.2.0/libjpeg-turbo-3.2.0.tar.gz" +BUILD_libjpegturbo=libjpegturbo-3.2.0 +BUILD_libjpegturbo_SRCURL="https://beehive.molgen.mpg.de/a80f9f1c735b57d3cdd66daed1ba6bb4/libjpeg-turbo-3.2.0.tar.gz" # SRCURL="https://storage.googleapis.com/downloads.webmproject.org/releases/webp/libwebp-1.6.0.tar.gz" BUILD_libwebp=libwebp-1.6.0 BUILD_libwebp_SRCURL="https://beehive.molgen.mpg.de/cceb6447180f961473b181c9ef38b630/${BUILD_libwebp}.tar.gz" -# wget "https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs10060/ghostscript-10.06.0.tar.gz" -BUILD_ghostscript=ghostscript-10.06.0 -BUILD_ghostscript_SRCURL="https://beehive.molgen.mpg.de/d472f289582b209731954642bd34ca86/${BUILD_ghostscript}.tar.gz" +# wget "https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs10071/ghostscript-10.07.1.tar.gz" +BUILD_ghostscript=ghostscript-10.07.1 +BUILD_ghostscript_SRCURL="https://beehive.molgen.mpg.de/de0ed6fd5dc92556e79160bb5efa1f09/${BUILD_ghostscript}.tar.gz" # gits -BUILD_imagemagick=imagemagick-7.1.2-15 +BUILD_imagemagick=imagemagick-7.1.2-33 BUILD_imagemagick_SRCURL="https://github.com/ImageMagick/ImageMagick.git" # git clone / git describe @@ -80,7 +80,7 @@ BUILD_pngquant_SRCURL="https://github.com/kornelski/pngquant.git" BUILD_unpaper=unpaper-7.0.0-127-g5bcef8a BUILD_unpaper_SRCURL="https://github.com/unpaper/unpaper.git" -BUILD_tesseract=tesseract-5.5.2 +BUILD_tesseract=tesseract-5.5.3 BUILD_tesseract_SRCURL="https://github.com/tesseract-ocr/tesseract.git" BUILD_tessdata=tessdata_best-4.1.0 diff --git a/checkversions.sh b/checkversions.sh deleted file mode 100755 index 8a8727a..0000000 --- a/checkversions.sh +++ /dev/null @@ -1,300 +0,0 @@ -#!/bin/bash -set -u - -. build.profile - -[[ -d "${TMPDIR}" ]] || mkdir -p "${TMPDIR}" - -tmp="${TMPDIR}/cv.$$.tmp" -#tmp=/tmp/cv.tmp - -function rm_tmp() { - : - rm -f "${tmp}" -} - -trap rm_tmp EXIT - -declare -A CHECK_URL - -function parsefile() { - U=$1 - P=$2 - D=${3:-''} - - curl -s -L "${U}" -o "${tmp}" - x1=$(grep --max-count=1 -oP -e "${P}" "${tmp}") - if [[ -n "${D}" ]]; then - x1=${x1}'-'$(grep --max-count=1 -oP -e "${D}" "${tmp}") - fi - echo $x1 -} - -function parselast() { - U=$1 - P=$2 - D=${3:-''} - - curl -s -L "${U}" -o "${tmp}" - x1=$(grep -oP -e "${P}" "${tmp}"|tail -1) - echo $x1 -} - -# git_describe_remote -function gdrem() { - - G=$1 - P=$2 - C=${3:-'(?<=Release ).*?(?=<)'} - D=${4:-''} - L=$(git ls-remote --refs --sort=-v:refname "https://github.com/${G}" "${P}" | head -1) - - - _IFS=$IFS - ${IFS+':'} unset _IFS - IFS=[$'\t'/] - set -- $L - IFS=$_IFS - ${_IFS+':'} unset IFS - - SHA=$1 - TAG=$4 - URL="https://github.com/${G}/releases" - parsefile "${URL}/tag/${TAG}" "${C}" "${D}" - echo "${URL}" -} - -function parse() { - U=$1 - P=$2 - parsefile "${U}" "${P}" - echo "${U}" -} - -function bparse() { - U=$1 - P=$2 - - if [[ $(curl -s -L "${U}") =~ ${P} ]]; then - echo "${BASH_REMATCH[1]}${BASH_REMATCH[2]:+-}${BASH_REMATCH[2]:-}" - fi - echo "${U}" -} - -function mtype() { - local _a=$1 - local _p=$(type -p $_a) - echo '...'${_p#$PREFIX} -} - -# https://api.github.com/repos/paperless-ngx/paperless-ngx/releases/latest& - -if [[ -e "$PROJECT"/profile ]]; then - . "$PROJECT"/profile - function exec_BUILD_node () { - node --version - mtype node - } - function exec_BUILD_python () { - python --version|cut -d' ' -f2 - mtype python - } - function exec_BUILD_redis () { - redis-cli --version|cut -d' ' -f2 - mtype redis-cli - } - function exec_BUILD_sqlite () { - sqlite3 --version|cut -d' ' -f1 - mtype sqlite3 - } - function exec_BUILD_nginx () { - nginx -v 2>&1|cut -d/ -f2|cut -d' ' -f1 - mtype nginx - } - function exec_BUILD_ffmpeg () { - ffmpeg -version | head -1 | cut -d' ' -f3 - mtype ffmpeg - } - function exec_BUILD_imagemagick () { - magick --version | head -1 | cut -d' ' -f3 - mtype magick - } - function exec_BUILD_unpaper () { - unpaper --version | head -1 - mtype unpaper - } - function exec_BUILD_tesseract () { - tesseract --version | head -1 | cut -d' ' -f2 - mtype tesseract - } - function exec_BUILD_zbar () { - zbarimg --version - mtype zbarimg - } - function exec_BUILD_leptonica() { - pkg-config --modversion lept - echo "no-exec" - } - function exec_BUILD_jbig2enc () { - jbig2 -V 2>&1|head -1|cut -d' ' -f2 - mtype jbig2 - } - function exec_BUILD_jbig2dec () { - jbig2dec -V 2>&1|cut -d' ' -f2 - mtype jbig2dec - } - function exec_BUILD_pngquant () { - pngquant -V - mtype pngquant - } - function exec_BUILD_libwebp () { - webpinfo -version|cut -d' ' -f4 - mtype webpinfo - } - function exec_BUILD_libjpegturbo () { - re='version (\S+)\s\(build\s(\S+)\)' - [[ $(cjpeg -version 2>&1) =~ $re ]] && echo "${BASH_REMATCH[1]}-${BASH_REMATCH[2]}" - mtype cjpeg - } - function exec_BUILD_libtiff () { - re='Version\s+(\S+)' - [[ $(tiffinfo 2>&1) =~ $re ]] && echo "${BASH_REMATCH[1]}" - mtype tiffinfo - } - function exec_BUILD_ghostscript () { - gs --version - mtype gs - } - -fi - -version_BUILD_paperless() { - gdrem paperless-ngx/paperless-ngx 'refs/tags/v*'|cut -d' ' -f2 -} -version_BUILD_imagemagick() { - gdrem ImageMagick/ImageMagick 'refs/tags/[0-9]*'|cut -d' ' -f1 -} -version_BUILD_node() { - parse 'https://github.com/nodejs/node/releases' '(?<=Version ).*?(?= ).*LTS\)'|cut -d' ' -f1 -} -version_BUILD_python() { - parselast 'https://raw.githubusercontent.com/python/peps/main/peps/pep-0693.rst' '(?<=^- )[0-9].*(?=: )' -} -version_BUILD_redis() { - parse 'https://github.com/redis/redis/releases' '(?<=/redis/redis/tree/)7.*?(?=")' -} -version_BUILD_sqlite() { - parse 'https://sqlite.org/index.html' '(?<=Version ).*(?=<)' -} -version_BUILD_nginx() { - parse 'https://nginx.org/' '(?<=nginx-).*?(?=<)' -} -version_BUILD_tesseract() { - gdrem tesseract-ocr/tesseract 'refs/tags/[0-9]*'|cut -d' ' -f1 -} -version_BUILD_tessdata() { - gdrem tesseract-ocr/tessdata_best 'refs/tags/[0-9]*' '(?<=Release ).*?(?=\s)'|cut -d' ' -f2 -} -version_BUILD_jbig2dec() { - gdrem ArtifexSoftware/jbig2dec 'refs/tags/[0-9]*'|cut -d' ' -f2 -} -version_BUILD_jbig2enc() { - gdrem agl/jbig2enc 'refs/tags/[0-9]*'|cut -d' ' -f1 -} -version_BUILD_leptonica() { - gdrem DanBloomberg/leptonica 'refs/tags/[0-9]*'|cut -d' ' -f3 -} -version_BUILD_ffmpeg() { - parse 'https://git.videolan.org/?p=ffmpeg.git;a=blob_plain;f=Changelog' '(?<=^version )\d.*(?=:)' -} -version_BUILD_pngquant() { - gdrem kornelski/pngquant 'refs/tags/*'|cut -d' ' -f1 -} -version_BUILD_zbar() { - gdrem mchehab/zbar 'refs/tags/[0-9]*'|cut -d' ' -f2 -} -version_BUILD_unpaper() { - gdrem unpaper/unpaper 'refs/tags/unpaper-*' '(?<=<title>Release unpaper-).*?(?=\s)' '(?<= )[0-9]+(?= commits)'|tr '\n' ' ' -} -version_BUILD_libfontttf() { - gdrem liberationfonts/liberation-fonts 'refs/tags/[0-9]*'|cut -d' ' -f1 -} -version_BUILD_libwebp() { - bparse 'https://chromium.googlesource.com/webm/libwebp/' 'v([1-9]+\.[0-9]+\.[0-9]+)' -} -version_BUILD_libjpegturbo() { - gdrem libjpeg-turbo/libjpeg-turbo 'refs/tags/[0-9]*'|cut -d' ' -f1 -} -version_BUILD_libtiff() { - bparse 'https://download.osgeo.org/libtiff/?C=M&O=D' 'tiff-(\S+?).tar.gz' -} -version_BUILD_qpdf() { - gdrem qpdf/qpdf 'refs/tags/v[0-9]*'|cut -d' ' -f2 -} - -function version() { - Bi=$1 - B=$(eval eval echo \\$\${Bi}) - shift -# printf "%-s\n" "$Bi" - printf "#### %26s\n" "$B" - if [[ $(type -t exec_${Bi}) = 'function' ]]; then -# printf " exe %32s %s\n" "$(exec_${Bi}|tr '\n' ' ')" - v=$(exec_${Bi}|tr '\n' ' ') - printf " exe %26s %s\n" $v - else - : # printf "no exec_()\n" - fi - - if [[ $(type -t version_${Bi}) = 'function' ]]; then - v=$(version_${Bi}) - printf " ver %26s %s\n" $v - else - printf "no version_()\n" - fi - echo -} - -_parts=( -BUILD_paperless -BUILD_node -BUILD_python -BUILD_redis -BUILD_sqlite -BUILD_nginx -BUILD_tesseract -BUILD_tessdata -BUILD_libwebp -BUILD_libjpegturbo -BUILD_libtiff -BUILD_jbig2dec -BUILD_jbig2enc -BUILD_leptonica -BUILD_ffmpeg -BUILD_imagemagick -BUILD_pngquant -BUILD_zbar -BUILD_unpaper -BUILD_libfontttf -BUILD_qpdf -BUILD_ghostscript -) - -_xparts=( -BUILD_ghostscript -) - -# list of versions -for p in "${_parts[@]}"; do - B=$(eval eval echo \\$\${p}) - echo -n "$B - " -done -echo - -# check versions -for p in "${_parts[@]}"; do - version $p -done - -# apptainer build --sandbox xxx docker://ghcr.io/paperless-ngx/paperless-ngx:latest -# grep PYTHON xxx/.singularity.d/env/10-docker2singularity.sh diff --git a/ffmpeg.build.sh b/ffmpeg.build.sh index ab1335f..ab918ab 100755 --- a/ffmpeg.build.sh +++ b/ffmpeg.build.sh @@ -26,7 +26,7 @@ EOF test -e "${BUILD_PKG}".tar.gz || wget -nv "${SRCURL}" -O "${BUILD_PKG}".tar.gz -test -d "${BUILD_PKG}" || mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}".tar.gz --strip-components=1 -C "${BUILD_PKG}" +test -d "${BUILD_PKG}" || { mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}".tar.gz --strip-components=1 -C "${BUILD_PKG}"; } cd "${BUILD_PKG}" diff --git a/ghostscript.build.sh b/ghostscript.build.sh index 9c43ee5..ff34fb1 100755 --- a/ghostscript.build.sh +++ b/ghostscript.build.sh @@ -28,7 +28,7 @@ EOF test -e "${BUILD_PKG}".tar.gz || wget -nv "${SRCURL}" -O "${BUILD_PKG}".tar.gz -test -d "${BUILD_PKG}" || mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}".tar.gz --strip-components=1 -C "${BUILD_PKG}" +test -d "${BUILD_PKG}" || { mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}".tar.gz --strip-components=1 -C "${BUILD_PKG}"; } cd "${BUILD_PKG}" diff --git a/leptonica.build.sh b/leptonica.build.sh index 841592a..573db24 100755 --- a/leptonica.build.sh +++ b/leptonica.build.sh @@ -28,7 +28,7 @@ LD_LIBRARY_PATH=${PREFIX}/lib\${LD_LIBRARY_PATH:+:}\${LD_LIBRARY_PATH:-} EOF test -e ${BUILD_PKG}.tar.gz || wget -nv ${SRCURL} -O ${BUILD_PKG}.tar.gz -test -d ${BUILD_PKG} || mkdir -pv ${BUILD_PKG} && tar -xf ${BUILD_PKG}.tar.gz --strip-components=1 -C ${BUILD_PKG} +test -d ${BUILD_PKG} || { mkdir -pv ${BUILD_PKG} && tar -xf ${BUILD_PKG}.tar.gz --strip-components=1 -C ${BUILD_PKG}; } cd ${BUILD_PKG} diff --git a/libfontttf.build.sh b/libfontttf.build.sh index 8546ade..08da260 100755 --- a/libfontttf.build.sh +++ b/libfontttf.build.sh @@ -19,7 +19,7 @@ PREFIX="${PREFIX}/${BUILD_PKG}" mkdir -p "${PREFIX}" test -e "${BUILD_PKG}.tar.gz" || wget -nv "${SRCURL}" -O "${BUILD_PKG}.tar.gz" -test -d "${BUILD_PKG}" || mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}.tar.gz" --strip-components=1 -C "${BUILD_PKG}" +test -d "${BUILD_PKG}" || { mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}.tar.gz" --strip-components=1 -C "${BUILD_PKG}"; } cd "${BUILD_PKG}" diff --git a/libjpegturbo.build.sh b/libjpegturbo.build.sh index e836b8f..6192190 100755 --- a/libjpegturbo.build.sh +++ b/libjpegturbo.build.sh @@ -24,7 +24,7 @@ EOF test -e "${BUILD_PKG}".tar.gz || wget -nv "${SRCURL}" -O "${BUILD_PKG}".tar.gz -test -d "${BUILD_PKG}" || mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}".tar.gz --strip-components=1 -C "${BUILD_PKG}" +test -d "${BUILD_PKG}" || { mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}".tar.gz --strip-components=1 -C "${BUILD_PKG}"; } pushd "${BUILD_PKG}" diff --git a/libtiff.build.sh b/libtiff.build.sh index 3b86eaf..824be2b 100755 --- a/libtiff.build.sh +++ b/libtiff.build.sh @@ -26,7 +26,7 @@ EOF test -e "${BUILD_PKG}".tar.gz || wget -nv "${SRCURL}" -O "${BUILD_PKG}".tar.gz -test -d "${BUILD_PKG}" || mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}".tar.gz --strip-components=1 -C "${BUILD_PKG}" +test -d "${BUILD_PKG}" || { mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}".tar.gz --strip-components=1 -C "${BUILD_PKG}"; } cd "${BUILD_PKG}" diff --git a/libwebp.build.sh b/libwebp.build.sh index 90d925c..1af37f0 100755 --- a/libwebp.build.sh +++ b/libwebp.build.sh @@ -24,7 +24,7 @@ PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig\${PKG_CONFIG_PATH:+:}\${PKG_CONFIG_PATH: EOF test -e "${BUILD_PKG}".tar.gz || wget -nv "${SRCURL}" -O "${BUILD_PKG}".tar.gz -test -d "${BUILD_PKG}" || mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}".tar.gz --strip-components=1 -C "${BUILD_PKG}" +test -d "${BUILD_PKG}" || { mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}".tar.gz --strip-components=1 -C "${BUILD_PKG}"; } cd "${BUILD_PKG}" diff --git a/nginx.build.sh b/nginx.build.sh index 1f7284d..39e9027 100755 --- a/nginx.build.sh +++ b/nginx.build.sh @@ -29,7 +29,7 @@ if [ -v NGINX_SSL_DHPARAM ]; then fi test -e ${BUILD_PKG}${BUILD_EXT} || wget -nv "${SRCURL}" -O ${BUILD_PKG}${BUILD_EXT} -test -d ${BUILD_PKG} || mkdir -pv ${BUILD_PKG} && tar -xf ${BUILD_PKG}${BUILD_EXT} --strip-components=1 -C ${BUILD_PKG} +test -d ${BUILD_PKG} || { mkdir -pv ${BUILD_PKG} && tar -xf ${BUILD_PKG}${BUILD_EXT} --strip-components=1 -C ${BUILD_PKG}; } pushd ${BUILD_PKG} diff --git a/node.build.sh b/node.build.sh index ee26fb1..aa6a655 100755 --- a/node.build.sh +++ b/node.build.sh @@ -27,7 +27,7 @@ npm_config_cache=${PREFIX}/cache EOF test -e "${BUILD_PKG}.tar.gz" || wget -nv "${SRCURL}" -O "${BUILD_PKG}.tar.gz" -test -d "${BUILD_PKG}" || mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}.tar.gz" --strip-components=1 -C "${BUILD_PKG}" +test -d "${BUILD_PKG}" || { mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}.tar.gz" --strip-components=1 -C "${BUILD_PKG}"; } . "${PREFIX}/profile" diff --git a/python.build.sh b/python.build.sh index 6685406..3c671cc 100755 --- a/python.build.sh +++ b/python.build.sh @@ -23,7 +23,7 @@ PATH="${PREFIX}"/bin:\${PATH} EOF test -e "${BUILD_PKG}.tar.gz" || wget -nv "${SRCURL}" -O "${BUILD_PKG}.tar.gz" -test -d "${BUILD_PKG}" || mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}.tar.gz" --strip-components=1 -C "${BUILD_PKG}" +test -d "${BUILD_PKG}" || { mkdir -pv "${BUILD_PKG}" && tar -xf "${BUILD_PKG}.tar.gz" --strip-components=1 -C "${BUILD_PKG}"; } pushd "${BUILD_PKG}" @@ -37,7 +37,7 @@ export PKG_CONFIG_PATH SQLITE_PREFIX=$(pkg-config sqlite3 --variable=prefix) -LDFLAGS="-Wl,-rpath -Wl,-L${SQLITE_PREFIX}/lib" \ +LDFLAGS="-Wl,-rpath,${SQLITE_PREFIX}/lib" \ ./configure "${_conf[@]}" make -j "${NPROC}" diff --git a/redis.build.sh b/redis.build.sh index 0c46112..507332d 100755 --- a/redis.build.sh +++ b/redis.build.sh @@ -23,7 +23,7 @@ REDISCLI_HISTFILE=${PREFIX}/.rediscli_history EOF test -e ${BUILD_PKG}.tar.gz || wget -nv ${SRCURL} -O ${BUILD_PKG}.tar.gz -test -d ${BUILD_PKG} || mkdir -pv ${BUILD_PKG} && tar -xf ${BUILD_PKG}.tar.gz --strip-components=1 -C ${BUILD_PKG} +test -d ${BUILD_PKG} || { mkdir -pv ${BUILD_PKG} && tar -xf ${BUILD_PKG}.tar.gz --strip-components=1 -C ${BUILD_PKG}; } pushd ${BUILD_PKG} diff --git a/sqlite.build.sh b/sqlite.build.sh index 82a9cc3..d19ba11 100755 --- a/sqlite.build.sh +++ b/sqlite.build.sh @@ -25,7 +25,7 @@ LD_LIBRARY_PATH=${PREFIX}/lib\${LD_LIBRARY_PATH:+:}\${LD_LIBRARY_PATH:-} EOF test -e ${BUILD_PKG}${BUILD_EXT} || wget -nv ${SRCURL} -O ${BUILD_PKG}${BUILD_EXT} -test -d ${BUILD_PKG} || mkdir -pv ${BUILD_PKG} && tar -xf ${BUILD_PKG}${BUILD_EXT} --strip-components=1 -C ${BUILD_PKG} +test -d ${BUILD_PKG} || { mkdir -pv ${BUILD_PKG} && tar -xf ${BUILD_PKG}${BUILD_EXT} --strip-components=1 -C ${BUILD_PKG}; } pushd ${BUILD_PKG} diff --git a/zbar.build.sh b/zbar.build.sh index 2c79771..a0ccd95 100755 --- a/zbar.build.sh +++ b/zbar.build.sh @@ -29,7 +29,7 @@ EOF test -e ${BUILD_PKG}${BUILD_EXT} || wget -nv ${SRCURL} -O ${BUILD_PKG}${BUILD_EXT} -test -d ${BUILD_PKG} || mkdir -pv ${BUILD_PKG} && tar -xf ${BUILD_PKG}${BUILD_EXT} --strip-components=1 -C ${BUILD_PKG} +test -d ${BUILD_PKG} || { mkdir -pv ${BUILD_PKG} && tar -xf ${BUILD_PKG}${BUILD_EXT} --strip-components=1 -C ${BUILD_PKG}; } cd ${BUILD_PKG} From 120aed53ec134e953d5fd7c3ea93efb61b2e10f9 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 10:42:57 +0200 Subject: [PATCH 02/21] remove unused asgi patch, fix cleanall.sh paperless-ngx.asgi.patch: disabled since 2.16.3 and broken as written (XXXASGIHandler, missing imports). The Django bug it backports (#36399, multiple Cookie headers joined with ",") is still present in 5.2.17, but cannot trigger here: nginx merges HTTP/2/3 cookie headers into a single Cookie header and proxies HTTP/1.1 to granian. cleanall.sh: build trees live in ${BUILD_DIR}, but the script removed relative to the cwd (and sourcing build.profile failed from BUILD_DIR), so it never removed anything. Now runs from the repo dir and pushd's into ${BUILD_DIR}. Add the missing libjpegturbo, libtiff, libwebp, ghostscript, imagemagick and qpdf; nginx stays excluded as before. Make it executable like all other scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- CLAUDE.md | 4 +- UPGRADE.md | 4 +- cleanall.sh | 29 +++++--- paperless-ngx.asgi.patch | 141 --------------------------------------- 4 files changed, 24 insertions(+), 154 deletions(-) mode change 100644 => 100755 cleanall.sh delete mode 100644 paperless-ngx.asgi.patch diff --git a/CLAUDE.md b/CLAUDE.md index 1e90d42..6c3fb34 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -75,7 +75,7 @@ PID/PGID files live in `$DEVSHM` (`/dev/shm/${USER}/${BUILD_TAG}`). Logs are in - **`nginx.chicken.patch`**: Renames nginx as "chicken" in HTTP headers, error pages, and version strings. Applied in `nginx.build.sh`. The Server response header advertises `chicken/1.32.0` instead of `nginx/...`. -- **`paperless-ngx.asgi.patch`**: Backports a fix for [Django ticket #36399](https://code.djangoproject.com/ticket/36399) into paperless-ngx's `asgi.py`. Has an `assert django.VERSION < (6,0)` guard so it will fail loudly when Django 6 ships and the upstream fix lands. **Currently NOT applied** — no script references it, and the patch is broken as written (`XXXASGIHandler`, missing imports). Open issue; see `UPGRADE.md`. +The former `paperless-ngx.asgi.patch` (backport of [Django ticket #36399](https://code.djangoproject.com/ticket/36399), multiple `Cookie` headers joined with `,`) was removed: it had been disabled since 2.16.3, and the bug cannot trigger here because nginx merges HTTP/2/3 cookie headers into a single `Cookie` header and talks HTTP/1.1 to granian. Django 5.2.x still has the bug, so only revisit this if granian is ever exposed directly. ## systemd @@ -105,7 +105,7 @@ Sources are downloaded from a private mirror at `beehive.molgen.mpg.de` (MD5 has ## Utility scripts - Version checks use the shared `../baremetals/checkversions.sh .` (run from this repo; compares `build.profile` pins against upstream). The old local `checkversions.sh` was removed. -- `cleanall.sh` — removes all build directories (not nginx) and logs; run from `${BUILD_DIR}` +- `cleanall.sh` — removes all component build trees (and with them their `.built` tags) and logs under `${BUILD_DIR}`, except nginx; cached source tarballs are kept. Run from the repo directory - `runifchanged.sh` — wraps `inotifywait` to re-execute a script when its file is saved (development helper) ## Management commands diff --git a/UPGRADE.md b/UPGRADE.md index c1074bd..c3d49dd 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -142,5 +142,5 @@ rm -rf ~/paperless-server/2.20.10 ~/paperless-server/2.20.15 - Bumping a tarball means running `md5repo.sh <upstream-url>`, then putting the printed beehive URL into `build.profile`. - Check versions with `../baremetals/checkversions.sh .` -- Open issue: `paperless-ngx.asgi.patch` is not applied by any script and is broken as written. - Either fix and apply it, or delete it. v3.3.1 still ships Django 5.2. +- `./cleanall.sh` (from the repo dir) wipes all component build trees except nginx. It's an alternative to + the `find … -name .built -delete` step, but it also throws away the extracted sources. diff --git a/cleanall.sh b/cleanall.sh old mode 100644 new mode 100755 index 5588e7a..b5d3a76 --- a/cleanall.sh +++ b/cleanall.sh @@ -3,26 +3,35 @@ set -e set -x set -u +# run from the repo directory; removes build trees (incl. their .built tags) +# and logs below ${BUILD_DIR}. cached source tarballs (*.tar*) are kept. + . build.profile declare -a RMRF RMRF=( + "${BUILD_sqlite}" + "${BUILD_redis}" + "${BUILD_python}" + "${BUILD_node}" + "${BUILD_libjpegturbo}" + "${BUILD_libtiff}" + "${BUILD_libwebp}" "${BUILD_ffmpeg}" "${BUILD_jbig2dec}" - "${BUILD_jbig2enc}" + "${BUILD_ghostscript}" + "${BUILD_imagemagick}" "${BUILD_leptonica}" - "${BUILD_libfontttf}" -# "${BUILD_nginx}" - "${BUILD_node}" - "${BUILD_paperless}" - "${BUILD_pngquant}" - "${BUILD_python}" - "${BUILD_redis}" - "${BUILD_sqlite}" + "${BUILD_jbig2enc}" "${BUILD_tesseract}" + "${BUILD_pngquant}" "${BUILD_unpaper}" "${BUILD_zbar}" + "${BUILD_libfontttf}" + "${BUILD_qpdf}" +# "${BUILD_nginx}" + "${BUILD_paperless}" ./logs ) @@ -35,4 +44,6 @@ function removeall { set -e } +pushd "${BUILD_DIR}" removeall +popd diff --git a/paperless-ngx.asgi.patch b/paperless-ngx.asgi.patch deleted file mode 100644 index 23fec7c..0000000 --- a/paperless-ngx.asgi.patch +++ /dev/null @@ -1,141 +0,0 @@ -diff --git a/src/paperless/asgi.py b/src/paperless/asgi.py -index 8d63c347a..45d8daa3a 100644 ---- a/src/paperless/asgi.py -+++ b/src/paperless/asgi.py -@@ -1,13 +1,134 @@ - import os - --from django.core.asgi import get_asgi_application -+# from django.core.asgi import get_asgi_application -+ -+import django -+from django.core.handlers.asgi import ASGIHandler -+ -+# https://code.djangoproject.com/ticket/36399 -+assert django.VERSION < (6,0), "Remove ASGIRequest backport." -+ -+class XXXASGIRequest(HttpRequest): -+ """ -+ Custom request subclass that decodes from an ASGI-standard request dict -+ and wraps request body handling. -+ """ -+ -+ # Number of seconds until a Request gives up on trying to read a request -+ # body and aborts. -+ body_receive_timeout = 60 -+ -+ def __init__(self, scope, body_file): -+ self.scope = scope -+ self._post_parse_error = False -+ self._read_started = False -+ self.resolver_match = None -+ self.path = scope["path"] -+ self.script_name = get_script_prefix(scope) -+ if self.script_name: -+ # TODO: Better is-prefix checking, slash handling? -+ self.path_info = scope["path"].removeprefix(self.script_name) -+ else: -+ self.path_info = scope["path"] -+ # HTTP basics. -+ self.method = self.scope["method"].upper() -+ # Ensure query string is encoded correctly. -+ query_string = self.scope.get("query_string", "") -+ if isinstance(query_string, bytes): -+ query_string = query_string.decode() -+ self.META = { -+ "REQUEST_METHOD": self.method, -+ "QUERY_STRING": query_string, -+ "SCRIPT_NAME": self.script_name, -+ "PATH_INFO": self.path_info, -+ # WSGI-expecting code will need these for a while -+ "wsgi.multithread": True, -+ "wsgi.multiprocess": True, -+ } -+ if self.scope.get("client"): -+ self.META["REMOTE_ADDR"] = self.scope["client"][0] -+ self.META["REMOTE_HOST"] = self.META["REMOTE_ADDR"] -+ self.META["REMOTE_PORT"] = self.scope["client"][1] -+ if self.scope.get("server"): -+ self.META["SERVER_NAME"] = self.scope["server"][0] -+ self.META["SERVER_PORT"] = str(self.scope["server"][1]) -+ else: -+ self.META["SERVER_NAME"] = "unknown" -+ self.META["SERVER_PORT"] = "0" -+ # Headers go into META. -+ for name, value in self.scope.get("headers", []): -+ name = name.decode("latin1") -+ if name == "content-length": -+ corrected_name = "CONTENT_LENGTH" -+ elif name == "content-type": -+ corrected_name = "CONTENT_TYPE" -+ else: -+ corrected_name = "HTTP_%s" % name.upper().replace("-", "_") -+ # HTTP/2 say only ASCII chars are allowed in headers, but decode -+ # latin1 just in case. -+ value = value.decode("latin1") -+ if corrected_name == "HTTP_COOKIE": -+ existing = self.META.get("HTTP_COOKIE") -+ if existing is not None: -+ value = existing + value -+ self.META["HTTP_COOKIE"] = value -+ else: -+ if corrected_name in self.META: -+ value = self.META[corrected_name] + "," + value -+ self.META[corrected_name] = value -+ # Pull out request encoding, if provided. -+ self._set_content_type_params(self.META) -+ # Directly assign the body file to be our stream. -+ self._stream = body_file -+ # Other bits. -+ self.resolver_match = None -+ -+ @cached_property -+ def GET(self): -+ return QueryDict(self.META["QUERY_STRING"]) -+ -+ def _get_scheme(self): -+ return self.scope.get("scheme") or super()._get_scheme() -+ -+ def _get_post(self): -+ if not hasattr(self, "_post"): -+ self._load_post_and_files() -+ return self._post -+ -+ def _set_post(self, post): -+ self._post = post -+ -+ def _get_files(self): -+ if not hasattr(self, "_files"): -+ self._load_post_and_files() -+ return self._files -+ -+ POST = property(_get_post, _set_post) -+ FILES = property(_get_files) -+ -+ @cached_property -+ def COOKIES(self): -+ return parse_cookie(self.META.get("HTTP_COOKIE", "")) -+ -+ def close(self): -+ super().close() -+ self._stream.close() -+ -+ -+class BackportASGIHandler(ASGIHandler): -+ request_class = XXXASGIHandler -+ - - # Fetch Django ASGI application early to ensure AppRegistry is populated - # before importing consumers and AuthMiddlewareStack that may import ORM - # models. - - os.environ.setdefault("DJANGO_SETTINGS_MODULE", "paperless.settings") --django_asgi_app = get_asgi_application() -+# django_asgi_app = get_asgi_application() -+ -+django.setup(set_prefix=False) -+ -+application = BackportASGIHandler() - - from channels.auth import AuthMiddlewareStack # noqa: E402 - from channels.routing import ProtocolTypeRouter # noqa: E402 From ffd9925d97fb69c627b6bcd6f43222d9b09d3262 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 10:44:01 +0200 Subject: [PATCH 03/21] build.profile: fix jbic2enc typo in BUILD_jbig2enc Only changes the build/install directory name, so jbig2enc gets rebuilt into ${PREFIX}/jbig2enc-0.30-20-g2f7f76b; profile.build.sh and paperless-ngx.build.sh reference it via ${BUILD_jbig2enc}. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- build.profile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.profile b/build.profile index 1ba0511..2bff0b3 100644 --- a/build.profile +++ b/build.profile @@ -89,7 +89,7 @@ BUILD_tessconfigs_SRCURL="https://github.com/tesseract-ocr/tessconfigs.git" # git clone / git describe #BUILD_jbig2enc=jbig2enc-0.30-16 -BUILD_jbig2enc=jbic2enc-0.30-20-g2f7f76b +BUILD_jbig2enc=jbig2enc-0.30-20-g2f7f76b BUILD_jbig2enc_SRCURL="https://github.com/agl/jbig2enc.git" BUILD_jbig2dec=jbig2dec-0.20 From 4f4e71369a9eecd655909ba800a7e9907519ed83 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 10:47:38 +0200 Subject: [PATCH 04/21] pin jbig2enc, pngquant and unpaper to their exact commits These were cloned at upstream HEAD; BUILD_<name> was only a label compared against "git describe" (jbig2enc/pngquant: warning only, unpaper: exit 1). So every host built whatever upstream had that day (jbig2enc HEAD is currently 70 commits past the pin). Now check out the <sha> after "-g" in BUILD_<name> right after clone/cache extraction, so stale cached .tar files from earlier HEAD clones are pinned too. pngquant also syncs its libimagequant submodule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- CLAUDE.md | 2 +- jbig2enc.build.sh | 11 +++-------- pngquant.build.sh | 11 ++++------- unpaper.build.sh | 11 +++-------- 4 files changed, 11 insertions(+), 24 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 6c3fb34..f4ca13d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -100,7 +100,7 @@ The build order in `buildall.sh` respects these dependencies (see `README.dot` f ## Source download strategy -Sources are downloaded from a private mirror at `beehive.molgen.mpg.de` (MD5 hash in the URL path) rather than upstream directly. Git-based components (imagemagick, pngquant, unpaper, tesseract, jbig2enc, jbig2dec, qpdf) are cloned from GitHub, then cached as `.tar` archives in `${BUILD_DIR}` to speed up rebuilds. +Sources are downloaded from a private mirror at `beehive.molgen.mpg.de` (MD5 hash in the URL path) rather than upstream directly. Git-based components (imagemagick, pngquant, unpaper, tesseract, jbig2enc, jbig2dec, qpdf) are cloned from GitHub, then cached as `.tar` archives in `${BUILD_DIR}` to speed up rebuilds. Tagged ones clone `--branch <tag>`; the snapshot ones (jbig2enc, pngquant, unpaper) use a `git describe`-style pin like `unpaper-7.0.0-127-g5bcef8a` and the build script checks out the `<sha>` after `-g`, so to bump one put the new `git describe --tags` output into `BUILD_<name>`. ## Utility scripts diff --git a/jbig2enc.build.sh b/jbig2enc.build.sh index c589031..f49420a 100755 --- a/jbig2enc.build.sh +++ b/jbig2enc.build.sh @@ -26,16 +26,11 @@ fi if [[ ! -d ${BUILD_PKG} ]]; then git clone "${SRCURL}" --single-branch "${BUILD_PKG}" - # check version - pushd "${BUILD_PKG}" - GITVER=$(git describe --tags) - popd - - if [ "${GITVER}" != "${BUILD_PKG#*-}" ]; then - echo "${BUILD_PKG} should be updated to ${GITVER} update!" - fi fi +# pin to the exact commit encoded in BUILD_jbig2enc (...-g<sha>) +git -C "${BUILD_PKG}" checkout --detach "${BUILD_PKG##*-g}" + if [[ ! -e ${BUILD_PKG}.tar ]]; then tar cf "${BUILD_PKG}".tar "${BUILD_PKG}" fi diff --git a/pngquant.build.sh b/pngquant.build.sh index 1d56950..a64e58a 100755 --- a/pngquant.build.sh +++ b/pngquant.build.sh @@ -20,15 +20,12 @@ fi if [[ ! -d ${BUILD_PKG} ]]; then git clone --recursive https://github.com/kornelski/pngquant.git --single-branch ${BUILD_PKG} - # check version - pushd ${BUILD_PKG} - GITVER=pngquant-$(git describe --tags) - popd - if [ "${GITVER}" != "${BUILD_PKG}" ]; then - echo "${BUILD_PKG} should be updated to ${GITVER} update!" - fi fi +# pin to the exact commit encoded in BUILD_pngquant (...-g<sha>), incl. the libimagequant submodule +git -C "${BUILD_PKG}" checkout --detach "${BUILD_PKG##*-g}" +git -C "${BUILD_PKG}" submodule update --init --recursive + if [[ ! -e "${BUILD_PKG}".tar ]]; then tar cf "${BUILD_PKG}".tar "${BUILD_PKG}" fi diff --git a/unpaper.build.sh b/unpaper.build.sh index 5053d22..2ad08d0 100755 --- a/unpaper.build.sh +++ b/unpaper.build.sh @@ -23,16 +23,11 @@ fi if [[ ! -d ${BUILD_PKG} ]]; then git clone "${SRCURL}" --single-branch "${BUILD_PKG}" - # check version - pushd "${BUILD_PKG}" - GITVER=$(git describe --tags) - popd - if [ "${GITVER}" != "${BUILD_PKG}" ]; then - echo "${BUILD_PKG} should be updated to ${GITVER} update!" - exit 1 - fi fi +# pin to the exact commit encoded in BUILD_unpaper (...-g<sha>) +git -C "${BUILD_PKG}" checkout --detach "${BUILD_PKG##*-g}" + if [[ ! -e ${BUILD_PKG}.tar ]]; then tar cf "${BUILD_PKG}".tar "${BUILD_PKG}" fi From e8708faaa182d633262c40bb903b84023497595c Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 10:35:48 +0200 Subject: [PATCH 05/21] update to paperless-ngx 3.3.1 Phase 2 of the upgrade (requires 2.20.15 to have run once, see UPGRADE.md). - drop zbar: v3 removed pyzbar in favour of zxing-cpp wheels - drop NLTK download and PAPERLESS_NLTK_DIR: NLTK is gone in v3 - drop pipenv (unused since paperless moved to uv) - uv pip install from the PyTorch CPU index like upstream's Dockerfile, torch is now a hard dependency - wrappers: remove decrypt_documents (gone), add document_llmindex - startstop: after migrate run "document_index reindex --if-needed" (Whoosh -> Tantivy) and "document_llmindex migrate", as upstream's bare-metal update docs require; both are no-ops when current - conf: PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER=X-Real-IP so allauth's login rate limiting sees the real client IP behind nginx Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- README.dot | 3 +- build.profile | 8 ++--- buildall.sh | 1 - cleanall.sh | 1 - paperless-ngx.build.sh | 23 +++++--------- paperless.conf.build.sh | 3 +- profile.build.sh | 1 - python.build.sh | 1 - startstop.build.sh | 4 +++ zbar.build.sh | 66 ----------------------------------------- 10 files changed, 17 insertions(+), 94 deletions(-) delete mode 100755 zbar.build.sh diff --git a/README.dot b/README.dot index edbd24e..b7847df 100644 --- a/README.dot +++ b/README.dot @@ -7,7 +7,6 @@ digraph depends { FF [ label="ffmpeg" ] PN [ label="pngquant" ] UN [ label="unpaper" ] - ZB [ label="zbar" ] PL [ label="paperless" ] LE [ label="leptonica" ] TE [ label="tesseract" ] @@ -39,7 +38,7 @@ digraph depends { { SQ, PY } -> NG - { PY, SQ, NO, FF, PN, ZB, UN, LE, TE, JD, JE, NG } -> PL + { PY, SQ, NO, FF, PN, UN, LE, TE, JD, JE, NG } -> PL } diff --git a/build.profile b/build.profile index 2bff0b3..bbc60af 100644 --- a/build.profile +++ b/build.profile @@ -1,6 +1,6 @@ -BUILD_TAG=2.20.15 +BUILD_TAG=3.3.1 -BUILD_paperless=paperless-ngx-2.20.15 +BUILD_paperless=paperless-ngx-3.3.1 BUILD_DIR=build BUILD_SRC="${PWD}" @@ -37,10 +37,6 @@ BUILD_node_SRCURL="https://beehive.molgen.mpg.de/a0d4131e52d9055f8428439542018d4 BUILD_ffmpeg=ffmpeg-8.0.3 BUILD_ffmpeg_SRCURL="https://beehive.molgen.mpg.de/3057126c124c8515ff7122a6d41d609b/${BUILD_ffmpeg}.tar.gz" -# SRCURL="https://github.com/mchehab/zbar/archive/refs/tags/0.23.93.tar.gz" -BUILD_zbar=zbar-0.23.93 -BUILD_zbar_SRCURL="https://beehive.molgen.mpg.de/3f69d17f6495de023b59b3539ce5e605/${BUILD_zbar}.tar.gz" - # wget "https://github.com/DanBloomberg/leptonica/releases/download/1.87.0/leptonica-1.87.0.tar.gz" BUILD_leptonica=leptonica-1.87.0 BUILD_leptonica_SRCURL="https://beehive.molgen.mpg.de/d523d2c883b71850a46af2517d8915bf/${BUILD_leptonica}.tar.gz" diff --git a/buildall.sh b/buildall.sh index 294313c..39e29af 100755 --- a/buildall.sh +++ b/buildall.sh @@ -85,7 +85,6 @@ _pkgs=( tesseract pngquant unpaper - zbar libfontttf qpdf nginx diff --git a/cleanall.sh b/cleanall.sh index b5d3a76..b84c998 100755 --- a/cleanall.sh +++ b/cleanall.sh @@ -27,7 +27,6 @@ RMRF=( "${BUILD_tesseract}" "${BUILD_pngquant}" "${BUILD_unpaper}" - "${BUILD_zbar}" "${BUILD_libfontttf}" "${BUILD_qpdf}" # "${BUILD_nginx}" diff --git a/paperless-ngx.build.sh b/paperless-ngx.build.sh index 27e492e..2904fa3 100755 --- a/paperless-ngx.build.sh +++ b/paperless-ngx.build.sh @@ -13,7 +13,6 @@ set -u . ${PREFIX}/${BUILD_unpaper}/profile . ${PREFIX}/${BUILD_sqlite}/profile . ${PREFIX}/${BUILD_imagemagick}/profile -. ${PREFIX}/${BUILD_zbar}/profile . ${PREFIX}/${BUILD_python}/profile . ${PREFIX}/${BUILD_node}/profile . ${PREFIX}/${BUILD_leptonica}/profile @@ -69,8 +68,6 @@ pushd $PREFIX pip install uv -pipenv install - pushd src-ui npm update -g pnpm npm install -g corepack@latest @@ -86,7 +83,12 @@ pushd src echo "Installing Python requirements" uv export --quiet --no-dev --all-extras --format requirements-txt --output-file requirements.txt -uv pip install --system --no-python-downloads --python-preference system --requirements requirements.txt +# torch must come from the CPU-only index (as in upstream's Dockerfile), otherwise PyPI's CUDA build is pulled +uv pip install --system --no-python-downloads --python-preference system \ + --index https://pypi.org/simple \ + --index https://download.pytorch.org/whl/cpu \ + --index-strategy unsafe-best-match \ + --requirements requirements.txt ./manage.py collectstatic --clear --no-input --link @@ -101,13 +103,13 @@ PAPERLESS_ADMIN_MAIL=${PAPERLESS_ADMIN_MAIL:-admin@localhost.lan} ./manage.py shell -c "from django.contrib.auth.models import User; User.objects.create_superuser('${PAPERLESS_ADMIN_USER}', '${PAPERLESS_ADMIN_MAIL}', '${PAPERLESS_ADMIN_PASSWORD}')" || true -for command in decrypt_documents \ - document_archiver \ +for command in document_archiver \ document_exporter \ document_importer \ mail_fetcher \ document_create_classifier \ document_index \ + document_llmindex \ document_renamer \ document_retagger \ document_thumbnails \ @@ -129,15 +131,6 @@ done popd -echo "Installing NLTK data" - -mkdir -pv ${PROJECT}/nltk - -python3 -W ignore::RuntimeWarning -m nltk.downloader -d "${PROJECT}/nltk" snowball_data -python3 -W ignore::RuntimeWarning -m nltk.downloader -d "${PROJECT}/nltk" stopwords -python3 -W ignore::RuntimeWarning -m nltk.downloader -d "${PROJECT}/nltk" punkt -python3 -W ignore::RuntimeWarning -m nltk.downloader -d "${PROJECT}/nltk" punkt_tab - popd } diff --git a/paperless.conf.build.sh b/paperless.conf.build.sh index 0275b9b..f73dfbf 100755 --- a/paperless.conf.build.sh +++ b/paperless.conf.build.sh @@ -26,13 +26,14 @@ PAPERLESS_CONSUMPTION_DIR=${PROJECT}/consume PAPERLESS_DATA_DIR=${PROJECT}/data PAPERLESS_LOGGING_DIR=${LOGDIR} PAPERLESS_MEDIA_ROOT=${PROJECT}/media -PAPERLESS_NLTK_DIR=${PROJECT}/nltk PAPERLESS_SCRATCH_DIR=${TMPDIR} PAPERLESS_STATICDIR=${PREFIX}/static PAPERLESS_URL=${PAPERLESS_URL} PAPERLESS_CSRF_TRUSTED_ORIGINS=${PAPERLESS_CSRF_TRUSTED_ORIGINS:-${PAPERLESS_URL}} PAPERLESS_COOKIE_PREFIX=ngx-${BUILD_TAG}- +# nginx sets X-Real-IP; granian is only reachable via its unix socket, so the header is trustworthy +PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER=X-Real-IP PAPERLESS_OCR_MODE=${PAPERLESS_OCR_MODE:-force} PAPERLESS_OCR_DESKEW=false diff --git a/profile.build.sh b/profile.build.sh index 55ec44e..41bc786 100755 --- a/profile.build.sh +++ b/profile.build.sh @@ -12,7 +12,6 @@ BUILD_TAG=${BUILD_TAG} PATH=$HOME/bin:${PROJECT}/bin:/bin:/sbin:/usr/bin:/usr/sbin -. ${PROJECT}/${BUILD_zbar}/profile . ${PROJECT}/${BUILD_jbig2enc}/profile . ${PROJECT}/${BUILD_jbig2dec}/profile . ${PROJECT}/${BUILD_sqlite}/profile diff --git a/python.build.sh b/python.build.sh index 3c671cc..aa72432 100755 --- a/python.build.sh +++ b/python.build.sh @@ -52,7 +52,6 @@ ln -vfs pip3 "${PREFIX}/bin/pip" . "${PREFIX}/profile" pip3 install --prefix="${PREFIX}" --upgrade pip -python3 -m pip install --no-cache-dir --upgrade pipenv==2024.0.1 # for unpaper pip3 install meson # for paperless-ngx diff --git a/startstop.build.sh b/startstop.build.sh index 1c2e349..5063c43 100755 --- a/startstop.build.sh +++ b/startstop.build.sh @@ -203,6 +203,10 @@ srv_migrate() { ./manage.py migrate "\${_opts[@]}" fi + # both are no-ops when already current (first v3 start: Whoosh -> Tantivy rebuild) + ./manage.py document_index reindex --if-needed --no-progress-bar + ./manage.py document_llmindex migrate + popd } diff --git a/zbar.build.sh b/zbar.build.sh deleted file mode 100755 index a0ccd95..0000000 --- a/zbar.build.sh +++ /dev/null @@ -1,66 +0,0 @@ -#!/bin/bash -{ -set -x -set -e -set -u - -. build.profile - -. ${PREFIX}/${BUILD_libjpegturbo}/profile -. ${PREFIX}/${BUILD_imagemagick}/profile - - -function B_ZBAR { - -SRCURL=${BUILD_zbar_SRCURL} - -BUILD_PKG=${BUILD_zbar} -BUILD_EXT=${SRCURL/${SRCURL%.*.*}/} - -PREFIX=${PREFIX}/${BUILD_PKG} - -mkdir -p $PREFIX - -cat >${PREFIX}/profile <<-EOF -PATH=${PREFIX}/bin:\$PATH -PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig\${PKG_CONFIG_PATH:+:}\${PKG_CONFIG_PATH:-} -LD_LIBRARY_PATH=${PREFIX}/lib\${LD_LIBRARY_PATH:+:}\${LD_LIBRARY_PATH:-} -EOF - - -test -e ${BUILD_PKG}${BUILD_EXT} || wget -nv ${SRCURL} -O ${BUILD_PKG}${BUILD_EXT} -test -d ${BUILD_PKG} || { mkdir -pv ${BUILD_PKG} && tar -xf ${BUILD_PKG}${BUILD_EXT} --strip-components=1 -C ${BUILD_PKG}; } - -cd ${BUILD_PKG} - -autoreconf -vfi - -local _conf=( - --prefix ${PREFIX} - --disable-video - --disable-nls - --disable-doc - --enable-introspection=no - --with-gir=no - --with-x=no - --with-qt=no - --with-gtk=no - --with-dbus=no - --with-python=no - --with-xshm=no -) - -./configure "${_conf[@]}" - -make -j $NPROC -make install - -} - -pushd "${BUILD_DIR}" - -B_ZBAR - -popd - -} From 106800e2b5cdf7b050d7f6789bf1ce5388806f57 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 11:20:14 +0200 Subject: [PATCH 06/21] buildall: propagate build failures as non-zero exit status A failing component printed "<pkg>.build.sh failed exit code N" but buildall.sh still exited 0: the bare "exit" took echo's status, and the whole body runs in { ... } | tee, whose status is tee's. A failing paperless-ngx.build.sh or generator script was masked the same way. Add pipefail, capture the component's status explicitly (so the failure message is still printed under set -e) and exit with it. Tested with stub component scripts: all-ok -> 0 (unchanged), failing component -> its exit code, failing paperless-ngx.build.sh -> its code (old version: 0 in both failure cases). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- buildall.sh | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/buildall.sh b/buildall.sh index 39e29af..b14493c 100755 --- a/buildall.sh +++ b/buildall.sh @@ -1,6 +1,7 @@ #!/bin/bash set -e set -u +set -o pipefail BUILD_LOCAL=build.local if [ ! -e ${BUILD_LOCAL} ]; then @@ -94,13 +95,14 @@ build() { local _pkg=$1 eval "_tag=${BUILD_DIR}/\$BUILD_${_pkg}/.built" if [[ ! -e "$_tag" ]]; then - ./${_pkg}.build.sh 2>&1 | tee "${LOGS}/${_pkg}.log"; - r=${PIPESTATUS[0]} + local r=0 + # pipefail: r is the build script's status (tee itself succeeds) + ./${_pkg}.build.sh 2>&1 | tee "${LOGS}/${_pkg}.log" || r=$? if [[ $r == 0 ]]; then touch "$_tag" else echo "${_pkg}.build.sh failed exit code $r" - exit + exit "$r" fi fi } From c718254a436443504958cb061f44282d012dbfdd Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 11:21:49 +0200 Subject: [PATCH 07/21] UPGRADE.md: hop 2 uses tag paperless-3.3.1-2 paperless-3.3.1 is already published; the buildall exit-code fix goes out as a new tag instead of moving the old one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- UPGRADE.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/UPGRADE.md b/UPGRADE.md index c3d49dd..dea57bd 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -11,7 +11,11 @@ So every host goes through two hops. Each hop is a git tag: | Tag | `BUILD_TAG` | What it is | |---|---|---| | `paperless-2.20.15` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 | -| `paperless-3.3.1` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | +| `paperless-3.3.1-2` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | + +`paperless-3.3.1-2` supersedes `paperless-3.3.1`. It builds the same software; the only change is that `buildall.sh` +now exits non-zero when a component fails (before, it always reported 0). On `paperless-2.20.15`, and on the old +`paperless-3.3.1`, a successful-looking `./buildall.sh` doesn't prove anything, so always check the logs. Do **not** skip hop 1. 2.20.15 must start once, so its DB migrations get applied, before you build 3.3.1. @@ -89,7 +93,7 @@ Read the "Breaking changes" section below first, especially if you use custom pr ```bash cd <path>/paperless-baremetal git fetch --tags origin -git checkout paperless-3.3.1 +git checkout paperless-3.3.1-2 find build/ -mindepth 2 -maxdepth 2 -name .built -delete ./buildall.sh ``` From d9bafa24b3778a23d54a8f454b631825e0696994 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 12:22:17 +0200 Subject: [PATCH 08/21] fix build/test failures found by a full hop-1 test build - libjpegturbo 3.2.0: build with -fPIC (as bigbluebutton-baremetal does). 3.2.0 bundles spng+zlib as static libs linked into libturbojpeg.so; without PIC the link fails with "relocation R_X86_64_32 ... can not be used when making a shared object". - nginx: the chicken patch no longer applied to 1.31.6 (nginx.h hunk rejected, and the Server header was never patched), hidden by "|| true" -> responses said "Server: nginx". Take immich's patch (also patches ngx_http_header_filter_module.c) and toss's handling: apply unless already applied, fail on rejects, and verify "nginx -v" reports chicken/ after install. - paperless-online-test.pdf lacked /Type /Catalog; ghostscript in the strict mode ocrmypdf uses (-dPDFSTOPONERROR) rejects it with any version, so the consume test in UPGRADE.md always failed. Rewritten as a valid PDF with qpdf (same "Hello World!" content). Verified on a fresh host: full build, all services up, consume test OCRs "Hello World!" with archive + thumbnail, Server: chicken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- libjpegturbo.build.sh | 1 + nginx.build.sh | 9 ++++++- nginx.chicken.patch | 54 ++++++++++++++++++++++---------------- paperless-online-test.pdf | Bin 896 -> 657 bytes 4 files changed, 41 insertions(+), 23 deletions(-) diff --git a/libjpegturbo.build.sh b/libjpegturbo.build.sh index 6192190..60e101f 100755 --- a/libjpegturbo.build.sh +++ b/libjpegturbo.build.sh @@ -33,6 +33,7 @@ local _conf=( -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_COLOR_MAKEFILE=OFF -DCMAKE_VERBOSE_MAKEFILE=ON + -DCMAKE_C_FLAGS='-O2 -fPIC' ) rm -rf b diff --git a/nginx.build.sh b/nginx.build.sh index 39e9027..4585044 100755 --- a/nginx.build.sh +++ b/nginx.build.sh @@ -37,7 +37,11 @@ pushd ${BUILD_PKG} # What you see here is the representation of the string "nginx" compressed with HPACK's Huffman encoding. # ~/go/src/github.com/wwwutz/hpack -patch -p1 --silent --backup --forward --input=${BUILD_SRC}/nginx.chicken.patch || true +# The patch carries nginx_version/NGINX_VERSION as context lines: check it on +# every nginx bump. Idempotent (already-applied is fine), anything else fails. +if ! patch -p1 --dry-run --silent --reverse --input=${BUILD_SRC}/nginx.chicken.patch >/dev/null; then + patch -p1 --silent --forward --input=${BUILD_SRC}/nginx.chicken.patch +fi local _conf=( --prefix="${PREFIX}" @@ -63,6 +67,9 @@ CFLAGS='-O2 -fPIC' \ make -j "${NPROC}" install +# a rejected/missing patch hunk must not go unnoticed +"${PREFIX}"/sbin/nginx -v 2>&1 | grep -q "^nginx version: chicken/" || { echo "chicken patch not effective"; exit 1; } + popd } diff --git a/nginx.chicken.patch b/nginx.chicken.patch index e05d2b2..10cf1b7 100644 --- a/nginx.chicken.patch +++ b/nginx.chicken.patch @@ -19,7 +19,7 @@ index a57c2f9..6ba724b 100644 </body> </html> diff --git a/html/index.html b/html/index.html -index 75bcddc..b6668f7 100644 +index e8f5622..b6668f7 100644 --- a/html/index.html +++ b/html/index.html @@ -1,7 +1,7 @@ @@ -41,7 +41,7 @@ index 75bcddc..b6668f7 100644 -API gateway, load balancer, content cache, or other features.</p> +<h1>Chicken!</h1> +<p>Bwok Puk bok pukak bwak-bwak bwok Cluck bwak-bwak waaaaak bwwwaak-bwwwaaaaak.</p> - + -<p>For online documentation and support please refer to -<a href="https://nginx.org/">nginx.org</a>.<br/> -To engage with the community please visit @@ -50,35 +50,32 @@ index 75bcddc..b6668f7 100644 -security features and capabilities please refer to -<a href="https://f5.com/nginx">f5.com/nginx</a>.</p> +<p>Cluckity... Cluckity Puk-puk Waaaaak cock-a-doodle-doo bwak-bwak-bwak cluckity bwwwaak! Cluckity bok-bok Bok bwok-bwok-bwok bwak-bwak-bwak cluck. Cluck wak-waaaak pukaaaak-pukaaaak-pukaaaaak Puk-puk-puk Puk Gobble-gobble bok Pukaaaaaaak-pukaaaaak bwwwak-bwwwak-bwwwaaaaak Puk bwak bwak-bwak waak-waaaaaaak cluck. Pukak Bwok?</p> - + -<p><em>Thank you for using nginx.</em></p> +<p><em>Cluck-cluck-cluck gobble-gobble-gobble honk Gobble-gobble bwwwaaaaaak.</em></p> </body> </html> diff --git a/src/core/nginx.h b/src/core/nginx.h -index f4ccc61..8ff917c 100644 +index 088aa1e..0e1632f 100644 --- a/src/core/nginx.h +++ b/src/core/nginx.h -@@ -9,9 +9,9 @@ - #define _NGINX_H_INCLUDED_ - - --#define nginx_version 1031000 --#define NGINX_VERSION "1.31.0" +@@ -10,8 +10,8 @@ + + + #define nginx_version 1031006 + #define NGINX_VERSION "1.31.6" -#define NGINX_VER "nginx/" NGINX_VERSION -+#define nginx_version 1032000 -+#define NGINX_VERSION "1.32.0" +#define NGINX_VER "chicken/" NGINX_VERSION - + #ifdef NGX_BUILD #define NGINX_VER_BUILD NGINX_VER " (" NGX_BUILD ")" diff --git a/src/http/ngx_http_special_response.c b/src/http/ngx_http_special_response.c -index 2984ffe..cc40bc6 100644 +index eaf42e3..fd682a3 100644 --- a/src/http/ngx_http_special_response.c +++ b/src/http/ngx_http_special_response.c @@ -33,7 +33,7 @@ static u_char ngx_http_error_build_tail[] = - - + + static u_char ngx_http_error_tail[] = -"<hr><center>nginx</center>" CRLF +"<hr><center>chicken</center>" CRLF @@ -92,33 +89,46 @@ index 6b73b1e..a7b8094 100644 @@ -120,7 +120,7 @@ ngx_http_v2_header_filter(ngx_http_request_t *r) ngx_http_core_srv_conf_t *cscf; u_char addr[NGX_SOCKADDR_STRLEN]; - + - static const u_char nginx[5] = { 0x84, 0xaa, 0x63, 0x55, 0xe7 }; + static const u_char nginx[6] = { 0x85, 0x24, 0xe6, 0x27, 0x52, 0xd5 }; #if (NGX_HTTP_GZIP) static const u_char accept_encoding[12] = { 0x8b, 0x84, 0x84, 0x2d, 0x69, 0x5b, 0x05, 0x44, 0x3c, 0x86, 0xaa, 0x6f diff --git a/src/http/v3/ngx_http_v3_filter_module.c b/src/http/v3/ngx_http_v3_filter_module.c -index e3f1536..ff97b0f 100644 +index 4d2276d..7b156a6 100644 --- a/src/http/v3/ngx_http_v3_filter_module.c +++ b/src/http/v3/ngx_http_v3_filter_module.c @@ -169,7 +169,7 @@ ngx_http_v3_header_filter(ngx_http_request_t *r) n = sizeof(NGINX_VER_BUILD) - 1; - + } else { - n = sizeof("nginx") - 1; + n = sizeof("chicken") - 1; } - + len += ngx_http_v3_encode_field_lri(NULL, 0, @@ -352,8 +352,8 @@ ngx_http_v3_header_filter(ngx_http_request_t *r) n = sizeof(NGINX_VER_BUILD) - 1; - + } else { - p = (u_char *) "nginx"; - n = sizeof("nginx") - 1; + p = (u_char *) "chicken"; + n = sizeof("chicken") - 1; } - + ngx_log_debug2(NGX_LOG_DEBUG_HTTP, c->log, 0, +diff --git a/src/http/ngx_http_header_filter_module.c b/src/http/ngx_http_header_filter_module.c +index 7899383..b8fcb8f 100644 +--- a/src/http/ngx_http_header_filter_module.c ++++ b/src/http/ngx_http_header_filter_module.c +@@ -47,7 +47,7 @@ ngx_module_t ngx_http_header_filter_module = { + }; + + +-static u_char ngx_http_server_string[] = "Server: nginx" CRLF; ++static u_char ngx_http_server_string[] = "Server: chicken" CRLF; + static u_char ngx_http_server_full_string[] = "Server: " NGINX_VER CRLF; + static u_char ngx_http_server_build_string[] = "Server: " NGINX_VER_BUILD CRLF; + diff --git a/paperless-online-test.pdf b/paperless-online-test.pdf index 9428352a2902822625e6126ae419f186a9302280..05428173734a8d0ed03ced590f00ba5bf6d51034 100644 GIT binary patch literal 657 zcmah`%TD7k6y>o&<^$N?O(g`HN$kWGm8wVsr4j-}RM;UFPID=NOg!VbqOf9tkl677 zumL`S4J&?uUqIsV3G5KI+fp=}aU$6|_nzx>jw^Aa`GfwXr1J27{Xr=BAd`JkuY(th z4wk?S(gJVm=nsOoB(%t72cn2zk=jOE9uk&hx6nYrTg}oEwjprTogukDZ$!1_5w*}* z=n}&Ib?caBVo~<Y6yJPR${?&UFtcKzOyf{i3rpEmZKKmFZc0;e2kN#(QKG}VUn&zh zc$?isJ0sSP5!!d<TB69fvi|_IYQOokx`txcu;}PL;DNa@%QY&q=R#ux+cI^eQY#d{ zN$;{GPp$azFxc|LFrHeOR3~@m;}=&G=|S)I_4#@J^lj$lc6NX6H2r&jHa12Mr*BNK z1NtE1p2A(?11J91f-2_$hh6x>C<j-CVuV$VXuG~)l$q53!C0+2!n6`uZXDSZTT<HR zbnm~J9V2k?RvNHfZ`Uwjw4EfWk9<Ke<cK^(MiUkaO8w~n9XmGLQVXS>y^z*OrLw%y FB%i8Bxqkov literal 896 zcmah|%}T>S5WeqIjE8_8G`npRDU^b(4T6fL4R{a_soB<)n3c^IsxRrmH_#VxW}9?l zP&${)&iBp##G775r{YX7@9X>HlQ97t)MCv5WiSALnyh3=4}cCrGYbrdOcp887#W-V z>nyG4@X#fPN9%#^e*E?-Z6MTb*gY~<o5JiM(M~`?nhB<fmKE6*MB&G(Fo;n0-o8=- zZe^M!Beg~Sfg=)%4)ppRSJ+IJYNMa1Pii^BKy_*8(-O<rRoOy*J&86-N=d=MztCBd z{~yzN`t!}g*g4s89nOQUZm<VRuhQW5T*8+2H|DG^SGC0Ux@h#xb=}BfWu9%8fY;>G zXqmjQQH%yZ5+DSJc!>}E_BiS(>c>|y&lTJ&ou@}95U<%Jrn9S@>jT_sxnvvy#`gkY z5Of1r!rmrAuqYJH7GkL+4Vwt@{9Cr5k7rifn9)g=OKlsW70;Ba4j1Up+5H~atv}D+ Y>thcr8)sOWL>pW0IS-iUP3|K01KJI^NdN!< From 9a65207e1625f08cd458eee8590ba426e9c168af Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 12:24:23 +0200 Subject: [PATCH 09/21] UPGRADE.md: use tags paperless-2.20.15-2 / paperless-3.3.1-3; CLAUDE.md: chicken patch handling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- CLAUDE.md | 2 +- UPGRADE.md | 18 +++++++++++------- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f4ca13d..e1faeb4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -73,7 +73,7 @@ PID/PGID files live in `$DEVSHM` (`/dev/shm/${USER}/${BUILD_TAG}`). Logs are in ## Key patches applied during build -- **`nginx.chicken.patch`**: Renames nginx as "chicken" in HTTP headers, error pages, and version strings. Applied in `nginx.build.sh`. The Server response header advertises `chicken/1.32.0` instead of `nginx/...`. +- **`nginx.chicken.patch`**: Renames nginx as "chicken" in HTTP headers, error pages, and version strings. Applied in `nginx.build.sh` (only if not already applied; a rejected hunk fails the build, and after install `nginx -v` must report `chicken/`). The Server response header says `chicken`. The patch carries the version lines of `src/core/nginx.h` as context, so check that it still applies on every nginx bump. The same patch is shared with immich/toss-baremetal. The former `paperless-ngx.asgi.patch` (backport of [Django ticket #36399](https://code.djangoproject.com/ticket/36399), multiple `Cookie` headers joined with `,`) was removed: it had been disabled since 2.16.3, and the bug cannot trigger here because nginx merges HTTP/2/3 cookie headers into a single `Cookie` header and talks HTTP/1.1 to granian. Django 5.2.x still has the bug, so only revisit this if granian is ever exposed directly. diff --git a/UPGRADE.md b/UPGRADE.md index dea57bd..c237972 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -10,12 +10,16 @@ So every host goes through two hops. Each hop is a git tag: | Tag | `BUILD_TAG` | What it is | |---|---|---| -| `paperless-2.20.15` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 | -| `paperless-3.3.1-2` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | +| `paperless-2.20.15-2` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 | +| `paperless-3.3.1-3` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | -`paperless-3.3.1-2` supersedes `paperless-3.3.1`. It builds the same software; the only change is that `buildall.sh` -now exits non-zero when a component fails (before, it always reported 0). On `paperless-2.20.15`, and on the old -`paperless-3.3.1`, a successful-looking `./buildall.sh` doesn't prove anything, so always check the logs. +**Use only these two tags.** The older `paperless-2.20.15`, `paperless-3.3.1` and `paperless-3.3.1-2` are superseded: +- libjpeg-turbo 3.2.0 fails to build on them. +- `buildall.sh` reports success even when a component failed (except on `-2`). +- nginx shows `Server: nginx` instead of chicken. +- The consume-test PDF is malformed. + +All of this was found and fixed by a full test build of hop 1. Do **not** skip hop 1. 2.20.15 must start once, so its DB migrations get applied, before you build 3.3.1. @@ -44,7 +48,7 @@ Do **not** skip hop 1. 2.20.15 must start once, so its DB migrations get applied ```bash cd <path>/paperless-baremetal git fetch --tags origin # or whichever remote this host pulls from -git checkout paperless-2.20.15 +git checkout paperless-2.20.15-2 # BUILD_TAG changed, so remove all .built tags. Otherwise components whose # version did not change get skipped and are missing from the new prefix. @@ -93,7 +97,7 @@ Read the "Breaking changes" section below first, especially if you use custom pr ```bash cd <path>/paperless-baremetal git fetch --tags origin -git checkout paperless-3.3.1-2 +git checkout paperless-3.3.1-3 find build/ -mindepth 2 -maxdepth 2 -name .built -delete ./buildall.sh ``` From 575020eb1dcdbb4c10e69bc52f1caccc71294531 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 13:00:54 +0200 Subject: [PATCH 10/21] management wrappers: pass arguments through The bin/<command> wrappers are generated from an unquoted heredoc, so "$@" was expanded at build time (empty) and every wrapper ran "manage.py <command> \"\"" -- arguments were silently dropped. E.g. "bin/document_exporter $OLD/export" (the backup step in UPGRADE.md) exported into paperless-ngx/src instead. Escape it so the wrapper gets a literal "$@". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- paperless-ngx.build.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paperless-ngx.build.sh b/paperless-ngx.build.sh index 2904fa3..2846438 100755 --- a/paperless-ngx.build.sh +++ b/paperless-ngx.build.sh @@ -124,7 +124,7 @@ tee ${PROJECT}/bin/${command} <<_EOSH_ #!/bin/bash . ${PROJECT}/profile cd ${PREFIX}/src -python3 manage.py ${command} "$@" +python3 manage.py ${command} "\$@" _EOSH_ chmod u=rwx,g=rwx,o=rx ${PROJECT}/bin/${command} done From 50dbec46fb59a914fe4d7ae6cc1f929c5e948086 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 14:44:26 +0200 Subject: [PATCH 11/21] UPGRADE.md: tags paperless-2.20.15-3 / paperless-3.3.1-4, safe backup step Both hops tested end to end on a fresh host. The backup step now calls manage.py directly: on the old install the bin/document_exporter wrapper still drops its arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- UPGRADE.md | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/UPGRADE.md b/UPGRADE.md index c237972..1eb2b19 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -10,16 +10,20 @@ So every host goes through two hops. Each hop is a git tag: | Tag | `BUILD_TAG` | What it is | |---|---|---| -| `paperless-2.20.15-2` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 | -| `paperless-3.3.1-3` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | +| `paperless-2.20.15-3` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 | +| `paperless-3.3.1-4` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | -**Use only these two tags.** The older `paperless-2.20.15`, `paperless-3.3.1` and `paperless-3.3.1-2` are superseded: +**Use only these two tags.** All older `paperless-2.20.15*` and `paperless-3.3.1*` tags are superseded: - libjpeg-turbo 3.2.0 fails to build on them. -- `buildall.sh` reports success even when a component failed (except on `-2`). +- On most of them, `buildall.sh` reports success even when a component failed. - nginx shows `Server: nginx` instead of chicken. - The consume-test PDF is malformed. +- The `bin/<command>` wrappers drop all their arguments. -All of this was found and fixed by a full test build of hop 1. +Both hops were tested end to end on a fresh host with these two tags: +- full build of each hop +- start, consume with OCR and login +- data migration 2.20.15 → 3.3.1, including the Tantivy reindex and search Do **not** skip hop 1. 2.20.15 must start once, so its DB migrations get applied, before you build 3.3.1. @@ -40,15 +44,18 @@ Do **not** skip hop 1. 2.20.15 must start once, so its DB migrations get applied ```bash OLD=~/paperless-server/2.20.10 # your current version mkdir -p "$OLD/export" - "$OLD/bin/document_exporter" "$OLD/export" + cd "$OLD/paperless-ngx/src" && ( . "$OLD/profile"; python3 manage.py document_exporter "$OLD/export" ) + ls "$OLD/export" # must contain manifest.json ``` + Call `manage.py` directly here. On installs built from older tags, `bin/document_exporter` drops its arguments + and exports into `paperless-ngx/src` instead. ## Hop 1: 2.20.x → 2.20.15 ```bash cd <path>/paperless-baremetal git fetch --tags origin # or whichever remote this host pulls from -git checkout paperless-2.20.15-2 +git checkout paperless-2.20.15-3 # BUILD_TAG changed, so remove all .built tags. Otherwise components whose # version did not change get skipped and are missing from the new prefix. @@ -97,7 +104,7 @@ Read the "Breaking changes" section below first, especially if you use custom pr ```bash cd <path>/paperless-baremetal git fetch --tags origin -git checkout paperless-3.3.1-3 +git checkout paperless-3.3.1-4 find build/ -mindepth 2 -maxdepth 2 -name .built -delete ./buildall.sh ``` From 57f0749a846470c6fb2a8219029ed3a83da2f556 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 14:49:17 +0200 Subject: [PATCH 12/21] paperless-ngx.build.sh: actually create the admin user on fresh installs create_superuser ran before any migration ('no such table: auth_user'), and '|| true' hid it, so a fresh install had no admin login. Migrate first (no-op on an existing current DB; on an upgrade the fresh data/ is replaced by the copied one), create the admin only if missing, and drop '|| true' so real failures stop the build. Tested against an empty data dir: first run creates the superuser, second run keeps it (password unchanged) and exits 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- paperless-ngx.build.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/paperless-ngx.build.sh b/paperless-ngx.build.sh index 2846438..7f2740d 100755 --- a/paperless-ngx.build.sh +++ b/paperless-ngx.build.sh @@ -94,14 +94,17 @@ uv pip install --system --no-python-downloads --python-preference system \ ./manage.py compilemessages -# migration done by startstop.sh -# ./manage.py migrate +# the admin user needs the auth tables: migrate first (no-op on an existing, +# current DB; startstop.sh also migrates on every start). On an upgrade the +# fresh data/ created here is replaced by the copied one anyway. +./manage.py migrate --no-input PAPERLESS_ADMIN_USER=${PAPERLESS_ADMIN_USER:-admin} PAPERLESS_ADMIN_PASSWORD=${PAPERLESS_ADMIN_PASSWORD:-$(</dev/urandom tr -dc '[:lower:][:upper:][:digit:]' | head -c64)} PAPERLESS_ADMIN_MAIL=${PAPERLESS_ADMIN_MAIL:-admin@localhost.lan} -./manage.py shell -c "from django.contrib.auth.models import User; User.objects.create_superuser('${PAPERLESS_ADMIN_USER}', '${PAPERLESS_ADMIN_MAIL}', '${PAPERLESS_ADMIN_PASSWORD}')" || true +# create only if missing (rebuilds keep the existing admin and its password) +./manage.py shell -c "from django.contrib.auth.models import User; User.objects.filter(username='${PAPERLESS_ADMIN_USER}').exists() or User.objects.create_superuser('${PAPERLESS_ADMIN_USER}', '${PAPERLESS_ADMIN_MAIL}', '${PAPERLESS_ADMIN_PASSWORD}')" for command in document_archiver \ document_exporter \ From 818cc21d6c390d12ed4dd32dc187a8817baaef4a Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 15:04:02 +0200 Subject: [PATCH 13/21] v3: keep always creating PDF/A archives; UPGRADE.md: host variations, hop 2 tag paperless-3.3.1-5 paperless.conf: PAPERLESS_ARCHIVE_FILE_GENERATION=always (overridable in build.local). v2's default OCR_SKIP_ARCHIVE_FILE=never always made an archive; v3 drops that setting and defaults to 'auto', which skips archives for born-digital PDFs. Verified on a 3.3.1 test install: a born-digital PDF gets its archive. UPGRADE.md: generic lessons from inspecting a production host: per-tag BUILD_DIR (no .built cleanup needed), shared consume/ symlink (must be recreated, not copied), prod symlink + wrapper switch-over, read-only deploy key fetch, pre-building both hops for a single maintenance window. Hop 2 now uses tag paperless-3.3.1-5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- UPGRADE.md | 38 +++++++++++++++++++++++++++++++++++--- paperless.conf.build.sh | 3 +++ 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/UPGRADE.md b/UPGRADE.md index 1eb2b19..3046b95 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -11,7 +11,7 @@ So every host goes through two hops. Each hop is a git tag: | Tag | `BUILD_TAG` | What it is | |---|---|---| | `paperless-2.20.15-3` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 | -| `paperless-3.3.1-4` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | +| `paperless-3.3.1-5` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | **Use only these two tags.** All older `paperless-2.20.15*` and `paperless-3.3.1*` tags are superseded: - libjpeg-turbo 3.2.0 fails to build on them. @@ -104,7 +104,7 @@ Read the "Breaking changes" section below first, especially if you use custom pr ```bash cd <path>/paperless-baremetal git fetch --tags origin -git checkout paperless-3.3.1-4 +git checkout paperless-3.3.1-5 find build/ -mindepth 2 -maxdepth 2 -name .built -delete ./buildall.sh ``` @@ -134,13 +134,45 @@ Search results are incomplete until it finishes. | Login rate limit uses client IP | conf sets `PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER=X-Real-IP`; nginx sets this header and granian only listens on a unix socket | none | | Consumer: `CONSUMER_POLLING*`/`INOTIFY_DELAY` renamed/removed, ignore patterns are now regex | not used here | update them if you added them by hand | | Duplicates no longer rejected by default | not set | set `PAPERLESS_CONSUMER_DELETE_DUPLICATES=true` if you want the old behaviour | -| `OCR_MODE=skip/skip_noarchive` and `OCR_SKIP_ARCHIVE_FILE` removed | we use `OCR_MODE=force`, still valid | check OCR settings in the admin UI if you changed them there | +| `OCR_MODE=skip/skip_noarchive` and `OCR_SKIP_ARCHIVE_FILE` removed; new `ARCHIVE_FILE_GENERATION` defaults to `auto` (no archive for born-digital PDFs) | we use `OCR_MODE=force` (still valid) and set `ARCHIVE_FILE_GENERATION=always`, which keeps the v2 behaviour of always making a PDF/A archive | set `PAPERLESS_ARCHIVE_FILE_GENERATION=auto` in `build.local` if you want the v3 default; check OCR settings in the admin UI if you changed them there | | Pre/post-consume scripts no longer get `$1…$8` | — | switch custom scripts to the `DOCUMENT_*` env vars | | Postgres/MariaDB need explicit `PAPERLESS_DBENGINE` | SQLite is the default here | only if you configured an external DB | | OIDC may need `token_auth_method` | — | only if you use OIDC and get `invalid_client` | | Task history cleared | — | none | | `torch` (CPU) is now a hard dependency | `paperless-ngx.build.sh` installs from the PyTorch CPU index, like upstream's Dockerfile | network access to `download.pytorch.org` | +## Host variations (check before you start) + +Real installs differ from the plain `~/paperless-server/<version>` layout assumed above. Adapt the steps if your host has any of these: + +- **A per-version `BUILD_DIR` in `build.local`** (e.g. `BUILD_DIR=/dev/shm/${USER}/build-${BUILD_TAG}`). Every version + gets a fresh build dir, so you can skip the `find build/ … -name .built -delete` step. The repo's `build` symlink may still + point to an old, deleted build dir; that's harmless. +- **A shared `consume/` symlink** (e.g. `consume -> ../consume`, fed by syncthing or scanners). The build creates `consume/` + as a real directory in the new install. Replace it with the same symlink, and don't run the `cp … consume/.` step, + or files land where nobody watches: + ```bash + rmdir "$NEW/consume" && ln -s "$(readlink "$OLD/consume")" "$NEW/consume" + ``` +- **A `prod` symlink plus a top-level `startstop.sh` wrapper** (`cd prod; ./startstop.sh "$@"`), which is what the boot + scripts call. Switch over by re-pointing the symlink, then start through the wrapper: + ```bash + ln -sfn <new-version> /path/to/prod + ``` + Everything that goes through `prod` (e.g. a `media -> prod/media` link) follows automatically. +- **Fetching with a read-only deploy key**, if the host's default ssh key has no GitHub access: + ```bash + GIT_SSH_COMMAND="ssh -i ~/.ssh/ghm_paperlessbm_ro -o IdentitiesOnly=yes" git fetch --tags origin + ``` + +**Both hops in one maintenance window.** Building doesn't touch the running install: it uses its own prefix, build dir and +`/dev/shm` paths, and it doesn't start any services. So you can build both versions ahead of time while the old one keeps +serving: +1. Check out the hop 1 tag and build. +2. Check out the hop 2 tag and build. +3. In the window: stop the old install, copy to 2.20.15, start it (this runs its migrations), verify, stop it. +4. Copy to 3.3.1, start it, verify. + ## Rollback Each hop leaves the previous install untouched in its own directory. To roll back, stop the new install diff --git a/paperless.conf.build.sh b/paperless.conf.build.sh index f73dfbf..4b12ba6 100755 --- a/paperless.conf.build.sh +++ b/paperless.conf.build.sh @@ -36,6 +36,9 @@ PAPERLESS_COOKIE_PREFIX=ngx-${BUILD_TAG}- PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER=X-Real-IP PAPERLESS_OCR_MODE=${PAPERLESS_OCR_MODE:-force} +# v2 behaviour (OCR_SKIP_ARCHIVE_FILE=never): always keep a PDF/A archive. +# v3's default "auto" would skip it for born-digital PDFs. Override in build.local. +PAPERLESS_ARCHIVE_FILE_GENERATION=${PAPERLESS_ARCHIVE_FILE_GENERATION:-always} PAPERLESS_OCR_DESKEW=false PAPERLESS_OCR_LANGUAGE=deu+eng # https://github.com/paperless-ngx/paperless-ngx/issues/416 From 9ed34a412c42a20a7ce75a62ab14c0bcee6b5995 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 16:27:44 +0200 Subject: [PATCH 14/21] paperless-ngx.build.sh: build-time migrate must not need redis The fresh-install admin fix (57f0749) runs 'manage.py migrate' during the build. On v3 that fails on a real build host: django-treenode's post_migrate handler writes to the Django cache, which is redis-backed, and redis isn't running at build time (found by the production build on barbie: buildall exit=1). My earlier test only passed because the test install's redis happened to be running. Use PAPERLESS_CACHE_BACKEND=LocMemCache for the two build-time commands only (paperless uses LocMemCache itself in DEBUG). Tested with redis unreachable: default cache fails as on barbie; with the override the fresh run creates the admin, a rerun keeps it, and the variable is unset afterwards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- paperless-ngx.build.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/paperless-ngx.build.sh b/paperless-ngx.build.sh index 7f2740d..64dd899 100755 --- a/paperless-ngx.build.sh +++ b/paperless-ngx.build.sh @@ -97,6 +97,10 @@ uv pip install --system --no-python-downloads --python-preference system \ # the admin user needs the auth tables: migrate first (no-op on an existing, # current DB; startstop.sh also migrates on every start). On an upgrade the # fresh data/ created here is replaced by the copied one anyway. +# redis isn't running at build time, but v3's post_migrate (django-treenode) +# writes to the Django cache, which is redis-backed: use an in-process cache +# for these two build-time commands only. +export PAPERLESS_CACHE_BACKEND=django.core.cache.backends.locmem.LocMemCache ./manage.py migrate --no-input PAPERLESS_ADMIN_USER=${PAPERLESS_ADMIN_USER:-admin} @@ -105,6 +109,7 @@ PAPERLESS_ADMIN_MAIL=${PAPERLESS_ADMIN_MAIL:-admin@localhost.lan} # create only if missing (rebuilds keep the existing admin and its password) ./manage.py shell -c "from django.contrib.auth.models import User; User.objects.filter(username='${PAPERLESS_ADMIN_USER}').exists() or User.objects.create_superuser('${PAPERLESS_ADMIN_USER}', '${PAPERLESS_ADMIN_MAIL}', '${PAPERLESS_ADMIN_PASSWORD}')" +unset PAPERLESS_CACHE_BACKEND for command in document_archiver \ document_exporter \ From 15d18321473970dab3ee3e4ffe85bce3e3ad5c24 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 16:27:44 +0200 Subject: [PATCH 15/21] UPGRADE.md: hop 2 tag paperless-3.3.1-6 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- UPGRADE.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/UPGRADE.md b/UPGRADE.md index 3046b95..f3d5968 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -11,7 +11,7 @@ So every host goes through two hops. Each hop is a git tag: | Tag | `BUILD_TAG` | What it is | |---|---|---| | `paperless-2.20.15-3` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 | -| `paperless-3.3.1-5` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | +| `paperless-3.3.1-6` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | **Use only these two tags.** All older `paperless-2.20.15*` and `paperless-3.3.1*` tags are superseded: - libjpeg-turbo 3.2.0 fails to build on them. @@ -104,7 +104,7 @@ Read the "Breaking changes" section below first, especially if you use custom pr ```bash cd <path>/paperless-baremetal git fetch --tags origin -git checkout paperless-3.3.1-5 +git checkout paperless-3.3.1-6 find build/ -mindepth 2 -maxdepth 2 -name .built -delete ./buildall.sh ``` From cc2cb56d598d987adeeead6fd2ecddd953d5b345 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 16:50:30 +0200 Subject: [PATCH 16/21] paperless.conf: make CONSUMER_DELETE_DUPLICATES configurable via build.local v2 rejected duplicates (and left the file in consume/); v3 imports them by default. Expose PAPERLESS_CONSUMER_DELETE_DUPLICATES (default false = v3 behaviour); true rejects duplicates like v2 but also deletes the file from consume/. Needed on barbie, where old duplicate files sit in a syncthing-fed consume folder and would otherwise be imported again on the first v3 start. Tested: with true, a duplicate is rejected, the document count is unchanged and the file is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- paperless.conf.build.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/paperless.conf.build.sh b/paperless.conf.build.sh index 4b12ba6..3afcde0 100755 --- a/paperless.conf.build.sh +++ b/paperless.conf.build.sh @@ -39,6 +39,9 @@ PAPERLESS_OCR_MODE=${PAPERLESS_OCR_MODE:-force} # v2 behaviour (OCR_SKIP_ARCHIVE_FILE=never): always keep a PDF/A archive. # v3's default "auto" would skip it for born-digital PDFs. Override in build.local. PAPERLESS_ARCHIVE_FILE_GENERATION=${PAPERLESS_ARCHIVE_FILE_GENERATION:-always} +# v3 imports duplicates by default (v2 rejected them and left the file in consume/). +# true = reject like v2, but v3 also deletes the duplicate file from consume/. +PAPERLESS_CONSUMER_DELETE_DUPLICATES=${PAPERLESS_CONSUMER_DELETE_DUPLICATES:-false} PAPERLESS_OCR_DESKEW=false PAPERLESS_OCR_LANGUAGE=deu+eng # https://github.com/paperless-ngx/paperless-ngx/issues/416 From 0abd09cd714e20b62059f008d09950bafff5d099 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 16:50:30 +0200 Subject: [PATCH 17/21] UPGRADE.md: duplicate leftovers in consume/, hop 2 tag paperless-3.3.1-7 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- UPGRADE.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/UPGRADE.md b/UPGRADE.md index f3d5968..93f8378 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -11,7 +11,7 @@ So every host goes through two hops. Each hop is a git tag: | Tag | `BUILD_TAG` | What it is | |---|---|---| | `paperless-2.20.15-3` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 | -| `paperless-3.3.1-6` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | +| `paperless-3.3.1-7` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | **Use only these two tags.** All older `paperless-2.20.15*` and `paperless-3.3.1*` tags are superseded: - libjpeg-turbo 3.2.0 fails to build on them. @@ -104,7 +104,7 @@ Read the "Breaking changes" section below first, especially if you use custom pr ```bash cd <path>/paperless-baremetal git fetch --tags origin -git checkout paperless-3.3.1-6 +git checkout paperless-3.3.1-7 find build/ -mindepth 2 -maxdepth 2 -name .built -delete ./buildall.sh ``` @@ -133,7 +133,7 @@ Search results are incomplete until it finishes. | New LLM index | `startstop.sh` runs `document_llmindex migrate`; `bin/document_llmindex` wrapper added | none | | Login rate limit uses client IP | conf sets `PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER=X-Real-IP`; nginx sets this header and granian only listens on a unix socket | none | | Consumer: `CONSUMER_POLLING*`/`INOTIFY_DELAY` renamed/removed, ignore patterns are now regex | not used here | update them if you added them by hand | -| Duplicates no longer rejected by default | not set | set `PAPERLESS_CONSUMER_DELETE_DUPLICATES=true` if you want the old behaviour | +| Duplicates no longer rejected by default | `PAPERLESS_CONSUMER_DELETE_DUPLICATES` passes through from `build.local` (default `false` = v3 behaviour) | **check `consume/` for old leftover files** that v2 kept rejecting: v3 would import them as duplicates. Set `PAPERLESS_CONSUMER_DELETE_DUPLICATES=true` in `build.local` to reject them like v2; v3 then also *deletes* them from `consume/`, which a syncthing send-receive folder propagates to the sender | | `OCR_MODE=skip/skip_noarchive` and `OCR_SKIP_ARCHIVE_FILE` removed; new `ARCHIVE_FILE_GENERATION` defaults to `auto` (no archive for born-digital PDFs) | we use `OCR_MODE=force` (still valid) and set `ARCHIVE_FILE_GENERATION=always`, which keeps the v2 behaviour of always making a PDF/A archive | set `PAPERLESS_ARCHIVE_FILE_GENERATION=auto` in `build.local` if you want the v3 default; check OCR settings in the admin UI if you changed them there | | Pre/post-consume scripts no longer get `$1…$8` | — | switch custom scripts to the `DOCUMENT_*` env vars | | Postgres/MariaDB need explicit `PAPERLESS_DBENGINE` | SQLite is the default here | only if you configured an external DB | From 4d5dce555ee84e9a6bf24b2d51aed9512ee582d9 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 17:08:46 +0200 Subject: [PATCH 18/21] UPGRADE.md: lessons from the barbie production upgrade - check consume/ for v2 duplicate leftovers before starting - match the old install's permissions (chmod -R g+w,o-rwx) - retrain the classifier after hop 2 (v3 clears task history) - expected consume errors that also existed on v2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- UPGRADE.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/UPGRADE.md b/UPGRADE.md index 93f8378..084df84 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -40,6 +40,8 @@ Do **not** skip hop 1. 2.20.15 must start once, so its DB migrations get applied - **`build.local`.** Keep the one you already have. Check that it sets a fixed `PAPERLESS_SECRET_KEY`. If it doesn't, copy the value out of the running install's `paperless-ngx/paperless.conf` into `build.local`. Otherwise every rebuild generates a new key, which logs everyone out, and v3 refuses to start without a key. +- **Leftovers in `consume/`.** List them: `find -L "$OLD/consume" -type f ! -path '*/.st*'`. v2 rejected duplicates and left + the files there, so they're retried on every start. v3 would *import* them; see the duplicates row in the table under hop 2. - **Backup.** With the old install running: ```bash OLD=~/paperless-server/2.20.10 # your current version @@ -119,7 +121,10 @@ Search results are incomplete until it finishes. - Search returns results once the reindex is done. - The consume test with `paperless-online-test.pdf` works. - A barcode/ASN test document still gets split/tagged, if you use barcodes. v3 uses zxing-cpp instead of zbar. -- `grep -il traceback "$NEW"/log/*` finds nothing. +- `grep -il traceback "$NEW"/log/*` finds nothing. Exceptions: consume errors for files that also failed on v2 + (e.g. Office files without Tika, 16-bit TIFFs) and duplicate rejections. +- Retrain the classifier once: `"$NEW/bin/document_create_classifier"`. v3 clears the task history, so the status + page warns "No classifier training tasks found" until the next scheduled run. ### Breaking changes in v3 and how this repo handles them @@ -145,6 +150,8 @@ Search results are incomplete until it finishes. Real installs differ from the plain `~/paperless-server/<version>` layout assumed above. Adapt the steps if your host has any of these: +- **Different permissions on the old install** (e.g. `drwxrws---` for a shared project group). New builds use the + builder's umask; match the old install before switching: `chmod -R g+w,o-rwx "$NEW"` (doesn't follow the consume symlink). - **A per-version `BUILD_DIR` in `build.local`** (e.g. `BUILD_DIR=/dev/shm/${USER}/build-${BUILD_TAG}`). Every version gets a fresh build dir, so you can skip the `find build/ … -name .built -delete` step. The repo's `build` symlink may still point to an old, deleted build dir; that's harmless. From 3982bcef2d94d5f4e8ebea4d559867df8b3a544f Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Tue, 13 Oct 2026 18:39:34 +0200 Subject: [PATCH 19/21] UPGRADE.md: classifier warning persists until the scheduled training run Seen on plvin: after a manual document_create_classifier the status page still warns, because it only counts scheduled training tasks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- UPGRADE.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/UPGRADE.md b/UPGRADE.md index 084df84..c6deec6 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -124,7 +124,8 @@ Search results are incomplete until it finishes. - `grep -il traceback "$NEW"/log/*` finds nothing. Exceptions: consume errors for files that also failed on v2 (e.g. Office files without Tika, 16-bit TIFFs) and duplicate rejections. - Retrain the classifier once: `"$NEW/bin/document_create_classifier"`. v3 clears the task history, so the status - page warns "No classifier training tasks found" until the next scheduled run. + page warns "No classifier training tasks found". A manual run doesn't clear that warning, because the page only counts + *scheduled* training tasks. It goes away after the next scheduled run (hourly by default). ### Breaking changes in v3 and how this repo handles them From 02c321b22ba8811d2d82969d9d4bcc12d438d087 Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Wed, 14 Oct 2026 10:44:37 +0200 Subject: [PATCH 20/21] unpaper: always configure a fresh meson builddir With a shared BUILD_DIR (e.g. BUILD_DIR=build) the unpaper source dir is reused across versions because the pinned unpaper version doesn't change. meson then finds the old builddir ('Directory already configured') and, after a meson upgrade, refuses it: found by the 2.20.15 build on platsch (builddir from meson 1.10.0, now 1.12.1). Remove builddir before meson setup, as libjpegturbo/qpdf already do with their cmake dir. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- unpaper.build.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/unpaper.build.sh b/unpaper.build.sh index 2ad08d0..f0dfef0 100755 --- a/unpaper.build.sh +++ b/unpaper.build.sh @@ -53,6 +53,9 @@ local _conf=( --pkg-config-path="${PKG_CONFIG_PATH}" ) +# a builddir left from an earlier build (shared BUILD_DIR, same unpaper +# version) is not reconfigured and fails after a meson upgrade +rm -rf builddir CFLAGS="${_cflags[*]}" meson setup "${_conf[@]}" builddir meson install -C builddir From 3cb75590c68f710a8665862c18b5bf0ea94b806d Mon Sep 17 00:00:00 2001 From: wwwutz personal clanker <Awwwutz@eldersoftheinternet.molgen.mpg.de> Date: Wed, 14 Oct 2026 10:44:37 +0200 Subject: [PATCH 21/21] UPGRADE.md: tags paperless-2.20.15-4 / paperless-3.3.1-8; shared BUILD_DIR needs .built cleanup per hop Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- UPGRADE.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/UPGRADE.md b/UPGRADE.md index c6deec6..1df2718 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -10,8 +10,8 @@ So every host goes through two hops. Each hop is a git tag: | Tag | `BUILD_TAG` | What it is | |---|---|---| -| `paperless-2.20.15-3` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 | -| `paperless-3.3.1-7` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | +| `paperless-2.20.15-4` | 2.20.15 | All dependencies at their current bugfix level, paperless 2.20.15 | +| `paperless-3.3.1-8` | 3.3.1 | paperless 3.3.1 plus the build/config changes v3 needs | **Use only these two tags.** All older `paperless-2.20.15*` and `paperless-3.3.1*` tags are superseded: - libjpeg-turbo 3.2.0 fails to build on them. @@ -57,7 +57,7 @@ Do **not** skip hop 1. 2.20.15 must start once, so its DB migrations get applied ```bash cd <path>/paperless-baremetal git fetch --tags origin # or whichever remote this host pulls from -git checkout paperless-2.20.15-3 +git checkout paperless-2.20.15-4 # BUILD_TAG changed, so remove all .built tags. Otherwise components whose # version did not change get skipped and are missing from the new prefix. @@ -106,7 +106,7 @@ Read the "Breaking changes" section below first, especially if you use custom pr ```bash cd <path>/paperless-baremetal git fetch --tags origin -git checkout paperless-3.3.1-7 +git checkout paperless-3.3.1-8 find build/ -mindepth 2 -maxdepth 2 -name .built -delete ./buildall.sh ``` @@ -153,6 +153,9 @@ Real installs differ from the plain `~/paperless-server/<version>` layout assume - **Different permissions on the old install** (e.g. `drwxrws---` for a shared project group). New builds use the builder's umask; match the old install before switching: `chmod -R g+w,o-rwx "$NEW"` (doesn't follow the consume symlink). +- **A shared `BUILD_DIR`** (the default `build`, or `BUILD_DIR=build` in `build.local`). The `.built` markers then belong to + whatever version built last. Run the `find build/ … -name .built -delete` step before **each** hop's build, not + just once. Otherwise the hop 2 build skips almost everything, and those components are missing from the 3.3.1 install. - **A per-version `BUILD_DIR` in `build.local`** (e.g. `BUILD_DIR=/dev/shm/${USER}/build-${BUILD_TAG}`). Every version gets a fresh build dir, so you can skip the `find build/ … -name .built -delete` step. The repo's `build` symlink may still point to an old, deleted build dir; that's harmless.