Update 3.3.1 #105
Merged
Update 3.3.1 #105
+369
−646
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Phase 1 of the 2.20.x -> 3.3.1 upgrade (v3 only migrates from 2.20.15).
Versions (siblings' pins where shared, else latest patch):
paperless 2.20.15, sqlite 3.53.4, redis 7.4.11, python 3.12.15,
node 24.21.0, nginx 1.31.6, ffmpeg 8.0.3, libjpegturbo 3.2.0,
libtiff 4.7.2, ghostscript 10.07.1, imagemagick 7.1.2-33,
tesseract 5.5.3. All tarballs pre-mirrored on beehive so production
hosts need neither md5repo.sh nor the sibling repos.
Script fixes ported from the sibling repos:
- brace-group "test -d X || { mkdir X && tar ...; }" (tar re-extracted
on every run due to && / || precedence)
- python: fix malformed rpath "-Wl,-rpath -Wl,-L..." -> "-Wl,-rpath,..."
Remove local checkversions.sh in favour of ../baremetals/checkversions.sh.
CLAUDE.md: note that .built tags are not per BUILD_TAG, that the asgi
patch is not actually applied, and point to UPGRADE.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
paperless-ngx.asgi.patch: disabled since 2.16.3 and broken as written
(XXXASGIHandler, missing imports). The Django bug it backports (#36399,
multiple Cookie headers joined with ",") is still present in 5.2.17, but
cannot trigger here: nginx merges HTTP/2/3 cookie headers into a single
Cookie header and proxies HTTP/1.1 to granian.
cleanall.sh: build trees live in ${BUILD_DIR}, but the script removed
relative to the cwd (and sourcing build.profile failed from BUILD_DIR),
so it never removed anything. Now runs from the repo dir and pushd's
into ${BUILD_DIR}. Add the missing libjpegturbo, libtiff, libwebp,
ghostscript, imagemagick and qpdf; nginx stays excluded as before.
Make it executable like all other scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only changes the build/install directory name, so jbig2enc gets rebuilt
into ${PREFIX}/jbig2enc-0.30-20-g2f7f76b; profile.build.sh and
paperless-ngx.build.sh reference it via ${BUILD_jbig2enc}.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
These were cloned at upstream HEAD; BUILD_<name> was only a label compared against "git describe" (jbig2enc/pngquant: warning only, unpaper: exit 1). So every host built whatever upstream had that day (jbig2enc HEAD is currently 70 commits past the pin). Now check out the <sha> after "-g" in BUILD_<name> right after clone/cache extraction, so stale cached .tar files from earlier HEAD clones are pinned too. pngquant also syncs its libimagequant submodule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phase 2 of the upgrade (requires 2.20.15 to have run once, see UPGRADE.md). - drop zbar: v3 removed pyzbar in favour of zxing-cpp wheels - drop NLTK download and PAPERLESS_NLTK_DIR: NLTK is gone in v3 - drop pipenv (unused since paperless moved to uv) - uv pip install from the PyTorch CPU index like upstream's Dockerfile, torch is now a hard dependency - wrappers: remove decrypt_documents (gone), add document_llmindex - startstop: after migrate run "document_index reindex --if-needed" (Whoosh -> Tantivy) and "document_llmindex migrate", as upstream's bare-metal update docs require; both are no-ops when current - conf: PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER=X-Real-IP so allauth's login rate limiting sees the real client IP behind nginx Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failing component printed "<pkg>.build.sh failed exit code N" but
buildall.sh still exited 0: the bare "exit" took echo's status, and the
whole body runs in { ... } | tee, whose status is tee's. A failing
paperless-ngx.build.sh or generator script was masked the same way.
Add pipefail, capture the component's status explicitly (so the
failure message is still printed under set -e) and exit with it.
Tested with stub component scripts: all-ok -> 0 (unchanged), failing
component -> its exit code, failing paperless-ngx.build.sh -> its code
(old version: 0 in both failure cases).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
paperless-3.3.1 is already published; the buildall exit-code fix goes out as a new tag instead of moving the old one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- libjpegturbo 3.2.0: build with -fPIC (as bigbluebutton-baremetal does). 3.2.0 bundles spng+zlib as static libs linked into libturbojpeg.so; without PIC the link fails with "relocation R_X86_64_32 ... can not be used when making a shared object". - nginx: the chicken patch no longer applied to 1.31.6 (nginx.h hunk rejected, and the Server header was never patched), hidden by "|| true" -> responses said "Server: nginx". Take immich's patch (also patches ngx_http_header_filter_module.c) and toss's handling: apply unless already applied, fail on rejects, and verify "nginx -v" reports chicken/ after install. - paperless-online-test.pdf lacked /Type /Catalog; ghostscript in the strict mode ocrmypdf uses (-dPDFSTOPONERROR) rejects it with any version, so the consume test in UPGRADE.md always failed. Rewritten as a valid PDF with qpdf (same "Hello World!" content). Verified on a fresh host: full build, all services up, consume test OCRs "Hello World!" with archive + thumbnail, Server: chicken. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…md: chicken patch handling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bin/<command> wrappers are generated from an unquoted heredoc, so "$@" was expanded at build time (empty) and every wrapper ran "manage.py <command> \"\"" -- arguments were silently dropped. E.g. "bin/document_exporter $OLD/export" (the backup step in UPGRADE.md) exported into paperless-ngx/src instead. Escape it so the wrapper gets a literal "$@". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… step Both hops tested end to end on a fresh host. The backup step now calls manage.py directly: on the old install the bin/document_exporter wrapper still drops its arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
create_superuser ran before any migration ('no such table: auth_user'),
and '|| true' hid it, so a fresh install had no admin login. Migrate
first (no-op on an existing current DB; on an upgrade the fresh data/
is replaced by the copied one), create the admin only if missing, and
drop '|| true' so real failures stop the build.
Tested against an empty data dir: first run creates the superuser,
second run keeps it (password unchanged) and exits 0.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… hop 2 tag paperless-3.3.1-5 paperless.conf: PAPERLESS_ARCHIVE_FILE_GENERATION=always (overridable in build.local). v2's default OCR_SKIP_ARCHIVE_FILE=never always made an archive; v3 drops that setting and defaults to 'auto', which skips archives for born-digital PDFs. Verified on a 3.3.1 test install: a born-digital PDF gets its archive. UPGRADE.md: generic lessons from inspecting a production host: per-tag BUILD_DIR (no .built cleanup needed), shared consume/ symlink (must be recreated, not copied), prod symlink + wrapper switch-over, read-only deploy key fetch, pre-building both hops for a single maintenance window. Hop 2 now uses tag paperless-3.3.1-5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fresh-install admin fix (57f0749) runs 'manage.py migrate' during the build. On v3 that fails on a real build host: django-treenode's post_migrate handler writes to the Django cache, which is redis-backed, and redis isn't running at build time (found by the production build on barbie: buildall exit=1). My earlier test only passed because the test install's redis happened to be running. Use PAPERLESS_CACHE_BACKEND=LocMemCache for the two build-time commands only (paperless uses LocMemCache itself in DEBUG). Tested with redis unreachable: default cache fails as on barbie; with the override the fresh run creates the admin, a rerun keeps it, and the variable is unset afterwards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d.local v2 rejected duplicates (and left the file in consume/); v3 imports them by default. Expose PAPERLESS_CONSUMER_DELETE_DUPLICATES (default false = v3 behaviour); true rejects duplicates like v2 but also deletes the file from consume/. Needed on barbie, where old duplicate files sit in a syncthing-fed consume folder and would otherwise be imported again on the first v3 start. Tested: with true, a duplicate is rejected, the document count is unchanged and the file is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- check consume/ for v2 duplicate leftovers before starting - match the old install's permissions (chmod -R g+w,o-rwx) - retrain the classifier after hop 2 (v3 clears task history) - expected consume errors that also existed on v2 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seen on plvin: after a manual document_create_classifier the status page still warns, because it only counts scheduled training tasks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With a shared BUILD_DIR (e.g. BUILD_DIR=build) the unpaper source dir is
reused across versions because the pinned unpaper version doesn't
change. meson then finds the old builddir ('Directory already
configured') and, after a meson upgrade, refuses it: found by the
2.20.15 build on platsch (builddir from meson 1.10.0, now 1.12.1).
Remove builddir before meson setup, as libjpegturbo/qpdf already do
with their cmake dir.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…D_DIR needs .built cleanup per hop Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign in
to join this conversation on GitHub.
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ugrade