Add rolling reboot #558
Open
Add rolling reboot #558
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
|
Not yet in |
donald
reviewed
Aug 12, 2026
mxq-rolling-reboot/mxrolling-reboot
Outdated
| # --------------------------------------------------------------------------- | ||
| # Configuration (override via environment or the command line flags below) | ||
| # --------------------------------------------------------------------------- | ||
| TAG=${TAG:-mxqd} # hostconfig tag selecting the nodes |
Collaborator
There was a problem hiding this comment.
Maybe just me, but I think, I would use quotes:
buczek@dose:~/git/mxtools/mxq-rolling-reboot (add-mxq-rolling-reboot)$ TAG="bla bla" ./mxrolling-reboot mariux-6.12.98-492
Number found where operator expected at (eval 1) line 1, near "0 0"
(Missing operator before 0?)
/bin/hostconfig: syntax error in expression: bla bla
2026-08-12 15:42:14 mxrolling-reboot: hostconfig --list bla bla failed
Contributor
Author
There was a problem hiding this comment.
Quoted. Every variable expansion on the right-hand side of an assignment is quoted now, in the config block and everywhere else (ed1b367). The hostconfig --list "$TAG" call itself was already quoted — what misled here was the log line printing the tag bare, so it read as two arguments. Now: hostconfig --list 'bla bla' failed.
donald
reviewed
Aug 12, 2026
mxq-rolling-reboot/mxrolling-reboot
Outdated
| --plain-max) PLAIN_MAX=$2; shift 2 ;; | ||
| --no-plain) DO_PLAIN=0; shift ;; | ||
| --poll-interval) POLL_INTERVAL=$2; shift 2 ;; | ||
| --tag) TAG=$2; shift 2 ;; |
Collaborator
There was a problem hiding this comment.
Quotes? Here and everywhere?
buczek@dose:~/git/mxtools/mxq-rolling-reboot (add-mxq-rolling-reboot)$ ./mxrolling-reboot --tag "bla bla" mariux-6.12.98-492
Number found where operator expected at (eval 1) line 1, near "0 0"
(Missing operator before 0?)
/bin/hostconfig: syntax error in expression: bla bla
2026-08-12 15:43:09 mxrolling-reboot: hostconfig --list bla bla failed
Contributor
Author
There was a problem hiding this comment.
Done — same pass as above, --tag) TAG="$2" and all the other =$2/=$1 sites.
donald
reviewed
Aug 12, 2026
mxq-rolling-reboot/mxrolling-reboot
Outdated
| POLL_INTERVAL=${POLL_INTERVAL:-30} # seconds between status polls | ||
| SCAN_PARALLEL=${SCAN_PARALLEL:-16} # parallelism for the initial kernel scan | ||
| SSH_USER=${SSH_USER:-} # empty = current user; e.g. root | ||
| SSH_OPTS=${SSH_OPTS:-"-o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new"} |
Collaborator
There was a problem hiding this comment.
ForwardX11=no ? We have the unusual default of ForwardX11=yes
Contributor
Author
There was a problem hiding this comment.
Added -o ForwardX11=no to the default SSH_OPTS, with a comment saying why (our ssh_config has ForwardX11 yes and none of these calls needs a display). Verified with ssh -G: forwardx11 no.
donald
reviewed
Aug 12, 2026
mxq-rolling-reboot/mxrolling-reboot
Outdated
| # are not in this list do not run mxqd (e.g. when --tag selects a non-cluster | ||
| # tag), so they need no drain and no mxqd start after the reboot. | ||
| detect_mxqd_nodes() { | ||
| awk '$1 !~ /^#/ && $2=="mxqd" {print $1}' /etc/hostconfig |
Contributor
Author
There was a problem hiding this comment.
Switched detect_mxqd_nodes to hostconfig --list mxqd — thanks, that drops the hand-rolled parsing. Checked both give the same 53 hosts and both skip the commented-out mxqd lines.
detect_mxqi_nodes has to keep reading /etc/hostconfig though: mxqi is not a hostconfig tag but an mxqd --prere value, so hostconfig --list mxqi returns nothing. Using it there would have silently emptied the mxqi pool.
Collaborator
|
okay, nice. |
pmenzel
force-pushed
the
add-mxq-rolling-reboot
branch
from
67983dc to
ed1b367
Compare
Contributor
Author
|
Added to All four review comments are folded into the commits that introduced the code rather than added on top, so the branch was force-pushed ( |
Reboot mxqd cluster nodes into a new kernel without killing running jobs and without draining the whole cluster at once. For each outdated node it drains mxqd (stop, then wait for the mxqd process to exit), installs a one-time systemd unit that runs mxqdctl-hostconfig start on the next boot and removes itself again, selects the kernel with mxgrub and reboots (kexec by default, or a normal reboot for firmware changes), all detached via systemd-run so long drains survive an SSH disconnect. Pacing is load-aware from the mxq server status page: at most BUSY_MAX nodes (default 2) when the cluster is fully busy, otherwise up to half of the currently idle nodes. Nodes already on the target kernel are detected via uname -r and skipped, so re-running is safe. The interactive (mxqi) nodes are handled as a separate, throttled pool: at most MXQI_MAX of them (default 1) are upgraded at a time and at least MXQI_MIN_UP (default 1) is always kept up and serving, so interactive sessions can always be started. mxqi nodes are auto-detected from /etc/hostconfig as the mxqd nodes whose prerequisites request 'mxqi'. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ocking The mxqi keep-serving guard counted only nodes serving on the status page and then assumed the node about to be taken down was one of them (serving - 1 >= MXQI_MIN_UP). When every mxqi node already had mxqd stopped or draining, none appeared on the page, so 'serving' was 0 and the guard held forever, waiting for a node that could not return on its own. Only protect nodes that are actually serving: prefer an mxqi node that is already down (rebooting it removes no interactive capacity), and apply the keep-serving throttle only when the remaining candidates are still serving. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the HTML scrape of the mxq server status page with a direct query of the mxq_daemon table via /etc/mxq/mysql_ro.cnf. This is a machine-readable, authoritative source and drops the curl dependency. Take the newest daemon row per host and keep only daemons that are actually accepting jobs (status 0 = idle, 10 = running); a draining daemon (status 200) or a gone one (>= 250) is left out, so the 'serving' notion the mxqi keep-serving guard relies on stays correct. Rows not refreshed within DB_FRESH_MIN minutes are ignored so a daemon that vanished without recording a stop is not mistaken for live. Skip a poll when the query fails instead of acting on an empty result, which would otherwise look like 'nothing is serving' and could wrongly free the mxqi guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
--tag takes any hostconfig tag, and non-cluster nodes have tags too, but so
far every selected node was treated as an mxqd node: it got the one-time
'mxqdctl-hostconfig start' unit, and the detached script ran
mxqdctl-hostconfig stop
while pgrep -x mxqd >/dev/null 2>&1; do sleep 10; done
which on a node without mxqd is pointless (mxqdctl-hostconfig is not even
installed there, so 'set -e' aborts the script before mxgrub and the node is
never rebooted), and the load-aware pacing derived from the mxq database is
meaningless for such nodes as well.
Read the mxqd nodes from /etc/hostconfig and split the work into three pools:
the load-paced batch mxqd nodes, the throttled mxqi nodes, and the nodes
without mxqd. The latter are rebooted strictly one after another (PLAIN_MAX,
default 1, --plain-max/--no-plain to change), and their remote script is just
'mxgrub <label>' plus the reboot - no drain, no start unit. A run that selects
no mxqd node does not query the mxq database at all.
If /etc/hostconfig lists no mxqd node the tool now refuses to run instead of
silently classifying every node as mxqd-free and rebooting busy nodes without
draining them; --mxqd-nodes overrides the detection ('' means none of them).
Assisted-by: Claude Opus 5
Selecting a single serving mxqi node made the tool loop forever:
mxrolling-reboot -H stairwaytoheaven mariux-6.18.42-494
mxqi: holding (1 serving; keep >= 1) - waiting for an mxqi node to return
MXQI_ALL held the mxqi nodes intersected with the selection, and the
keep-serving guard counted serving nodes over that intersection only. With
one mxqi node selected, serving - 1 = 0 never reaches MXQI_MIN_UP, so the
node was never taken down, even though the other mxqi nodes were up and
serving interactive sessions all along.
Count over every mxqi node of the cluster instead and restrict only the
upgrade list to the selection: MXQI_ALL is now the full set, MXQI_SEL the
selected ones. The guard still holds when taking the node down would really
drop the cluster below MXQI_MIN_UP serving mxqi nodes.
Assisted-by: Claude Opus 5
The batch concurrency limit was derived from a cluster-wide idle count read from the mxq database, so a narrow selection was paced by nodes the run never touches: $ ./mxrolling-reboot --dry-run -H a,b,c,d mariux-6.12.98-492 DRY-RUN: 50 idle nodes now -> would upgrade up to 25 batch node(s) at a time Count the idle nodes over the batch pool instead, i.e. over the selected mxqd nodes that are not mxqi nodes (BATCH_ALL), so the limit reflects the pool it paces: DRY-RUN: 2 of the 4 selected batch node(s) idle now -> would upgrade up to 1 at a time A full-cluster run is unchanged except that the mxqi nodes no longer count towards the batch limit - they have their own pool and pacing. The two copies of the "how many at a time" arithmetic and of the status parsing move into batch_limit(), batch_idle() and load_status(). Assisted-by: Claude Opus 5 (claude-opus-5)
pmenzel
force-pushed
the
add-mxq-rolling-reboot
branch
from
ed1b367 to
392f8ca
Compare
Sign in
to join this conversation on GitHub.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tool to reboot mxq cluster nodes into new Linux kernel:
Known limitation (follow-up, not addressed here)
If the cluster has exactly one mxqi node and it is both selected and serving, the
keep-serving guard holds forever (
serving - 1 = 0 < MXQI_MIN_UP), logging"waiting for an mxqi node to return" for a node that will never be initiated.
It is escapable with
MXQI_MIN_UP=0, but the tool spins instead of saying so.Worth a follow-up: detect that no eligible mxqi node can ever free up and either
bail out with that hint or skip the mxqi pool.