Add rolling reboot

install.sh

@@ -263,6 +263,7 @@ install_data pdist/pdist-bootcheck.service                "$DESTDIR$systemdunitd
 install_exec wakeonlan/wake                               "$DESTDIR$usr_bindir/wake"
 install_data make-automaps/make-automaps.service          "$DESTDIR$systemdunitdir/make-automaps.service"
 install_exec mxqi/mxqi                                    "$DESTDIR$usr_bindir/mxqi"
+install_exec mxq-rolling-reboot/mxrolling-reboot          "$DESTDIR$usr_sbindir/mxrolling-reboot"
 install_exec mozilla-launcher/mozilla-launcher            "$DESTDIR$usr_libdir/mariux64/mozilla_launcher"
 install_exec mozilla-launcher/fix-thunderbird-amd         "$DESTDIR$usr_bindir/fix-thunderbird-amd"
 install_exec syncthing/startstop-syncthing.sh             "$DESTDIR$usr_sbindir/startstop-syncthing.sh"

mxq-rolling-reboot/README.md

@@ -0,0 +1,174 @@
+# mxrolling-reboot
+
+Rolling kernel reboot of cluster nodes: reboots every host into a new kernel
+**without killing running mxq jobs** and **without draining the whole cluster at
+once**. Hosts are picked up by a hostconfig tag expression (see
+[Selecting the hosts](#selecting-the-hosts)), and each of the three kinds of
+host gets its own pacing (see [Pacing](#pacing)):
+
+| pool                  | pacing                                     | mxqd handling      |
+| --------------------- | ------------------------------------------ | ------------------ |
+| batch `mxqd` nodes    | load-aware (`--busy-max`/`--idle-fraction`)| drain, start again |
+| interactive `mxqi`    | at most `--mxqi-max`, keep `MXQI_MIN_UP` up| drain, start again |
+| hosts without `mxqd`  | one after another (`--plain-max`)          | none               |
+
+## What it does
+
+For each `mxqd` node still running the old kernel it launches, detached via
+`systemd-run`, the sequence:
+
+1. `mxqdctl-hostconfig stop` — mxqd stops accepting new jobs and exits once its
+   running jobs finished.
+2. wait for the `mxqd` process(es) to disappear — i.e. wait (indefinitely) for
+   the node to be fully drained. `mxqdctl-hostconfig stop` only *signals* mxqd
+   and returns immediately, so this explicit wait is what makes the reboot
+   safe.
+3. install a one-time systemd unit that runs `mxqdctl-hostconfig start` on the
+   next boot and then removes itself again (see below).
+4. `mxgrub <label>` — select the requested kernel for the next boot.
+5. reboot into it — `mxgrub --reboot` (kexec, default) or `systemctl reboot`
+   (full firmware/POST cycle).
+
+Because the work runs in a transient systemd unit on the node, a drain that
+takes as long as a job's max walltime survives an SSH disconnect and even
+survives this orchestrator being interrupted.
+
+Hosts already on the target kernel are detected with `uname -r` and skipped, so
+the tool is safe to re-run to pick up hosts that were still busy earlier.
+
+## Starting mxqd again after the reboot
+
+mxqd is not started automatically at boot, so the tool installs a one-time unit
+`mxq-rolling-reboot-start.service` on the node before rebooting. On the next
+boot it runs `mxqdctl-hostconfig start`, then disables and deletes itself, so it
+leaves no permanent change on the node. `KillMode=process` keeps the daemonized
+mxqd alive after the one-shot unit finishes. Pass `--no-start` to skip this and
+leave the node without mxqd running after the reboot.
+
+## Pacing
+
+For the batch `mxqd` nodes, how many nodes are upgraded concurrently is
+load-aware and recomputed on every poll, using the idle/busy state read from the
+mxq database (the `mxq_daemon` table, via the MySQL config
+`/etc/mxq/mysql_ro.cnf`):
+
+* none of the batch nodes idle (pool busy at maximum) → at most `BUSY_MAX` nodes
+  at a time (default **2**);
+* otherwise → up to `IDLE_FRACTION` of the idle ones at a time (default **50%**).
+
+Idle nodes are counted over the **selected** batch nodes only — the nodes that
+run mxqd, are not `mxqi` nodes, and are part of the selection — so the limit is
+derived from the pool it paces. A narrow `-H`/`--tag` selection is therefore
+paced by its own idle nodes and not by the idle nodes of the rest of the cluster,
+which this run never touches.
+
+Idle nodes are always preferred when choosing what to take down next, so busy
+nodes keep serving as long as possible.
+
+## Interactive (mxqi) nodes
+
+The interactive `mxqi` nodes run the interactive sshd sessions, and those
+sessions do not drain quickly. There are only a couple of them, so draining
+them all at once means nobody can start an mxqi session. They are therefore
+handled as a **separate, throttled pool**:
+
+* at most `MXQI_MAX` mxqi nodes are upgraded at a time (default **1**);
+* at least `MXQI_MIN_UP` mxqi nodes (default **1**) are always kept up **and
+  running mxqd** — a still-serving mxqi node is only drained once enough others
+  are serving to honour this minimum (checked via the mxq database, not just
+  "kernel is new").
+
+The keep-serving guard only protects nodes that are *actually serving*. An mxqi
+node whose mxqd is already stopped or draining is not registered as an active
+daemon in the database, so rebooting it costs no interactive capacity and it is
+upgraded right away. (Without this, pointing the tool at mxqi nodes whose
+mxqd was already told to stop would dead-lock, waiting for a node that can never
+come back on its own.)
+
+mxqi nodes are auto-detected from `/etc/hostconfig` as the active `mxqd` nodes
+whose prerequisites request `mxqi` (the batch-only nodes carry `!mxqi`). Use
+`--mxqi-nodes a,b` to override the detection, `--mxqi-max N` to change the
+concurrency, or `--no-mxqi` to leave the interactive nodes untouched.
+
+## Selecting the hosts
+
+Hosts come from `hostconfig --list <expr>`, where `<expr>` is a **tag
+expression** (default `mxqd`), not a plain tag: tags are the terms, `!` negates,
+`&` and `|` combine them and `(` `)` group. **Whitespace is not an operator** —
+`--tag 'mxqd cuda'` is a syntax error, not "mxqd and cuda" — so combine tags
+explicitly and quote the expression:
+
+```sh
+./mxrolling-reboot --tag 'mxqd & cuda'          mariux-6.12.98-492
+./mxrolling-reboot --tag 'mxqd & !cuda'         mariux-6.12.98-492
+./mxrolling-reboot --tag 'nfsserver | distmaster' mariux-6.12.98-492
+```
+
+`--tag` can also be given as the `TAG` environment variable. To operate on an
+explicit list of hosts instead, use `-H a,b,c` (`--hosts`), which skips the
+hostconfig selection entirely.
+
+## Hosts without mxqd
+
+The tag expression may select any host, not just `mxqd` ones, and non-cluster
+hosts have tags too. Selected hosts that `hostconfig --list mxqd` does not list
+are handled as a third pool:
+
+* mxqd does not run there, so there is nothing to drain: no
+  `mxqdctl-hostconfig stop`, no wait for `mxqd` to vanish, and no one-time start
+  unit either — the detached script is just `mxgrub <label>` + reboot;
+* they are rebooted **strictly one after another** — the next one is only
+  started once the previous is back on the target kernel. `--plain-max N` raises
+  the concurrency, `--no-plain` leaves them untouched.
+
+The load-aware pacing therefore only ever applies to `mxqd` nodes, and a run
+that selects no `mxqd` node at all does not query the mxq database.
+
+Which hosts run mxqd is read from `hostconfig --list mxqd`; if that yields no
+`mxqd` node at all the tool refuses to run rather than treat the whole selection
+as mxqd-free (which would reboot busy nodes without draining them). Override the
+detection with `--mxqd-nodes a,b`, or state that none of the hosts runs mxqd with
+`--mxqd-nodes ''`.
+
+## Usage
+
+```sh
+# kexec everything onto the new kernel, default pacing:
+./mxrolling-reboot mariux-6.12.98-492
+
+# full reboot (needed when a firmware change must also take effect):
+./mxrolling-reboot --normal mariux-6.12.98-492
+
+# see the plan and the first pacing decision without changing anything:
+./mxrolling-reboot --dry-run mariux-6.12.98-492
+
+# just a few named hosts, at most 4 busy ones at a time:
+./mxrolling-reboot -H dingo,dugong,echidna --busy-max 4 mariux-6.12.98-492
+
+# a non-cluster tag: no drain, one host after another:
+./mxrolling-reboot --tag nfsserver mariux-6.12.98-492
+
+# a tag expression - note '&', whitespace is not an operator:
+./mxrolling-reboot --tag 'mxqd & cuda' mariux-6.12.98-492
+```
+
+Run `./mxrolling-reboot --help` for all options (`--reboot-mode`, `--no-start`,
+`--busy-max`, `--idle-fraction`, `--mxqi-max`, `--no-mxqi`, `--mxqi-nodes`,
+`--mxqd-nodes`, `--plain-max`, `--no-plain`, `--poll-interval`, `--tag`,
+`-H`/`--hosts`, `--ssh-user`).
+
+## Requirements
+
+Run it from a host that has `hostconfig`, `/etc/hostconfig`, `mysql`, `ssh` and
+`awk`, with read access to the mxq database through the MySQL config
+`/etc/mxq/mysql_ro.cnf` (override with `MXQ_DB_CNF`; only needed when mxqd nodes
+are selected), and that can run the privileged host commands — either by SSHing
+in as `root` (`--ssh-user root`) or with passwordless `sudo` on the hosts for
+`mxqdctl-hostconfig`, `mxgrub`, `systemctl` and `systemd-run`.
+
+## Kernel label mapping
+
+```
+mariux-<ver>-<num>  ->  uname -r <ver>.mx64.<num>  ->  image bzImage-<ver>.mx64.<num>
+mariux-6.12.98-492  ->  6.12.98.mx64.492           ->  bzImage-6.12.98.mx64.492
+```