Add mxq rolling reboot

mxq-rolling-reboot/README.md

@@ -0,0 +1,112 @@
+# mxrolling-reboot
+
+Rolling kernel reboot of the `mxqd` cluster nodes: reboots every node into a
+new kernel **without killing running jobs** and **without draining the whole
+cluster at once**.
+
+## What it does
+
+For each node still running the old kernel it launches, detached via
+`systemd-run`, the sequence:
+
+1. `mxqdctl-hostconfig stop` — mxqd stops accepting new jobs and exits once its
+   running jobs finished.
+2. wait for the `mxqd` process(es) to disappear — i.e. wait (indefinitely) for
+   the node to be fully drained. `mxqdctl-hostconfig stop` only *signals* mxqd
+   and returns immediately, so this explicit wait is what makes the reboot
+   safe.
+3. install a one-time systemd unit that runs `mxqdctl-hostconfig start` on the
+   next boot and then removes itself again (see below).
+4. `mxgrub <label>` — select the requested kernel for the next boot.
+5. reboot into it — `mxgrub --reboot` (kexec, default) or `systemctl reboot`
+   (full firmware/POST cycle).
+
+Because the work runs in a transient systemd unit on the node, a drain that
+takes as long as a job's max walltime survives an SSH disconnect and even
+survives this orchestrator being interrupted.
+
+Nodes already on the target kernel are detected with `uname -r` and skipped, so
+the tool is safe to re-run to pick up nodes that were still busy earlier.
+
+## Starting mxqd again after the reboot
+
+mxqd is not started automatically at boot, so the tool installs a one-time unit
+`mxq-rolling-reboot-start.service` on the node before rebooting. On the next
+boot it runs `mxqdctl-hostconfig start`, then disables and deletes itself, so it
+leaves no permanent change on the node. `KillMode=process` keeps the daemonized
+mxqd alive after the one-shot unit finishes. Pass `--no-start` to skip this and
+leave the node without mxqd running after the reboot.
+
+## Pacing
+
+How many nodes are upgraded concurrently is load-aware and recomputed on every
+poll, using the idle/busy state read from the mxq database (the `mxq_daemon`
+table, via the MySQL config `/etc/mxq/mysql_ro.cnf`):
+
+* cluster busy at maximum (no idle nodes) → at most `BUSY_MAX` nodes at a time
+  (default **2**);
+* otherwise → up to `IDLE_FRACTION` of the currently idle nodes at a time
+  (default **50%**).
+
+Idle nodes are always preferred when choosing what to take down next, so busy
+nodes keep serving as long as possible.
+
+## Interactive (mxqi) nodes
+
+The interactive `mxqi` nodes run the interactive sshd sessions, and those
+sessions do not drain quickly. There are only a couple of them, so draining
+them all at once means nobody can start an mxqi session. They are therefore
+handled as a **separate, throttled pool**:
+
+* at most `MXQI_MAX` mxqi nodes are upgraded at a time (default **1**);
+* at least `MXQI_MIN_UP` mxqi nodes (default **1**) are always kept up **and
+  running mxqd** — a still-serving mxqi node is only drained once enough others
+  are serving to honour this minimum (checked via the mxq database, not just
+  "kernel is new").
+
+The keep-serving guard only protects nodes that are *actually serving*. An mxqi
+node whose mxqd is already stopped or draining is not registered as an active
+daemon in the database, so rebooting it costs no interactive capacity and it is
+upgraded right away. (Without this, pointing the tool at mxqi nodes whose
+mxqd was already told to stop would dead-lock, waiting for a node that can never
+come back on its own.)
+
+mxqi nodes are auto-detected from `/etc/hostconfig` as the active `mxqd` nodes
+whose prerequisites request `mxqi` (the batch-only nodes carry `!mxqi`). Use
+`--mxqi-nodes a,b` to override the detection, `--mxqi-max N` to change the
+concurrency, or `--no-mxqi` to leave the interactive nodes untouched.
+
+## Usage
+
+```sh
+# kexec everything onto the new kernel, default pacing:
+./mxrolling-reboot mariux-6.12.98-492
+
+# full reboot (needed when a firmware change must also take effect):
+./mxrolling-reboot --normal mariux-6.12.98-492
+
+# see the plan and the first pacing decision without changing anything:
+./mxrolling-reboot --dry-run mariux-6.12.98-492
+
+# just a few named nodes, at most 4 busy ones at a time:
+./mxrolling-reboot --nodes dingo,dugong,echidna --busy-max 4 mariux-6.12.98-492
+```
+
+Run `./mxrolling-reboot --help` for all options (`--reboot-mode`, `--no-start`,
+`--busy-max`, `--idle-fraction`, `--mxqi-max`, `--no-mxqi`, `--mxqi-nodes`,
+`--poll-interval`, `--tag`, `--nodes`, `--ssh-user`).
+
+## Requirements
+
+Run it from a host that has `hostconfig`, `mysql`, `ssh` and `awk`, with read
+access to the mxq database through the MySQL config `/etc/mxq/mysql_ro.cnf`
+(override with `MXQ_DB_CNF`), and that can run the privileged node commands —
+either by SSHing in as `root` (`--ssh-user root`) or with passwordless `sudo` on
+the nodes for `mxqdctl-hostconfig`, `mxgrub`, `systemctl` and `systemd-run`.
+
+## Kernel label mapping
+
+```
+mariux-<ver>-<num>  ->  uname -r <ver>.mx64.<num>  ->  image bzImage-<ver>.mx64.<num>
+mariux-6.12.98-492  ->  6.12.98.mx64.492           ->  bzImage-6.12.98.mx64.492
+```