From beae30cadbed09e79047c959002ee6e65001868e Mon Sep 17 00:00:00 2001
From: Paul Menzel <pmenzel@molgen.mpg.de>
Date: Wed, 29 Jul 2026 08:54:33 +0200
Subject: [PATCH 1/3] mxrolling-reboot: add load-aware rolling kernel reboot
 for mxqd nodes

Reboot mxqd cluster nodes into a new kernel without killing running jobs
and without draining the whole cluster at once. For each outdated node it
drains mxqd (stop, then wait for the mxqd process to exit), installs a
one-time systemd unit that runs mxqdctl-hostconfig start on the next boot
and removes itself again, selects the kernel with mxgrub and reboots
(kexec by default, or a normal reboot for firmware changes), all detached
via systemd-run so long drains survive an SSH disconnect.

Pacing is load-aware from the mxq server status page: at most BUSY_MAX
nodes (default 2) when the cluster is fully busy, otherwise up to half of
the currently idle nodes. Nodes already on the target kernel are detected
via uname -r and skipped, so re-running is safe.

The interactive (mxqi) nodes are handled as a separate, throttled pool:
at most MXQI_MAX of them (default 1) are upgraded at a time and at least
MXQI_MIN_UP (default 1) is always kept up and serving, so interactive
sessions can always be started. mxqi nodes are auto-detected from
/etc/hostconfig as the mxqd nodes whose prerequisites request 'mxqi'.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 mxq-rolling-reboot/README.md        | 104 ++++++
 mxq-rolling-reboot/mxrolling-reboot | 504 ++++++++++++++++++++++++++++
 2 files changed, 608 insertions(+)
 create mode 100644 mxq-rolling-reboot/README.md
 create mode 100755 mxq-rolling-reboot/mxrolling-reboot

diff --git a/mxq-rolling-reboot/README.md b/mxq-rolling-reboot/README.md
new file mode 100644
index 0000000..f467e50
--- /dev/null
+++ b/mxq-rolling-reboot/README.md
@@ -0,0 +1,104 @@
+# mxrolling-reboot
+
+Rolling kernel reboot of the `mxqd` cluster nodes: reboots every node into a
+new kernel **without killing running jobs** and **without draining the whole
+cluster at once**.
+
+## What it does
+
+For each node still running the old kernel it launches, detached via
+`systemd-run`, the sequence:
+
+1. `mxqdctl-hostconfig stop` — mxqd stops accepting new jobs and exits once its
+   running jobs finished.
+2. wait for the `mxqd` process(es) to disappear — i.e. wait (indefinitely) for
+   the node to be fully drained. `mxqdctl-hostconfig stop` only *signals* mxqd
+   and returns immediately, so this explicit wait is what makes the reboot
+   safe.
+3. install a one-time systemd unit that runs `mxqdctl-hostconfig start` on the
+   next boot and then removes itself again (see below).
+4. `mxgrub <label>` — select the requested kernel for the next boot.
+5. reboot into it — `mxgrub --reboot` (kexec, default) or `systemctl reboot`
+   (full firmware/POST cycle).
+
+Because the work runs in a transient systemd unit on the node, a drain that
+takes as long as a job's max walltime survives an SSH disconnect and even
+survives this orchestrator being interrupted.
+
+Nodes already on the target kernel are detected with `uname -r` and skipped, so
+the tool is safe to re-run to pick up nodes that were still busy earlier.
+
+## Starting mxqd again after the reboot
+
+mxqd is not started automatically at boot, so the tool installs a one-time unit
+`mxq-rolling-reboot-start.service` on the node before rebooting. On the next
+boot it runs `mxqdctl-hostconfig start`, then disables and deletes itself, so it
+leaves no permanent change on the node. `KillMode=process` keeps the daemonized
+mxqd alive after the one-shot unit finishes. Pass `--no-start` to skip this and
+leave the node without mxqd running after the reboot.
+
+## Pacing
+
+How many nodes are upgraded concurrently is load-aware and recomputed on every
+poll, using the idle/busy state from the mxq server status page
+(`http://afk.molgen.mpg.de/mxq/mxq/server`):
+
+* cluster busy at maximum (no idle nodes) → at most `BUSY_MAX` nodes at a time
+  (default **2**);
+* otherwise → up to `IDLE_FRACTION` of the currently idle nodes at a time
+  (default **50%**).
+
+Idle nodes are always preferred when choosing what to take down next, so busy
+nodes keep serving as long as possible.
+
+## Interactive (mxqi) nodes
+
+The interactive `mxqi` nodes run the interactive sshd sessions, and those
+sessions do not drain quickly. There are only a couple of them, so draining
+them all at once means nobody can start an mxqi session. They are therefore
+handled as a **separate, throttled pool**:
+
+* at most `MXQI_MAX` mxqi nodes are upgraded at a time (default **1**);
+* at least `MXQI_MIN_UP` mxqi nodes (default **1**) are always kept up **and
+  running mxqd** — the next mxqi node is only drained once the previous one is
+  back on the new kernel *and* serving sessions again (checked via the server
+  status page, not just "kernel is new").
+
+mxqi nodes are auto-detected from `/etc/hostconfig` as the active `mxqd` nodes
+whose prerequisites request `mxqi` (the batch-only nodes carry `!mxqi`). Use
+`--mxqi-nodes a,b` to override the detection, `--mxqi-max N` to change the
+concurrency, or `--no-mxqi` to leave the interactive nodes untouched.
+
+## Usage
+
+```sh
+# kexec everything onto the new kernel, default pacing:
+./mxrolling-reboot mariux-6.12.98-492
+
+# full reboot (needed when a firmware change must also take effect):
+./mxrolling-reboot --normal mariux-6.12.98-492
+
+# see the plan and the first pacing decision without changing anything:
+./mxrolling-reboot --dry-run mariux-6.12.98-492
+
+# just a few named nodes, at most 4 busy ones at a time:
+./mxrolling-reboot --nodes dingo,dugong,echidna --busy-max 4 mariux-6.12.98-492
+```
+
+Run `./mxrolling-reboot --help` for all options (`--reboot-mode`, `--no-start`,
+`--busy-max`, `--idle-fraction`, `--mxqi-max`, `--no-mxqi`, `--mxqi-nodes`,
+`--poll-interval`, `--tag`, `--nodes`, `--ssh-user`).
+
+## Requirements
+
+Run it from a host that has `hostconfig`, `curl`, `ssh` and `awk`, and that can
+run the privileged node commands — either by SSHing in as `root`
+(`--ssh-user root`) or with passwordless `sudo` on the nodes for
+`mxqdctl-hostconfig`, `mxgrub`, `systemctl` and `systemd-run`.
+
+## Kernel label mapping
+
+```
+mariux-<ver>-<num>  ->  uname -r <ver>.mx64.<num>  ->  image bzImage-<ver>.mx64.<num>
+mariux-6.12.98-492  ->  6.12.98.mx64.492           ->  bzImage-6.12.98.mx64.492
+```
diff --git a/mxq-rolling-reboot/mxrolling-reboot b/mxq-rolling-reboot/mxrolling-reboot
new file mode 100755
index 0000000..d1492ca
--- /dev/null
+++ b/mxq-rolling-reboot/mxrolling-reboot
@@ -0,0 +1,504 @@
+#!/bin/bash
+#
+# mxrolling-reboot - rolling kernel reboot of the mxqd cluster nodes
+#
+# Reboots every mxqd node into a new kernel without killing running jobs and
+# without taking the whole cluster offline at once. For each node it:
+#
+#   1. tells mxqd to stop accepting new jobs and to exit once running jobs
+#      finished   (mxqdctl-hostconfig stop)
+#   2. waits (indefinitely) for the mxqd process(es) to actually vanish, i.e.
+#      for the node to be fully drained
+#   3. installs a one-time systemd unit that runs 'mxqdctl-hostconfig start'
+#      on the next boot and then deletes itself again (mxqd is not started at
+#      boot automatically, so this replaces the otherwise manual start)
+#   4. selects the requested kernel for the next boot   (mxgrub <label>)
+#   5. reboots into it   (kexec via 'mxgrub --reboot', or a normal reboot)
+#
+# The whole sequence is launched detached on the node via 'systemd-run', so a
+# drain that takes as long as a job's max walltime survives an SSH disconnect
+# (and even survives this orchestrator being interrupted).
+#
+# Pacing (how many nodes are upgraded concurrently) is load-aware and
+# recomputed every poll:
+#
+#   * if the cluster is busy at maximum (no idle mxqd nodes) -> at most
+#     BUSY_MAX nodes at a time (default 2)
+#   * otherwise                                              -> up to
+#     IDLE_FRACTION of the currently idle nodes at a time (default 50%)
+#
+# The interactive (mxqi) nodes are handled as a separate, throttled pool: at
+# most MXQI_MAX of them are upgraded at a time (default 1) and at least
+# MXQI_MIN_UP of them (default 1) is always kept up and running mxqd, so
+# interactive sessions can always be started somewhere. They are auto-detected
+# from /etc/hostconfig as the active mxqd nodes whose prerequisites request
+# 'mxqi' (the batch-only nodes carry '!mxqi').
+#
+# Idle/busy state comes from the mxq server status page; the running kernel of
+# a node is read with 'uname -r' over SSH, so nodes already on the target
+# kernel are skipped and re-running the tool is safe.
+#
+# Requirements on the host running this script:
+#   * hostconfig, curl, ssh, awk
+#   * SSH access to the nodes and the right to run the privileged commands
+#     there, either by SSHing in as root (SSH_USER=root) or with passwordless
+#     sudo for mxqdctl-hostconfig, mxgrub, systemctl and systemd-run.
+#
+# This tool encodes knowledge of our current setup (mxqd tag, server URL, the
+# mariux-<ver>-<num> kernel label scheme) and is not meant to be portable.
+
+set -u
+
+# ---------------------------------------------------------------------------
+# Configuration (override via environment or the command line flags below)
+# ---------------------------------------------------------------------------
+TAG=${TAG:-mxqd}                                        # hostconfig tag selecting the nodes
+SERVER_URL=${SERVER_URL:-http://afk.molgen.mpg.de/mxq/mxq/server}
+BUSY_MAX=${BUSY_MAX:-2}                                 # concurrency when the cluster is fully busy
+IDLE_FRACTION=${IDLE_FRACTION:-0.5}                     # fraction of idle nodes to take at once otherwise
+MXQI_MAX=${MXQI_MAX:-1}                                 # max interactive (mxqi) nodes to upgrade at once
+MXQI_MIN_UP=${MXQI_MIN_UP:-1}                           # keep at least this many mxqi nodes serving
+DO_MXQI=${DO_MXQI:-1}                                   # 1 = also upgrade the interactive (mxqi) nodes
+REBOOT_MODE=${REBOOT_MODE:-kexec}                       # kexec | normal
+START_AFTER=${START_AFTER:-1}                           # 1 = start mxqd after reboot via a one-time unit
+POLL_INTERVAL=${POLL_INTERVAL:-30}                      # seconds between status polls
+SCAN_PARALLEL=${SCAN_PARALLEL:-16}                      # parallelism for the initial kernel scan
+SSH_USER=${SSH_USER:-}                                  # empty = current user; e.g. root
+SSH_OPTS=${SSH_OPTS:-"-o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=accept-new"}
+
+DRY_RUN=0
+NODES_OVERRIDE=""
+MXQI_OVERRIDE=""
+
+PROG=${0##*/}
+
+usage() {
+	cat >&2 <<EOF
+usage: $PROG [options] <kernel-label>
+
+  <kernel-label>   target kernel, e.g. mariux-6.12.98-492
+
+options:
+  --reboot-mode kexec|normal   how to reboot (default: $REBOOT_MODE)
+                               use 'normal' when a firmware update / firmware
+                               change must also take effect, 'kexec' otherwise
+  --normal                     shorthand for --reboot-mode normal
+  --kexec                      shorthand for --reboot-mode kexec
+  --no-start                   do not start mxqd after the reboot (default is to
+                               start it via a one-time, self-removing unit)
+  --busy-max N                 max concurrent upgrades when the cluster is
+                               fully busy (default: $BUSY_MAX)
+  --idle-fraction F            fraction of idle nodes to upgrade at once
+                               otherwise (default: $IDLE_FRACTION)
+  --mxqi-max N                 max interactive (mxqi) nodes to upgrade at once
+                               (default: $MXQI_MAX); mxqi nodes are always kept
+                               to at least $MXQI_MIN_UP serving node(s)
+  --no-mxqi                    do not upgrade the interactive (mxqi) nodes
+  --mxqi-nodes a,b             treat exactly these as the interactive nodes
+                               instead of auto-detecting them from /etc/hostconfig
+  --poll-interval S            seconds between status polls (default: $POLL_INTERVAL)
+  --tag TAG                    hostconfig tag selecting nodes (default: $TAG)
+  --nodes a,b,c                operate on exactly these nodes instead of the tag
+  --ssh-user USER              SSH as USER (default: current user; 'root' skips sudo)
+  --dry-run                    show the plan and the decisions, change nothing
+  -h, --help                   this help
+
+The target kernel label maps to 'uname -r' as
+  mariux-<ver>-<num>  ->  <ver>.mx64.<num>
+e.g. mariux-6.12.98-492 -> 6.12.98.mx64.492 (image bzImage-6.12.98.mx64.492).
+EOF
+	exit "${1:-2}"
+}
+
+# ---------------------------------------------------------------------------
+# Argument parsing
+# ---------------------------------------------------------------------------
+KERNEL=""
+while [ $# -gt 0 ]; do
+	case "$1" in
+		--reboot-mode) REBOOT_MODE=$2; shift 2 ;;
+		--normal)      REBOOT_MODE=normal; shift ;;
+		--kexec)       REBOOT_MODE=kexec; shift ;;
+		--no-start)    START_AFTER=0; shift ;;
+		--busy-max)    BUSY_MAX=$2; shift 2 ;;
+		--idle-fraction) IDLE_FRACTION=$2; shift 2 ;;
+		--mxqi-max)    MXQI_MAX=$2; shift 2 ;;
+		--no-mxqi)     DO_MXQI=0; shift ;;
+		--mxqi-nodes)  MXQI_OVERRIDE=$2; shift 2 ;;
+		--poll-interval) POLL_INTERVAL=$2; shift 2 ;;
+		--tag)         TAG=$2; shift 2 ;;
+		--nodes)       NODES_OVERRIDE=$2; shift 2 ;;
+		--ssh-user)    SSH_USER=$2; shift 2 ;;
+		--dry-run)     DRY_RUN=1; shift ;;
+		-h|--help)     usage 0 ;;
+		--)            shift; break ;;
+		-*)            echo "$PROG: unknown option: $1" >&2; usage ;;
+		*)             [ -n "$KERNEL" ] && { echo "$PROG: unexpected argument: $1" >&2; usage; }
+		               KERNEL=$1; shift ;;
+	esac
+done
+
+[ -n "$KERNEL" ] || { echo "$PROG: missing <kernel-label>" >&2; usage; }
+
+case "$REBOOT_MODE" in
+	kexec|normal) ;;
+	*) echo "$PROG: --reboot-mode must be 'kexec' or 'normal'" >&2; exit 2 ;;
+esac
+
+# ---------------------------------------------------------------------------
+# Derive uname string and boot image from the label
+#   mariux-6.12.98-492  ->  uname 6.12.98.mx64.492 , image bzImage-6.12.98.mx64.492
+# ---------------------------------------------------------------------------
+case "$KERNEL" in
+	mariux-*-*) ;;
+	*) echo "$PROG: '$KERNEL' does not look like a mariux-<ver>-<num> label" >&2; exit 2 ;;
+esac
+rest=${KERNEL#mariux-}          # 6.12.98-492
+ver=${rest%-*}                  # 6.12.98
+num=${rest##*-}                 # 492
+UNAME_TARGET="${ver}.mx64.${num}"       # 6.12.98.mx64.492
+IMAGE="bzImage-${UNAME_TARGET}"         # bzImage-6.12.98.mx64.492
+
+if [ "$SSH_USER" = "root" ]; then
+	SUDO=""
+else
+	SUDO="sudo"
+fi
+
+case "$REBOOT_MODE" in
+	kexec)  REBOOT_CMD="mxgrub --reboot" ;;      # kexec into the chosen kernel
+	normal) REBOOT_CMD="systemctl reboot" ;;     # full firmware/POST cycle
+esac
+
+# One-time "start mxqd after reboot" unit installed on the node.
+START_UNIT_NAME="mxq-rolling-reboot-start.service"
+START_UNIT_PATH="/etc/systemd/system/$START_UNIT_NAME"
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+log() { echo "$(date '+%F %T') $PROG: $*"; }
+
+# ssh to a node with our standard options; SSH_USER prefix if set.
+rsh() {
+	local node=$1; shift
+	# shellcheck disable=SC2086
+	ssh $SSH_OPTS "${SSH_USER:+$SSH_USER@}$node" "$@"
+}
+
+# Running kernel of a node, or UNREACHABLE.
+node_kernel() {
+	local node=$1 out
+	out=$(rsh "$node" uname -r 2>/dev/null) && [ -n "$out" ] && { echo "$out"; return; }
+	echo UNREACHABLE
+}
+
+# Fetch the mxq server page and emit "host STATUS slots_in_use" per node.
+cluster_status() {
+	curl -s --max-time 30 "$SERVER_URL" | awk '
+		BEGIN { RS="</tr>" }
+		/<td>(RUNNING|IDLE)/ {
+			n = split($0, c, /<\/td>/)
+			for (i = 1; i <= n; i++) {
+				x = c[i]
+				sub(/.*<td[^>]*>/, "", x)
+				gsub(/<[^>]*>/, "", x)
+				gsub(/^[ \t\r\n]+|[ \t\r\n]+$/, "", x)
+				c[i] = x
+			}
+			# 3=status 4=host 13=slots in use
+			print c[4], c[3], c[13]
+		}'
+}
+
+# floor(IDLE_FRACTION * n)
+idle_share() { awk -v f="$IDLE_FRACTION" -v n="$1" 'BEGIN { print int(f*n) }'; }
+
+# The interactive (mxqi) nodes: active mxqd lines in /etc/hostconfig whose
+# prerequisites request 'mxqi' (as opposed to the '!mxqi' the batch-only nodes
+# carry). These run the interactive sshd sessions and must not all be drained
+# at once, or nobody can start an mxqi session.
+detect_mxqi_nodes() {
+	awk '$1 !~ /^#/ && $2=="mxqd" && /mxqi/ && !/!mxqi/ {print $1}' /etc/hostconfig
+}
+
+# Pick an idle-preferred entry from the named array, remove it from the array
+# and return it in the global PICK. Must NOT be called in a subshell ($( )), or
+# the array mutation is lost.
+PICK=""
+take_idle_pref() {
+	local -n _arr=$1
+	local i node idx=-1
+	PICK=""
+	for i in "${!_arr[@]}"; do
+		node=${_arr[$i]}
+		if [ "${STAT[$node]:-}" = "IDLE" ]; then PICK=$node; idx=$i; break; fi
+	done
+	[ -z "$PICK" ] && { idx=0; PICK=${_arr[0]}; }
+	_arr=("${_arr[@]:0:$idx}" "${_arr[@]:$((idx+1))}")
+}
+
+# Emit (on stdout) the script that runs detached on the node:
+#   [install one-time start unit] -> drain -> select kernel -> reboot.
+# It is transferred to the node and executed under systemd-run, so quoting is
+# local only; only "\$0" is meant to expand on the node (to delete this script).
+build_remote_script() {
+	echo '#!/bin/bash'
+	echo 'rm -f "$0"'          # self-delete the transferred temp script
+	echo 'set -e'
+
+	if [ "$START_AFTER" = 1 ]; then
+		# Install a one-time unit that starts mxqd on the next boot and then
+		# removes itself. KillMode=process keeps the daemonized mxqd alive
+		# after this oneshot service finishes (default control-group would
+		# kill it during cgroup cleanup).
+		cat <<EOF
+cat > $START_UNIT_PATH <<'UNIT'
+[Unit]
+Description=One-time mxqd start after mxq-rolling-reboot
+After=network-online.target remote-fs.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+KillMode=process
+ExecStart=/usr/sbin/mxqdctl-hostconfig start
+ExecStartPost=/bin/sh -c 'systemctl disable $START_UNIT_NAME; rm -f $START_UNIT_PATH; systemctl daemon-reload'
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+systemctl daemon-reload
+systemctl enable $START_UNIT_NAME
+EOF
+	fi
+
+	# Drain: stop accepting jobs, wait for running jobs to finish (mxqd exits).
+	echo 'mxqdctl-hostconfig stop'
+	echo 'while pgrep -x mxqd >/dev/null 2>&1; do sleep 10; done'
+	# Select the target kernel for the next boot and reboot into it.
+	echo "mxgrub $KERNEL"
+	echo "$REBOOT_CMD"
+}
+
+# ---------------------------------------------------------------------------
+# Build the node list
+# ---------------------------------------------------------------------------
+if [ -n "$NODES_OVERRIDE" ]; then
+	IFS=', ' read -r -a NODES <<<"$NODES_OVERRIDE"
+else
+	# shellcheck disable=SC2207
+	NODES=($(hostconfig --list "$TAG")) || { log "hostconfig --list $TAG failed"; exit 1; }
+fi
+[ "${#NODES[@]}" -gt 0 ] || { log "no nodes selected"; exit 1; }
+
+# Interactive (mxqi) nodes, intersected with the selected node set. These are
+# handled as a separate, throttled pool (see the scheduler below).
+if [ -n "$MXQI_OVERRIDE" ]; then
+	IFS=', ' read -r -a mxqi_detected <<<"$MXQI_OVERRIDE"
+else
+	# shellcheck disable=SC2207
+	mxqi_detected=($(detect_mxqi_nodes))
+fi
+declare -A IS_MXQI=()
+MXQI_ALL=()
+for n in "${NODES[@]}"; do
+	for m in "${mxqi_detected[@]:-}"; do
+		if [ "$n" = "$m" ]; then IS_MXQI[$n]=1; MXQI_ALL+=("$n"); break; fi
+	done
+done
+
+log "target kernel : $KERNEL  (uname $UNAME_TARGET, image $IMAGE)"
+log "reboot mode   : $REBOOT_MODE ($REBOOT_CMD)"
+log "nodes         : ${#NODES[@]} (tag '$TAG')"
+log "pacing        : busy-max $BUSY_MAX, idle-fraction $IDLE_FRACTION"
+if [ "$DO_MXQI" = 1 ]; then
+	log "mxqi nodes    : ${#MXQI_ALL[@]} [${MXQI_ALL[*]:-}] (max $MXQI_MAX at once, keep >= $MXQI_MIN_UP up)"
+else
+	log "mxqi nodes    : ${#MXQI_ALL[@]} [${MXQI_ALL[*]:-}] (skipped: --no-mxqi)"
+fi
+
+# ---------------------------------------------------------------------------
+# Initial scan: which nodes are already on the target kernel?
+# ---------------------------------------------------------------------------
+log "scanning current kernels (parallel $SCAN_PARALLEL) ..."
+scan_out=$(mktemp "${TMPDIR:-/tmp}/$PROG.scan.XXXXXX")
+trap 'rm -f "$scan_out"' EXIT
+for n in "${NODES[@]}"; do
+	{ printf '%s %s\n' "$n" "$(node_kernel "$n")"; } >>"$scan_out" &
+	while [ "$(jobs -r | wc -l)" -ge "$SCAN_PARALLEL" ]; do wait -n; done
+done
+wait
+
+OUTDATED=()
+DONE=()
+UNREACHABLE=()
+while read -r host kern; do
+	case "$kern" in
+		"$UNAME_TARGET") DONE+=("$host") ;;
+		UNREACHABLE)     UNREACHABLE+=("$host") ;;
+		*)               OUTDATED+=("$host") ;;
+	esac
+done < <(sort "$scan_out")
+
+# Partition the work into the regular pool and the throttled mxqi pool.
+OUTDATED_REG=()
+OUTDATED_MXQI=()
+SKIPPED_MXQI=()
+for host in "${OUTDATED[@]}"; do
+	if [ -n "${IS_MXQI[$host]:-}" ]; then
+		if [ "$DO_MXQI" = 1 ]; then OUTDATED_MXQI+=("$host"); else SKIPPED_MXQI+=("$host"); fi
+	else
+		OUTDATED_REG+=("$host")
+	fi
+done
+
+log "already on $UNAME_TARGET : ${#DONE[@]}"
+log "to upgrade (batch)      : ${#OUTDATED_REG[@]}"
+log "to upgrade (mxqi)       : ${#OUTDATED_MXQI[@]}  [${OUTDATED_MXQI[*]:-}]"
+if [ "${#SKIPPED_MXQI[@]}" -gt 0 ]; then
+	log "mxqi skipped (--no-mxqi): ${#SKIPPED_MXQI[@]}  [${SKIPPED_MXQI[*]}]"
+fi
+if [ "${#UNREACHABLE[@]}" -gt 0 ]; then
+	log "unreachable (skipped)   : ${#UNREACHABLE[@]}  [${UNREACHABLE[*]}]"
+fi
+if [ "$((${#OUTDATED_REG[@]} + ${#OUTDATED_MXQI[@]}))" -eq 0 ]; then
+	log "nothing to do."; exit 0
+fi
+
+# ---------------------------------------------------------------------------
+# Initiate the detached drain+reboot on one node.
+# ---------------------------------------------------------------------------
+initiate() {
+	local node=$1
+
+	# Pre-flight: make sure the kernel image is present, else mxgrub would
+	# fail and the node would never come back on the new kernel.
+	if ! rsh "$node" test -e "/boot/$IMAGE" 2>/dev/null; then
+		log "  $node: /boot/$IMAGE missing -> SKIP"
+		return 1
+	fi
+
+	if [ "$DRY_RUN" = 1 ]; then
+		log "  DRY-RUN would run detached on $node:"
+		build_remote_script | sed 's/^/      | /'
+		return 0
+	fi
+
+	# Transfer the detached script to a temp file on the node and launch it
+	# under systemd-run, so it survives our SSH session (and us). "\$f" is
+	# expanded on the node; $SUDO and $KERNEL are expanded here.
+	local bootstrap
+	bootstrap="f=\$(mktemp /tmp/mxq-rolling-reboot.XXXXXX) && cat > \"\$f\" && chmod 700 \"\$f\" && $SUDO systemd-run --collect --description=mxq-rolling-reboot-to-$KERNEL /bin/bash \"\$f\""
+
+	if build_remote_script | rsh "$node" "$bootstrap"; then
+		log "  $node: draining, will reboot ($REBOOT_MODE) into $KERNEL when idle"
+		return 0
+	else
+		log "  $node: failed to launch drain/reboot -> SKIP"
+		return 1
+	fi
+}
+
+# ---------------------------------------------------------------------------
+# Dry-run: show the plan and the first pacing decision, then stop.
+# ---------------------------------------------------------------------------
+if [ "$DRY_RUN" = 1 ]; then
+	idle_now=$(cluster_status | awk '$2=="IDLE"{c++} END{print c+0}')
+	if [ "$idle_now" -eq 0 ]; then limit=$BUSY_MAX; else
+		limit=$(idle_share "$idle_now"); [ "$limit" -lt 1 ] && limit=1
+	fi
+	log "DRY-RUN: ${idle_now} idle nodes now -> would upgrade up to $limit batch node(s) at a time"
+	log "DRY-RUN: batch nodes: ${OUTDATED_REG[*]:-(none)}"
+	log "DRY-RUN: mxqi nodes : ${OUTDATED_MXQI[*]:-(none)} (up to $MXQI_MAX at once, keep >= $MXQI_MIN_UP up)"
+	for n in "${OUTDATED_REG[@]}" "${OUTDATED_MXQI[@]}"; do initiate "$n"; done
+	exit 0
+fi
+
+# ---------------------------------------------------------------------------
+# Scheduler loop
+# ---------------------------------------------------------------------------
+declare -A INFLIGHT=()   # node -> reg|mxqi while draining/rebooting
+FAILED=()
+UPGRADED=()
+remaining_reg=("${OUTDATED_REG[@]}")
+remaining_mxqi=("${OUTDATED_MXQI[@]}")
+
+# Count in-flight nodes of a given kind (reg|mxqi).
+inflight_of() {
+	local kind=$1 k n=0
+	for k in "${!INFLIGHT[@]}"; do [ "${INFLIGHT[$k]}" = "$kind" ] && n=$((n+1)); done
+	echo "$n"
+}
+
+while [ "${#remaining_reg[@]}" -gt 0 ] || [ "${#remaining_mxqi[@]}" -gt 0 ] \
+      || [ "${#INFLIGHT[@]}" -gt 0 ]; do
+
+	# --- refresh cluster status -------------------------------------------
+	status=$(cluster_status)
+	declare -A STAT=()
+	while read -r host st _; do STAT[$host]=$st; done <<<"$status"
+	idle_now=$(awk '$2=="IDLE"{c++} END{print c+0}' <<<"$status")
+
+	# --- reap finished in-flight nodes ------------------------------------
+	for node in "${!INFLIGHT[@]}"; do
+		k=$(node_kernel "$node")
+		if [ "$k" = "$UNAME_TARGET" ]; then
+			log "  $node: back up on $KERNEL  [done]"
+			unset 'INFLIGHT[$node]'
+			UPGRADED+=("$node")
+		fi
+	done
+
+	# --- refill the batch pool, preferring currently-idle nodes -----------
+	if [ "$idle_now" -eq 0 ]; then
+		limit=$BUSY_MAX
+	else
+		limit=$(idle_share "$idle_now")
+		[ "$limit" -lt 1 ] && limit=1
+	fi
+	while [ "$(inflight_of reg)" -lt "$limit" ] && [ "${#remaining_reg[@]}" -gt 0 ]; do
+		take_idle_pref remaining_reg; pick=$PICK
+		if initiate "$pick"; then INFLIGHT[$pick]=reg; else FAILED+=("$pick"); fi
+	done
+
+	# --- refill the mxqi pool, keeping at least MXQI_MIN_UP nodes serving --
+	# An mxqi node counts as "serving" when it is up on the server page and not
+	# one we are draining. We only start another mxqi upgrade while enough
+	# others are still serving, so interactive users always have a node.
+	while [ "$(inflight_of mxqi)" -lt "$MXQI_MAX" ] && [ "${#remaining_mxqi[@]}" -gt 0 ]; do
+		serving=0
+		for m in "${MXQI_ALL[@]}"; do
+			[ -n "${INFLIGHT[$m]:-}" ] && continue
+			[ -n "${STAT[$m]:-}" ] && serving=$((serving+1))
+		done
+		if [ "$((serving - 1))" -lt "$MXQI_MIN_UP" ]; then
+			log "  mxqi: holding ($serving serving; keep >= $MXQI_MIN_UP) - waiting for an mxqi node to return"
+			break
+		fi
+		take_idle_pref remaining_mxqi; pick=$PICK
+		if initiate "$pick"; then INFLIGHT[$pick]=mxqi; else FAILED+=("$pick"); fi
+	done
+
+	# --- done? ------------------------------------------------------------
+	if [ "${#remaining_reg[@]}" -eq 0 ] && [ "${#remaining_mxqi[@]}" -eq 0 ] \
+	   && [ "${#INFLIGHT[@]}" -eq 0 ]; then
+		break
+	fi
+
+	log "status: in-flight ${#INFLIGHT[@]} (batch $(inflight_of reg)/$limit, mxqi $(inflight_of mxqi)/$MXQI_MAX), remaining batch ${#remaining_reg[@]} mxqi ${#remaining_mxqi[@]}, idle $idle_now, done ${#UPGRADED[@]}"
+	sleep "$POLL_INTERVAL"
+done
+
+# ---------------------------------------------------------------------------
+# Summary
+# ---------------------------------------------------------------------------
+log "finished."
+log "upgraded   : ${#UPGRADED[@]}  [${UPGRADED[*]:-}]"
+log "already ok : ${#DONE[@]}"
+[ "${#FAILED[@]}"      -gt 0 ] && log "failed     : ${#FAILED[@]}  [${FAILED[*]}]"
+[ "${#UNREACHABLE[@]}" -gt 0 ] && log "unreachable: ${#UNREACHABLE[@]}  [${UNREACHABLE[*]}]"
+if [ "${#FAILED[@]}" -gt 0 ] || [ "${#UNREACHABLE[@]}" -gt 0 ]; then
+	exit 1
+fi
+exit 0

From e227767abe2624ba14cc13bb68da57c1c549ea3a Mon Sep 17 00:00:00 2001
From: Paul Menzel <pmenzel@molgen.mpg.de>
Date: Wed, 29 Jul 2026 19:03:31 +0200
Subject: [PATCH 2/3] mxrolling-reboot: reboot already-drained mxqi nodes
 instead of dead-locking

The mxqi keep-serving guard counted only nodes serving on the status
page and then assumed the node about to be taken down was one of them
(serving - 1 >= MXQI_MIN_UP). When every mxqi node already had mxqd
stopped or draining, none appeared on the page, so 'serving' was 0 and
the guard held forever, waiting for a node that could not return on its
own.

Only protect nodes that are actually serving: prefer an mxqi node that
is already down (rebooting it removes no interactive capacity), and
apply the keep-serving throttle only when the remaining candidates are
still serving.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 mxq-rolling-reboot/README.md        | 13 ++++++++++---
 mxq-rolling-reboot/mxrolling-reboot | 30 +++++++++++++++++++++++------
 2 files changed, 34 insertions(+), 9 deletions(-)

diff --git a/mxq-rolling-reboot/README.md b/mxq-rolling-reboot/README.md
index f467e50..b7d1ff8 100644
--- a/mxq-rolling-reboot/README.md
+++ b/mxq-rolling-reboot/README.md
@@ -60,9 +60,16 @@ handled as a **separate, throttled pool**:
 
 * at most `MXQI_MAX` mxqi nodes are upgraded at a time (default **1**);
 * at least `MXQI_MIN_UP` mxqi nodes (default **1**) are always kept up **and
-  running mxqd** — the next mxqi node is only drained once the previous one is
-  back on the new kernel *and* serving sessions again (checked via the server
-  status page, not just "kernel is new").
+  running mxqd** — a still-serving mxqi node is only drained once enough others
+  are serving to honour this minimum (checked via the server status page, not
+  just "kernel is new").
+
+The keep-serving guard only protects nodes that are *actually serving*. An mxqi
+node whose mxqd is already stopped or draining shows up on neither the status
+page nor as a running daemon, so rebooting it costs no interactive capacity and
+it is upgraded right away. (Without this, pointing the tool at mxqi nodes whose
+mxqd was already told to stop would dead-lock, waiting for a node that can never
+come back on its own.)
 
 mxqi nodes are auto-detected from `/etc/hostconfig` as the active `mxqd` nodes
 whose prerequisites request `mxqi` (the batch-only nodes carry `!mxqi`). Use
diff --git a/mxq-rolling-reboot/mxrolling-reboot b/mxq-rolling-reboot/mxrolling-reboot
index d1492ca..ebe02e3 100755
--- a/mxq-rolling-reboot/mxrolling-reboot
+++ b/mxq-rolling-reboot/mxrolling-reboot
@@ -464,19 +464,37 @@ while [ "${#remaining_reg[@]}" -gt 0 ] || [ "${#remaining_mxqi[@]}" -gt 0 ] \
 
 	# --- refill the mxqi pool, keeping at least MXQI_MIN_UP nodes serving --
 	# An mxqi node counts as "serving" when it is up on the server page and not
-	# one we are draining. We only start another mxqi upgrade while enough
-	# others are still serving, so interactive users always have a node.
+	# one we are draining. The guard only protects still-serving nodes: taking
+	# down a node whose mxqd is already stopped/draining (not on the page) costs
+	# no interactive capacity, so such a node is always eligible - otherwise the
+	# tool would dead-lock when every mxqi node is already down.
 	while [ "$(inflight_of mxqi)" -lt "$MXQI_MAX" ] && [ "${#remaining_mxqi[@]}" -gt 0 ]; do
 		serving=0
 		for m in "${MXQI_ALL[@]}"; do
 			[ -n "${INFLIGHT[$m]:-}" ] && continue
 			[ -n "${STAT[$m]:-}" ] && serving=$((serving+1))
 		done
-		if [ "$((serving - 1))" -lt "$MXQI_MIN_UP" ]; then
-			log "  mxqi: holding ($serving serving; keep >= $MXQI_MIN_UP) - waiting for an mxqi node to return"
-			break
+
+		# Choose the next mxqi node to take down. Prefer nodes that are already
+		# not serving (free to reboot), then idle serving nodes; only touch a
+		# still-serving node while enough others remain to honour MXQI_MIN_UP.
+		pick=""; pick_idx=-1
+		for i in "${!remaining_mxqi[@]}"; do          # 1) an already-down node
+			m=${remaining_mxqi[$i]}
+			[ -z "${STAT[$m]:-}" ] && { pick=$m; pick_idx=$i; break; }
+		done
+		if [ -z "$pick" ]; then                       # 2) else a serving node
+			if [ "$((serving - 1))" -lt "$MXQI_MIN_UP" ]; then
+				log "  mxqi: holding ($serving serving; keep >= $MXQI_MIN_UP) - waiting for an mxqi node to return"
+				break
+			fi
+			for i in "${!remaining_mxqi[@]}"; do  # prefer an idle one
+				m=${remaining_mxqi[$i]}
+				[ "${STAT[$m]:-}" = "IDLE" ] && { pick=$m; pick_idx=$i; break; }
+			done
+			[ -z "$pick" ] && { pick_idx=0; pick=${remaining_mxqi[0]}; }
 		fi
-		take_idle_pref remaining_mxqi; pick=$PICK
+		remaining_mxqi=("${remaining_mxqi[@]:0:$pick_idx}" "${remaining_mxqi[@]:$((pick_idx+1))}")
 		if initiate "$pick"; then INFLIGHT[$pick]=mxqi; else FAILED+=("$pick"); fi
 	done
 

From a3dc7d6dae89cb19e7a7b28f21b756e74687c6e0 Mon Sep 17 00:00:00 2001
From: Paul Menzel <pmenzel@molgen.mpg.de>
Date: Wed, 29 Jul 2026 19:33:13 +0200
Subject: [PATCH 3/3] mxrolling-reboot: read idle/busy state from the mxq
 database

Replace the HTML scrape of the mxq server status page with a direct
query of the mxq_daemon table via /etc/mxq/mysql_ro.cnf. This is a
machine-readable, authoritative source and drops the curl dependency.

Take the newest daemon row per host and keep only daemons that are
actually accepting jobs (status 0 = idle, 10 = running); a draining
daemon (status 200) or a gone one (>= 250) is left out, so the
'serving' notion the mxqi keep-serving guard relies on stays correct.
Rows not refreshed within DB_FRESH_MIN minutes are ignored so a daemon
that vanished without recording a stop is not mistaken for live.

Skip a poll when the query fails instead of acting on an empty result,
which would otherwise look like 'nothing is serving' and could wrongly
free the mxqi guard.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 mxq-rolling-reboot/README.md        | 23 +++++-----
 mxq-rolling-reboot/mxrolling-reboot | 66 ++++++++++++++++++-----------
 2 files changed, 53 insertions(+), 36 deletions(-)

diff --git a/mxq-rolling-reboot/README.md b/mxq-rolling-reboot/README.md
index b7d1ff8..a656d7b 100644
--- a/mxq-rolling-reboot/README.md
+++ b/mxq-rolling-reboot/README.md
@@ -40,8 +40,8 @@ leave the node without mxqd running after the reboot.
 ## Pacing
 
 How many nodes are upgraded concurrently is load-aware and recomputed on every
-poll, using the idle/busy state from the mxq server status page
-(`http://afk.molgen.mpg.de/mxq/mxq/server`):
+poll, using the idle/busy state read from the mxq database (the `mxq_daemon`
+table, via the MySQL config `/etc/mxq/mysql_ro.cnf`):
 
 * cluster busy at maximum (no idle nodes) → at most `BUSY_MAX` nodes at a time
   (default **2**);
@@ -61,13 +61,13 @@ handled as a **separate, throttled pool**:
 * at most `MXQI_MAX` mxqi nodes are upgraded at a time (default **1**);
 * at least `MXQI_MIN_UP` mxqi nodes (default **1**) are always kept up **and
   running mxqd** — a still-serving mxqi node is only drained once enough others
-  are serving to honour this minimum (checked via the server status page, not
-  just "kernel is new").
+  are serving to honour this minimum (checked via the mxq database, not just
+  "kernel is new").
 
 The keep-serving guard only protects nodes that are *actually serving*. An mxqi
-node whose mxqd is already stopped or draining shows up on neither the status
-page nor as a running daemon, so rebooting it costs no interactive capacity and
-it is upgraded right away. (Without this, pointing the tool at mxqi nodes whose
+node whose mxqd is already stopped or draining is not registered as an active
+daemon in the database, so rebooting it costs no interactive capacity and it is
+upgraded right away. (Without this, pointing the tool at mxqi nodes whose
 mxqd was already told to stop would dead-lock, waiting for a node that can never
 come back on its own.)
 
@@ -98,10 +98,11 @@ Run `./mxrolling-reboot --help` for all options (`--reboot-mode`, `--no-start`,
 
 ## Requirements
 
-Run it from a host that has `hostconfig`, `curl`, `ssh` and `awk`, and that can
-run the privileged node commands — either by SSHing in as `root`
-(`--ssh-user root`) or with passwordless `sudo` on the nodes for
-`mxqdctl-hostconfig`, `mxgrub`, `systemctl` and `systemd-run`.
+Run it from a host that has `hostconfig`, `mysql`, `ssh` and `awk`, with read
+access to the mxq database through the MySQL config `/etc/mxq/mysql_ro.cnf`
+(override with `MXQ_DB_CNF`), and that can run the privileged node commands —
+either by SSHing in as `root` (`--ssh-user root`) or with passwordless `sudo` on
+the nodes for `mxqdctl-hostconfig`, `mxgrub`, `systemctl` and `systemd-run`.
 
 ## Kernel label mapping
 
diff --git a/mxq-rolling-reboot/mxrolling-reboot b/mxq-rolling-reboot/mxrolling-reboot
index ebe02e3..3af2395 100755
--- a/mxq-rolling-reboot/mxrolling-reboot
+++ b/mxq-rolling-reboot/mxrolling-reboot
@@ -34,12 +34,14 @@
 # from /etc/hostconfig as the active mxqd nodes whose prerequisites request
 # 'mxqi' (the batch-only nodes carry '!mxqi').
 #
-# Idle/busy state comes from the mxq server status page; the running kernel of
-# a node is read with 'uname -r' over SSH, so nodes already on the target
-# kernel are skipped and re-running the tool is safe.
+# Idle/busy state comes from the mxq database (the mxq_daemon table, read via
+# /etc/mxq/mysql_ro.cnf); the running kernel of a node is read with 'uname -r'
+# over SSH, so nodes already on the target kernel are skipped and re-running the
+# tool is safe.
 #
 # Requirements on the host running this script:
-#   * hostconfig, curl, ssh, awk
+#   * hostconfig, mysql, ssh, awk
+#   * read access to the mxq database via the MySQL config MXQ_DB_CNF
 #   * SSH access to the nodes and the right to run the privileged commands
 #     there, either by SSHing in as root (SSH_USER=root) or with passwordless
 #     sudo for mxqdctl-hostconfig, mxgrub, systemctl and systemd-run.
@@ -53,7 +55,8 @@ set -u
 # Configuration (override via environment or the command line flags below)
 # ---------------------------------------------------------------------------
 TAG=${TAG:-mxqd}                                        # hostconfig tag selecting the nodes
-SERVER_URL=${SERVER_URL:-http://afk.molgen.mpg.de/mxq/mxq/server}
+MXQ_DB_CNF=${MXQ_DB_CNF:-/etc/mxq/mysql_ro.cnf}         # MySQL client config for the mxq database
+DB_FRESH_MIN=${DB_FRESH_MIN:-15}                        # ignore daemon rows not updated within this many minutes
 BUSY_MAX=${BUSY_MAX:-2}                                 # concurrency when the cluster is fully busy
 IDLE_FRACTION=${IDLE_FRACTION:-0.5}                     # fraction of idle nodes to take at once otherwise
 MXQI_MAX=${MXQI_MAX:-1}                                 # max interactive (mxqi) nodes to upgrade at once
@@ -193,22 +196,26 @@ node_kernel() {
 	echo UNREACHABLE
 }
 
-# Fetch the mxq server page and emit "host STATUS slots_in_use" per node.
+# Emit "host STATUS slots_in_use" for every mxqd currently registered and
+# accepting jobs, read straight from the mxq database. Only actually-serving
+# daemons are emitted: status 0 = idle, 10 = running jobs; a daemon that is
+# draining (status 200, told to stop) or gone (>=250) is left out, matching the
+# old status-page scrape (which only listed IDLE/RUNNING). We take the newest
+# daemon row per host and ignore rows not refreshed within DB_FRESH_MIN minutes,
+# so a daemon that vanished without recording a stop is not mistaken for live.
+# Returns non-zero if the database cannot be queried, so the caller can skip the
+# poll instead of acting on missing data.
 cluster_status() {
-	curl -s --max-time 30 "$SERVER_URL" | awk '
-		BEGIN { RS="</tr>" }
-		/<td>(RUNNING|IDLE)/ {
-			n = split($0, c, /<\/td>/)
-			for (i = 1; i <= n; i++) {
-				x = c[i]
-				sub(/.*<td[^>]*>/, "", x)
-				gsub(/<[^>]*>/, "", x)
-				gsub(/^[ \t\r\n]+|[ \t\r\n]+$/, "", x)
-				c[i] = x
-			}
-			# 3=status 4=host 13=slots in use
-			print c[4], c[3], c[13]
-		}'
+	mysql --defaults-file="$MXQ_DB_CNF" -B -N -e "
+		SELECT SUBSTRING_INDEX(d.hostname, '.', 1),
+		       IF(d.daemon_slots_running = 0, 'IDLE', 'RUNNING'),
+		       d.daemon_slots_running
+		FROM mxq_daemon d
+		JOIN (SELECT hostname, MAX(daemon_id) mid
+		        FROM mxq_daemon GROUP BY hostname) t ON d.daemon_id = t.mid
+		WHERE d.status IN (0, 10)
+		  AND d.mtime > NOW() - INTERVAL $DB_FRESH_MIN MINUTE
+		ORDER BY 1;"
 }
 
 # floor(IDLE_FRACTION * n)
@@ -404,7 +411,10 @@ initiate() {
 # Dry-run: show the plan and the first pacing decision, then stop.
 # ---------------------------------------------------------------------------
 if [ "$DRY_RUN" = 1 ]; then
-	idle_now=$(cluster_status | awk '$2=="IDLE"{c++} END{print c+0}')
+	if ! status=$(cluster_status); then
+		log "cannot query the mxq database ($MXQ_DB_CNF)"; exit 1
+	fi
+	idle_now=$(awk '$2=="IDLE"{c++} END{print c+0}' <<<"$status")
 	if [ "$idle_now" -eq 0 ]; then limit=$BUSY_MAX; else
 		limit=$(idle_share "$idle_now"); [ "$limit" -lt 1 ] && limit=1
 	fi
@@ -435,9 +445,15 @@ while [ "${#remaining_reg[@]}" -gt 0 ] || [ "${#remaining_mxqi[@]}" -gt 0 ] \
       || [ "${#INFLIGHT[@]}" -gt 0 ]; do
 
 	# --- refresh cluster status -------------------------------------------
-	status=$(cluster_status)
+	# On a query failure, skip this poll rather than act on missing data: an
+	# empty status would look like "nothing is serving" and could wrongly free
+	# the mxqi keep-serving guard.
+	if ! status=$(cluster_status); then
+		log "cluster status query failed ($MXQ_DB_CNF) - skipping this poll"
+		sleep "$POLL_INTERVAL"; continue
+	fi
 	declare -A STAT=()
-	while read -r host st _; do STAT[$host]=$st; done <<<"$status"
+	while read -r host st _; do [ -n "$host" ] && STAT[$host]=$st; done <<<"$status"
 	idle_now=$(awk '$2=="IDLE"{c++} END{print c+0}' <<<"$status")
 
 	# --- reap finished in-flight nodes ------------------------------------
@@ -463,9 +479,9 @@ while [ "${#remaining_reg[@]}" -gt 0 ] || [ "${#remaining_mxqi[@]}" -gt 0 ] \
 	done
 
 	# --- refill the mxqi pool, keeping at least MXQI_MIN_UP nodes serving --
-	# An mxqi node counts as "serving" when it is up on the server page and not
+	# An mxqi node counts as "serving" when it is registered in the database and not
 	# one we are draining. The guard only protects still-serving nodes: taking
-	# down a node whose mxqd is already stopped/draining (not on the page) costs
+	# down a node whose mxqd is already stopped/draining (not registered) costs
 	# no interactive capacity, so such a node is always eligible - otherwise the
 	# tool would dead-lock when every mxqi node is already down.
 	while [ "$(inflight_of mxqi)" -lt "$MXQI_MAX" ] && [ "${#remaining_mxqi[@]}" -gt 0 ]; do