[RFC] Power management

Makefile

@@ -781,3 +781,20 @@ clean: CLEAN += test_parser.o
 test_parser: test_parser.o parser.tab.o keywordset.o
 test_parser.o: parser.tab.h keywordset.h
 clean: CLEAN += parser.tab.o
+
+########################################################################
+### mxq-powerd -- node power manager (opt-in; not part of `make install`)
+
+MAN8DIR := ${MANDIR}/man8
+
+.PHONY: install-powerd
+install-powerd: powerd/mxq-powerd manpages/mxq-powerd.8
+	$(call quiet-installdir,0755,${DESTDIR}${LIBEXECDIR}/mxq)
+	$(call quiet-installdir,0755,${DESTDIR}${SYSCONFDIR}/mxq)
+	$(call quiet-installdir,0755,${DESTDIR}${MAN8DIR})
+	$(call quiet-install,0755,powerd/mxq-powerd,${DESTDIR}${LIBEXECDIR}/mxq/mxq-powerd)
+	$(call quiet-install,0644,manpages/mxq-powerd.8,${DESTDIR}${MAN8DIR}/mxq-powerd.8)
+	$(call quiet-install,0644,powerd/powerd.conf.example,${DESTDIR}${SYSCONFDIR}/mxq/powerd.conf.example)
+	$(call quiet-install,0644,powerd/nodes.conf.example,${DESTDIR}${SYSCONFDIR}/mxq/nodes.conf.example)
+	$(call quiet-install,0600,powerd/powerd-secrets.example,${DESTDIR}${SYSCONFDIR}/mxq/powerd-secrets.example)
+	@echo "  NOTE  copy powerd/mxq-powerd.{service,timer} to /etc/systemd/system/ to enable the timer"

docs/power-management-design.md

@@ -0,0 +1,225 @@
+# Powering off idle MXQ nodes to save energy — design notes
+
+Status: **design exploration** (no code written yet). Feasibility study requested by
+Paul Menzel, 2026-07-18.
+
+## 1. The question
+
+Can MXQ power down compute nodes when the cluster is idle and power them back on
+when work arrives, to save energy? If so, what has to be built?
+
+Short answer: **yes, and mostly as a new, self-contained central component** — MXQ
+does not need to be rewritten. But the piece that Slurm gets "for free" (a central
+controller that already knows and commands every node) does **not exist in MXQ** and
+has to be created. Paul's instinct in the request is correct: an independent
+timer/cron that (a) picks idle nodes to switch off, oldest generations first, and
+(b) a second loop that watches the queue and switches nodes back on.
+
+## 2. How MXQ works today (the relevant parts)
+
+MXQ is a **pull-based** scheduler. There is a central MySQL database and autonomous
+per-node daemons; nothing ever *commands* a node.
+
+- **Three tables** (`mysql/create_tables.sql`):
+  - `mxq_group` — one row per array of identical jobs. Carries the per-job resource
+    requirements (`job_threads`, `job_memory`, `job_time`, `job_gpu`, `job_tmpdir_size`)
+    and trigger-maintained counters, notably `group_jobs_inq` (pending) and
+    `group_jobs_running`.
+  - `mxq_job` — individual jobs; placement columns `daemon_id`, `host_hostname`,
+    `job_status`, and timestamps `date_start` / `date_end`.
+  - `mxq_daemon` — **the per-node table**. One row per *daemon instance* (a fresh row
+    is INSERTed every time a daemon starts). Holds `hostname`, capacity
+    (`daemon_slots`, `daemon_memory`, `daemon_gpus_max`), live load
+    (`daemon_slots_running`, `daemon_memory_used`, …), `status`, `tags`,
+    `prerequisites`, and timestamps `mtime` / `daemon_start` / `daemon_stop`.
+
+- **The daemon (`mxqd`)** — `mxqd.c`, `mxq_daemon.c`:
+  - On start: `mxq_daemon_register()` INSERTs its row and gets an auto-increment
+    `daemon_id` (`mxq_daemon.c:48`). Stable *logical* node identity is the pair
+    `(hostname, daemon_name)`.
+  - Main loop (`mxqd.c:2715`): `sigtimedwait()` with a **20 s** poll interval. Each
+    pass it reads the global queue via `mxq_load_running_groups()`
+    (`SELECT … FROM mxq_group WHERE group_jobs_inq>0 OR group_jobs_running>0`,
+    `mxq_group.c:292`), then race-safely claims INQ jobs
+    (`UPDATE mxq_job … WHERE job_id=? AND job_status=INQ`, `mxq_job.c:311`).
+  - **Idle signal**: `update_status()` sets `mxq_daemon.status = IDLE (0)` exactly when
+    `slots_running == 0` (`mxqd.c:2595`). Status enum in `mxq_daemon.h:9`:
+    IDLE=0, RUNNING=10, WAITING=20, FULL=30, BACKFILL=40, CPUOPTIMAL=50,
+    TERMINATING=200, EXITED=250, CRASHED=255.
+  - **Liveness / heartbeat**: `mtime` is refreshed on every stats/status write, and at
+    least every 5 min even when idle (rate-limited in `mxq_daemon.c:235`). There is
+    **no fast heartbeat and no central watchdog**. A crashed node is only flagged
+    CRASHED when a *replacement* daemon starts (`recover_from_previous_crash`,
+    `mxqd.c:2487`).
+  - **Clean shutdown**: SIGTERM exits the main loop and enters a **drain loop**
+    (`mxqd.c:2762`) that starts no new jobs and waits for running jobs to finish, then
+    `mxq_daemon_shutdown()` sets `status=EXITED, daemon_stop=NOW()` (`mxq_daemon.c:151`).
+    SIGINT additionally kills running jobs; SIGQUIT/SIGUSR1 leave jobs running.
+
+- **Host bring-up today** — `mxqdctl-hostconfig.sh`: reads `/etc/hostconfig`, starts
+  one `mxqd` per matching line, and controls local daemons via signals. It **assumes
+  the host is already powered on**. There is no PXE/IPMI/WoL hook anywhere.
+
+- **Admin surface** — `mxqadmin.c` only closes/reopens groups. There is **no**
+  drain-node / remove-daemon / power command. No cron, systemd unit, or scheduler
+  callback exists in the repo (the recent `--callback` is a *per-job*, runs-as-user,
+  on-node hook — not suitable for privileged node power control).
+
+### What this gives us for free
+
+Everything a power decision needs is already observable in the DB *without touching
+the nodes*:
+
+| Question | Where the answer is |
+|---|---|
+| Is there pending work, and how much? | `SUM(group_jobs_inq * job_threads)`, `job_memory`, `job_gpu`, `tags`/`prerequisites` over `mxq_group WHERE group_jobs_inq>0` |
+| Is node H idle right now? | `mxq_daemon.status=0` and `daemon_slots_running=0`; cross-check `mxq_job` has no rows for H in status (100,150,200) |
+| Is node H alive? | freshness of `mxq_daemon.mtime` (stale ⇒ gone) |
+| How long has H been idle? | **not stored** — infer from `NOW() - MAX(mxq_job.date_end)` for H, else from `daemon_start` |
+| What can H do (once it's off)? | last `mxq_daemon` row's capacity columns, but see §5 — better kept in a static inventory |
+
+## 3. Why we can't just copy Slurm
+
+Slurm's `slurmctld` is a central controller that already tracks every node and issues
+commands to them; power-save is just two site scripts it invokes —
+`SuspendProgram` (node idle > `SuspendTime` ⇒ power down) and `ResumeProgram`
+(queued work needs a node ⇒ power up), rate-limited by `SuspendRate`/`ResumeRate`,
+with `SuspendExcNodes` protecting infrastructure. Power-*on* is done out-of-band via
+**IPMI `chassis power on`** or **Wake-on-LAN**; power-*off* by an ordered shutdown.
+
+MXQ has no `slurmctld`. So the equivalent of "the controller that decides and acts"
+is exactly the new component we must add. The good news: because nodes *pull*, the
+resume path is trivial — we only have to get the box powered on; `mxqd` then boots,
+registers, and starts pulling with **zero scheduler involvement**.
+
+## 4. Proposed design — a central "power manager"
+
+A single new component on the management host (where `mxqdump`/`mxqadmin` already
+run), driven by a **systemd timer or cron** every 1–2 min. It never runs on the
+compute nodes. Two independent decision loops (they can be one script):
+
+```
+          ┌─────────────────── management host ───────────────────┐
+          │   mxq-powerd  (cron/systemd timer, every ~1–2 min)     │
+          │                                                        │
+  reads   │  ┌── suspend loop ──┐        ┌── resume loop ──┐        │
+ ◄────────┤  │ idle > T & safe? │        │ pending demand  │        │
+  MySQL   │  │  → drain+off     │        │ > free capacity?│        │
+  (mxq_*) │  └────────┬─────────┘        └────────┬────────┘        │
+          │           │ ssh: mxqdctl stop; poweroff│ IPMI power on / │
+          └───────────┼────────────────────────────┼─WoL────────────┘
+                      ▼                            ▼
+              compute node (in-band,        compute node (out-of-band,
+              graceful, deregisters)        node is off → BMC/WoL only)
+```
+
+### 4a. Suspend loop (power **off**)
+
+1. **Find safe idle candidates.** Per hostname (aggregating *all* live daemon rows,
+   since a host may run several daemons): every daemon `status=IDLE(0)` and
+   `daemon_slots_running=0`, `mtime` fresh (node alive), and no `mxq_job` rows for the
+   host in status (100,150,200).
+2. **Idle-long-enough gate** (`SuspendTime`, e.g. 30 min): `NOW() - MAX(date_end)`
+   for the host exceeds the threshold (fallback `daemon_start` if it never ran a job).
+   This is the one signal MXQ doesn't store directly — see §6 for an optional column.
+3. **Policy gates:** exclude-list (login/storage/infra nodes — the `SuspendExcNodes`
+   analog); keep a **minimum warm pool** of idle nodes on so short bursts don't wait
+   for a boot; rate-limit (`SuspendRate`) to avoid a sudden power drop.
+4. **Ordering: oldest generations first.** Rank candidates by a per-node
+   power-priority from the inventory (§5) so the least efficient hardware powers off
+   first.
+5. **Graceful power-off (in-band):** for each chosen node
+   `ssh HOST 'mxqdctl-hostconfig stop'` (SIGTERM ⇒ stops claiming, drains, sets
+   `status=EXITED`), **wait until its `mxq_daemon` row shows EXITED**, then
+   `ssh HOST 'systemctl poweroff'`. Waiting for EXITED closes the claim race: the
+   daemon only marks EXITED *after* draining, so once EXITED no job can be running or
+   half-claimed on that node. In-band shutdown (not IPMI power-off) lets the daemon
+   deregister cleanly — no stale RUNNING rows to reconcile.
+
+### 4b. Resume loop (power **on**)
+
+1. **Pending demand** from `mxq_group WHERE group_jobs_inq>0`: total pending cores
+   `SUM(group_jobs_inq*job_threads)`, plus memory / GPU / time / tag & prerequisite
+   constraints.
+2. **Free capacity** of nodes already on:
+   `SUM(daemon_slots - daemon_slots_running)` over live daemons (respecting the same
+   tag/GPU/memory constraints).
+3. **Deficit ⇒ power on.** Also power on when a *specific* pending job can only run on
+   a class of node that is currently all-off (e.g. GPU or high-memory / a required
+   tag), even if raw core capacity looks sufficient.
+4. **Ordering: newest / most efficient first** (inverse of suspend), and pick nodes
+   whose inventoried capabilities actually satisfy the waiting jobs.
+5. **Power on out-of-band:** IPMI `chassis power on` (preferred, reliable) or a WoL
+   magic packet. Rate-limit (`ResumeRate`) to avoid inrush current.
+6. **Boot ⇒ self-heal:** the node's init/systemd runs `mxqdctl-hostconfig.sh`, `mxqd`
+   registers a fresh `mxq_daemon` row and starts pulling. Nothing else to do.
+7. **Timeout:** if no fresh `mxq_daemon` row appears within `ResumeTimeout`
+   (e.g. 10 min), flag the node as failed, exclude it, and try the next candidate.
+
+### 4c. Anti-thrash
+
+Boot takes minutes, so hysteresis matters: generous `SuspendTime`, a warm pool,
+independent rate limits both directions, and never suspend below the warm-pool floor.
+
+## 5. The one new piece of persistent state: a node inventory
+
+The live DB describes daemons *that are currently up*. To decide which **powered-off**
+node to wake — and to reach its BMC — we need static per-node facts the DB can't give
+once the node is off:
+
+`hostname, power_priority(generation), slots, memory, gpus, tags, bmc_address, mac`
+
+Recommended: a **flat site config file** (e.g. `/etc/mxq/nodes.conf`), matching the
+existing `/etc/hostconfig` style, rather than a new DB table. It is simplest, keeps
+site/hardware facts out of the runtime DB, and MXQ is already comfortable with flat
+config + shell. A DB table (`mxq_node`) is a possible later step if the inventory
+needs to be queried by other tools.
+
+## 6. Optional, later changes inside MXQ (not needed for v1)
+
+v1 can be **100% external** — no changes to `mxqd`, the schema, or `mxqadmin`. Cleaner
+follow-ups if this graduates from experiment to production:
+
+- **A real DRAIN state.** Today "stop taking work but stay up" is only achievable by
+  SIGTERM (which also exits). A `daemon_flags` DRAINING bit that `mxqd` checks before
+  claiming (in the `mxq_load_running_groups` path) would let the manager mark a node
+  "don't take work", pause, and *cancel* cheaply if demand spikes — removing the
+  SIGTERM-then-wait dance and the residual race window entirely.
+- **An `idle_since` timestamp** on `mxq_daemon`, set when `slots_running` hits 0, so
+  idle duration is read directly instead of inferred from `MAX(mxq_job.date_end)`.
+- **`mxqadmin` verbs** `--drain HOST` / `--poweroff HOST` / `--poweron HOST`, so
+  operators share one tested code path with the automated manager.
+- **A central watchdog** that sweeps stale `mtime` rows to CRASHED (today only a
+  restarting daemon does this) — useful independent of power management, and needed so
+  a node that fails to power on/off is noticed.
+
+## 7. Suggested phasing
+
+1. **v1 — external, read-only-ish:** the `mxq-powerd` script (suspend + resume loops)
+   + `nodes.conf` inventory + IPMI/WoL + `ssh poweroff`, on a timer. Start in
+   **dry-run/log-only** mode; verify decisions against a real idle/busy cluster before
+   arming actuation. No MXQ code touched.
+2. **v2 — coordinate with the daemon:** add the DRAINING flag + `idle_since` +
+   `mxqadmin` verbs to remove the race window and simplify the manager.
+3. **v3 — policy:** smarter warm-pool sizing / predictive wake based on submission
+   patterns (cf. the queueing-penalty research on ill-timed transitions).
+
+## 8. Open questions for Paul
+
+- Power-*on* transport available in the fleet: **IPMI/BMC**, **WoL**, both, or
+  something site-specific (redfish, PDU)?
+- Is `ssh HOST poweroff` acceptable for power-*off*, or must that also go through the
+  BMC?
+- Where should node generation / power-priority and BMC addresses live — a new
+  `/etc/mxq/nodes.conf`, or is that data already in an existing inventory/CMDB we
+  should read instead?
+- Which nodes must **never** be suspended (login, storage, license servers)?
+- Target `SuspendTime`, warm-pool size, and rate limits.
+
+## References
+
+- Slurm Power Saving Guide — https://slurm.schedmd.com/power_save.html
+- OleHolmNielsen/Slurm_tools power_save (example IPMI scripts) — https://github.com/OleHolmNielsen/Slurm_tools/tree/master/power_save
+- GSI Slurm power-management notes (WoL/IPMI resume) — https://web-docs.gsi.de/~vpenso/notes/posts/hpc/cluster/slurm/power-management.html
+- Thinkbox Deadline "Machine Startup" (WoL/IPMI wake on submit) — https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/power-management.html
+- SPARS: RL simulator for HPC power management (transition-timing penalties) — https://arxiv.org/pdf/2512.13268