Skip to content

[RFC] Power management #173

Jump to bottom
Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

[RFC] Power management #173

wants to merge 4 commits into from
+2,343 −3

Conversation

@pmenzel
Copy link
Contributor

@pmenzel pmenzel commented Jul 19, 2026 •
edited
Loading

Everything still work in progress.

Separate daemon controlling nodes to power them off and start them.

  • We currently don't start mxqd autmatically after boot. This would need to be changed.
  • We would not be able to use the provided .timer as-is because it should be enabled on a single Mariux64 node only. Maybe on afk , maybe somewhere else with a ssh key which allows the required commands do stop mxqd or shutdown only.

pmenzel and others added 3 commits May 28, 2026 08:13
@pmenzel
mxqsub: add --callback/-c option to run an executable when a job fini…
e1616a6 
…shes

mxqsub accepts --callback=EXECUTABLE (absolute path). The daemon runs
the executable after every terminal outcome (finished, failed, killed,
unknown), as the submitting user in the job workdir, with MXQ_JOB_ID,
MXQ_GROUP_ID, MXQ_JOB_STATUS, and MXQ_JOB_WORKDIR in the environment.
The callback is double-forked so it does not block the daemon, and runs
under RLIMIT_CPU=60s, RLIMIT_AS=256MiB, RLIMIT_CORE=0.

Adds job_callback column to mxq_job; migration in migrate_019.
Also fixes mxq_set_job_status_unknown not updating job->job_status
in-struct, inconsistent with the other status-update functions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@pmenzel
mxqsub.1: document --callback option
7565dba 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@pmenzel
mxq-powerd: add external node power manager (suspend idle / resume on…
5958cae 
… demand)

Add mxq-powerd, a self-contained power manager for the MXQ cluster that
powers idle compute nodes down to save energy and powers them back up when
queued work needs them, oldest/least-efficient hardware off first. It runs
on the management host on a fixed interval (systemd timer or cron).

The manager is entirely external to MXQ: it only READS the MySQL database
(through the stock mysql client) and actuates nodes with stock CLIs
(ipmitool, wakeonlan, ssh). It issues no DB writes and requires no changes
to mxqd, the schema, or mxqadmin.

Implements docs/power-management-implementation.md:
  - static inventory /etc/mxq/nodes.conf (capability + transport + priority)
  - global policy /etc/mxq/powerd.conf (dry_run defaults to true)
  - IPMI credentials in /etc/mxq/powerd-secrets (0600; password via
    ipmitool -E, never on the command line)
  - per-node state machine in /var/lib/mxq/powerd-state.json, advanced each
    tick from the DB snapshot (pending demand, live capacity/liveness,
    inferred idle duration)
  - resume: wake lowest-gen nodes on core deficit / capability gap / warm
    pool shortfall, rate-limited
  - suspend: drain highest-gen idle nodes (ssh mxqdctl-hostconfig stop), wait
    for EXITED to close the claim race, then poweroff, with BMC soft/hard-off
    escalation; honours the surplus test, warm pool floor and rate limits
  - bounded transitions: stuck nodes land in FAILED, are excluded, and are
    left for an operator (mxq-powerd --clear HOST)

Ships example configs, systemd service + timer, an mxq-powerd.8 man page, a
README, and an opt-in `make install-powerd` target (not wired into the
default install).

DB access reuses MXQ's read-only defaults file (/etc/mxq/mysql_ro.cnf, group
mxqclient) via my_print_defaults rather than adding a Python MySQL driver
dependency; selectable in powerd.conf. Installs the program to
/usr/libexec/mxq to match the existing helper convention.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@pmenzel pmenzel force-pushed the power-management branch from 5958cae to 6ab8416 Compare July 19, 2026 13:37
donald
donald requested changes Jul 20, 2026
View reviewed changes
powerd/mxq-powerd Outdated
return False
if node.gpus < group.gpu:
return False
if group.tags and not group.tags.issubset(node.tags):
Copy link
Contributor

@donald donald Jul 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not correct. I think, group tags are evaluated against host prerequistes expressions and visa versa, not tags against tags. Also, is blacklist/whitelist missing?

Same in the following _alive_can_serve()

Sorry, I have limited access currently (no time, on mobile) can't really look into this now.

Copy link
Contributor

@donald donald Jul 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The one parameter we don't know have to decide whether a node can accept a job is the free tmpdir size. It's PITA that we allow that resource to be consumed by users outside of mxqds control via /scratch/local2 on the same filesystem.

@donald
Copy link
Contributor

donald commented Jul 20, 2026

We currently don't start mxqd autmatically after boot. This would need to be changed.

@donald
Copy link
Contributor

donald commented Jul 20, 2026

We would not be able to use the provided .timer as-is because it should be enabled on a single Mariux64 node only. Maybe on afk , maybe somewhere else with a ssh key which allows the required commands do stop mxqd or shutdown only.

@pmenzel
mxq-powerd: match jobs to hosts the way MXQ does (PR #173 review)
b3d0089 
Reviewer feedback (@donald) on the RFC: the job/host match was wrong.  It
tested group tags as a subset of node tags, but MXQ (server_is_qualified()
in mxqd_control.c) does something quite different, and the host
white/blacklist was missing entirely.

Reproduce the real model for powered-off nodes:

  * group_whitelist / group_blacklist gate on the hostname;
  * the group's `prerequisites` EXPRESSION is evaluated against the host's
    tags, and the host's own `prerequisites` against the group's tags;
  * a GPU group needs a GPU host.

- Add a small evaluator mirroring parser.y (tags as identifiers, `! & |`,
  parens, precedence ! > & > |; empty => true, syntax error => false).
- Pull tags, prerequisites, group_whitelist and group_blacklist per pending
  group; tokenise tag/white/blacklist strings on whitespace like
  keywordset_new().
- Give nodes.conf a `prerequisites` column and treat each node's tags as
  including the implicit `true`, hostname and short hostname that mxqd adds.
- Fold qualification + a capacity check into _host_satisfies()/_alive_can_serve().

Also document the operational points raised in review: enable the timer on a
single management node, mxqd must start at boot for resume to work, use an
ssh key restricted to the drain/poweroff commands, and the free-tmpdir blind
spot.  README, man page and design doc updated to match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@pmenzel pmenzel force-pushed the power-management branch from 6ab8416 to b3d0089 Compare July 20, 2026 08:02
pmenzel
pmenzel commented Jul 20, 2026
View reviewed changes
docs/power-management-implementation.md
2. **Power-*off* is graceful and in-band** (`ssh HOST poweroff` after the daemon has
drained), with a **BMC soft-off** (`ipmitool … chassis power soft`) fallback and a
**BMC hard-off** last resort only if soft-off also times out.
3. **New static inventory** `/etc/mxq/nodes.conf` (no existing CMDB integration in v1).
Copy link
Contributor Author

@pmenzel pmenzel Jul 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No idea, if this is needed, and if BMC address (if actually needed) and CPU/GPU generation could be derived without a file.

Copy link
Contributor

@donald donald Jul 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, if poweroff fails, we should investigate and fix the problem and not create complex code to work around it.

docs/power-management-implementation.md
systemd: mxq-powerd.service + .timer (or a root cron entry) — runs on the mgmt host
```

The manager runs **on the management host** (where `mxqdump`/`mxqadmin` already run),
Copy link
Contributor Author

@pmenzel pmenzel Jul 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incorrect assumption. mxqd and database run on the management host. The rest can run anywhere.

Copy link
Contributor

@donald donald Jul 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mxqd runs on the cluster nodes. I wouldn't call the node where mysql happens to run the management host. Just remove the paragraph?

docs/power-management-implementation.md
One invocation = one evaluation tick; it is safe to run back-to-back and safe to miss
a tick.

## 2. Inventory file — `/etc/mxq/nodes.conf`
Copy link
Contributor Author

@pmenzel pmenzel Jul 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above. Hopefully we won’t need it.

docs/power-management-implementation.md
(login/storage/infra).
- `bmc_or_mac` — BMC IP/hostname for `ipmi`, MAC for `wol`.

**Matching model.** Deciding whether a powered-off node could accept a pending group
Copy link
Contributor Author

@pmenzel pmenzel Jul 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this overengineering? Besides for GPU to my knowledge our scientists only ever want the newest systems.

Copy link
Contributor

@donald donald Jul 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or a mxqi node or a mxqi node
with a gpu or one node with 500GB memory or a node which is not "strangenodename" because my jobs happen to crash on that node (because they were compiled for another ISA extension).

I'm not sure we need the slot metrics at all. Why not like this: "If we have a pending job which no running node could start but a standby node could than start that standby node" Maybe "start as many capable standby nodes as needed by the pending jobs".

One problem though: The started standbynode might not pick up the job because of not enough free disk space. Perhaps we should address that first.

docs/power-management-implementation.md
Actuation per node:
```bash
# WoL
wakeonlan e4:3d:1a:00:11:22 # or: ether-wake -i <iface> <mac>
Copy link
Contributor Author

@pmenzel pmenzel Jul 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s wake on MarIuX.

@donald
Copy link
Contributor

donald commented Jul 21, 2026 •
edited
Loading

  • We neeed to address the problem with the free diskspace otherwise it is not deterministic what job can run on what node. In fact its not even deterministic for mxqd this is why it has the ugly feature to "unload" a job: Once it decided to take the job because all constraints are fullfilled, it allocates it in the database and tries to start it. But that might fail if another user job allocates to much space from /scratch/local2. In that case, the job is returned to the queue for other nodes to try. Ugly, ugly.

@donald
Copy link
Contributor

donald commented Jul 21, 2026

I often wondered if an external scheduler would be better. IMO that had many advantages. With that it would also be easier to manage power state on nodes. We wouldn't have to duplicate scheduling code from mxqd to powerd and from C to Python. This duplication would make future scheduling progress much more difficult.

Sign in to join this conversation on GitHub.

Reviewers

@donald donald donald requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pmenzel @donald