Add black and whitelists #83
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Avoid unneeded warning when scanning JOB_TMPDIR_MNTDIR for possible leftover job mounts. The parent directory of JOB_TMPDIR_MNTDIR is created when the first job with the --tmpdir feature is started. So it won't exist in a new server. This cleanup path is only used in the exceptional case that the usual cleanup path (via job_has_finished) didn't succeed, e.g. when mxqd was killed. The cleanup is not essential for the currently running server.
Add function attribute ((unusged)) to avoid a compiler warning when the static inline function defined in the header file is not used by the compilation unit.
Add header file for the quasi-standard [1] xmalloc call. The current approach implemented in mx_util is to wait and retry on ENOMEM malloc failure. However, this overhead doesn't seem to be justified, because it is difficult to imagine a case, where a malloc would fail with ENOMEM at one time and a retry would succeed. [1] https://www.gnu.org/software/libc/manual/html_node/Malloc-Examples.html
Add a utility module which can store a set of keywords. The set can be
created and updated from a string and can be serialized into a string.
The canonical string representation produced by this utilits is the space
separated list of the sorted keywords in lexical order.
Input strings used to create or update a keyword set contain keywords
separated by whitespace. The keywords may be prefxied with "+" and "-".
For updates, first all unprefixed keywords are processes to create the
initial set, than updates are applied from the prefixed keywords.
An empty string used for updates is considered "no change", not
"set to empty set".
Usage example:
struct keywordset *kws = keywordset_new("xx yy")
keywordset_update(kws, "-yy +zz") // remove yy, add zz
keywordset_update(kws, "aa bb") // set to aa, bb
if (keywordset_ismember(kws,"bb")) ... // true
char *s = keywordset_get(kws): // s now "aa bb"
free(s); // caller must free()
keywordset_purge(kws) // set nowto empty.
keywordset_free(kws);
Add list of blacklisted and whitelisted servers to group. The string is supposed to be canonical (lexical sorted, space separated) as produced by keywordset_get(), so that groups with equal sets can be found by sql string compare.
Add list of blacklisted and whitelisted servers to group. The string is supposed to be canonical (lexical sorted, space separated) as produced by keywordset_get(), so that groups with equal sets can be found by sql string compare.
Add a new options so that specific mxqd servers can be excluded from starting
jobs for this group.
mxqsub --whitelist "acedia avaritia" sleep 10 // start only on these
mxqsub --blacklist dontpanic sleep 10 // start on any node but this one
Blacklist has priority, so
mxqsub --white "kronos, uselessbox" --black kronos
would not start on kronos. The lists of blacklisted and whitelisted
servers can be modified with mxqset.
Add a new command which can be used to modify an existing group.
Examples:
mxqset group 123 --closed
mxqset group 123 --open
mxqset group 123 --blacklist "dontpanic" # replace
mxqset group 123 --whitelist "+uselessbox +gula" # add
mxqset group 123 --blacklist -dontpanic # remove
mxqset group 123 --whitelist "" # clear
The flags open and closed can be set from mxqadmin as well
(`mxqadmin --close=123`) , but the synopsis, in which the options take
the groupid as a value is difficult to expand to new options, which take a
value.
Check, whether this server is qualified to start jobs from a group. Lazy-evaluate qualification criteria defined in the group and cache the result. The qualification criteria need to be reevaluated when the active groups were reloaded, because they may be changed by the user in existing groups. For now, the only qualification criteria are the groups blacklist and whitelist. If a group has a whitelist, the short or long hostname of the mxqd server needs to be on that list, otherwise the server is not qualified for the group. If the servers name is on the blacklist of the group, the server is not qualified for the group. Don't start jobs we are not qualified for. This can later be expanded to additional criteria (e.g. hostconfig or processor flags).
This was referenced Apr 15, 2020
Closed
Contributor
|
I can't wait to see this in action. |
Contributor
Author
|
Go ahead, its running on all servers now. was executed ... on ... acedia :-) |
Contributor
|
ok - I just blacklisted 'dontpanic' and did not need to panic anymore. Hurray! |
Contributor
|
... wait a sec ... - I need more testing - just saw a job crashing on dontpanic - http://afk.molgen.mpg.de/mxq/mxq/job/20716429 Did I do something wrong? |
Contributor
|
probably the dashes |
Contributor
Author
|
Yes, the should all do. All the same. Witha list (spaces) you'd need some kind of quotes. And is the same as |
Contributor
Author
|
Or even |
Contributor
|
ok - klappt |
Sign in
to join this conversation on GitHub.
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add --blacklist and --whitelist to mxqsub and add mxset command. Replaces #81 and #82