Add clusterd #55
Merged
Add clusterd #55
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
native seems to be broken and delivers newlines into the logile
unter die knute des bee
we now create release tags .e.g. bee init https://github.molgen.mpg.de/donald/clusterd/archive/v1.104.tar.gz would work
for the rolling upgrade we need to request another rpc version, because the current receiver trusts the sender to send only plain files
symlinks need to be send to a new rpc "filedata.2" because the old rpc can't handle it. the updated receiver is installed under the old name "filedata" too in case older daemons send push
* symlinks: update receiver to handle symlinks enable push to send symlink add lchown() , lmitime()
This command might be usefull, e.g. if sshd is not responding. Examples: clusterd --exec @theinternet restart sshd.service clusterd --exec @theinternet dmesg clusterd --exec @theinternet 'grep sshd /var/log/messages|tail -100' The remote command is executed by bash, so the pipe in the last example is on the remote node stdout and stderr are delivered seperatly and might be redirected to different channels on the local side. If the remote command exits with a non-zero exit status, the local command fails with exit status 1. The slave part already existed, because we once had a remote execution command. We removed it, because it was considered to be to dangerous. The former remote execution command allowed parallel execution on all nodes, which enabled an admin to kill all systems by mistake in an instance. Now we only enable remote execution on a single node.
* add-remote-exec: Implement --exec @node cmd [args...]
Use the namespace My:: for packages declared inside the main script.
We imported some lines which do no longer have a function now. Remove them.
Non-functional change to bring source into sync with another working branch.
Use refenerces to subs as callback arguments. If the caller wants to pass additional arguments, he can use closures.
This hash is not used but it prevents these sockets to be able to be closed by running out of scope.
The command "clusterd push file..." has a new syntax with a command verb. (as opposed to "clusterd --push file") The push command is distributed via the area routers to the cluster daemons on all nodes. It is intended that the cluster daemons call back over tcp to the originator to pull the file if needed.
When a daemon receices a push command, it checks whether it already has the offered file or not. If not, it calls back to the daemon where the push originated and pulls the file over tcp.
Implement the server side to pull a file over tcp.
While 5 seconds seems to be enough for normal behaviour, we might need more time if the daemon is slowed down .e.g. by strace.
With 234 hosts pulling a file we failed to service a single on with the old setting listen=1.
If we have dead nfs mounts, lsof might hangover. Limit a timeout and return an error message if lsof fails.
Refactor the code, so that the mltime specific settings are done from inside the machine type switch. This makes it easier to add another machine type which can set the time of a symlink.
amd automounted is gone on all systems, so don't try to call amq.
Currently we may move received files in place, even whe the write or close were unsuccessfull (eg. disk full). This is a serious bug and we may end up with an empty passwd. Add checks for all write() and close() calls. Also log rename errors.
The clusterd repository has been prepared by
git filter-branch \
--msg-filter 'sed -r "1s/^(clusterd: )?/clusterd: /"' \
--index-filter 'git ls-files -s | sed "s-\t\"*-&clusterd/-" | GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' \
-- master
to move all files into a subdirectory clusterd/ and make sure,
the commit messages are prefixed wih "clusterd:".
Not merge its master branch.
* clusterd/master:
clusterd: Remove redundant blank lines.
clusterd: Add checks for write errors
clusterd: Remove call to amq
clusterd: Add ppc64le architecture
clusterd: Refactor lmtime setting
clusterd: Increase lsof timeout
clusterd: Do not resolve hostname when doing lsof
clusterd: Bump up version
clusterd: Add timeout for lsof
clusterd: Bump up version
clusterd: Increase listen queue size for clp port
clusterd: Increase TCP timeout from 5 to 30
clusterd: Implement PULL in daemon
clusterd: Implement PUSH in daemon
clusterd: Add command "clusterd push files..."
clusterd: Add send_tcp_cp() to send a cluster protocol message over tcp
clusterd: Add functions for nonblocking tcp to My::Select::INET
clusterd: Remove clp_sockets hash
clusterd: Remove/Move imports
clusterd: Remove My::Callback
clusterd: Remove/fix some comments
clusterd: Remove unused sub heartbeat.
clusterd: Remove declarations no longer needeid after import
clusterd: Rename Donald::Select to My::Select
clusterd: Import Donald::Select into script
clusterd: Remove mlx ping
clusterd: Dereference CLP tcp socket after hangup
clusterd: Clean up indentation mix
clusterd: implement --make-automaps
clusterd: bump up version
clusterd: remove some debug relicts
clusterd: Implement --exec @node cmd [args...]
clusterd: fix shebang to use /usr/bin/perl
clusterd: bump up revision
clusterd: update receiver to handle symlinks
clusterd: enable push to send symlink
clusterd: add lchown() , lmitime()
clusterd: enable push to send empty files
clusterd: increase push file limit from 5*8*1024 to 8*8*1024
clusterd: make push_file errors like "to big" fatal
clusterd: change wait to happen before broadcast if we need to throttle
clusterd: style: remove some extra whitespace
clusterd: update comment pointing to github home
clusterd: bump up revision
clusterd: ping mlx once a day
clusterd: remove mkbee.sh
clusterd: nach /usr/sbin installieren aber in /sbin aufrufen is nix gut
clusterd: unter die knute des bee
clusterd: use unix socket for syslog
clusterd: we live on github now
clusterd: add clusterd.service
clusterd: add --flush-gidcache
clusterd: add primitive Makefile
clusterd: initial
Add clusterd files to installation script and remove its obsolete stand-alone Makefile.
|
As discussed, I am not convinced, that putting everything in one repository is the way to go, but as you are the main developer, you decide. |
Sign in
to join this conversation on GitHub.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I want to move clusterd (script and systemd unit) from its own repository mariux64/clusterd with releases and bee based installation into this repository for an easier rapid development.
The history has been rewritten (subdir and commit message prefix). See 773d10f.
The merged version includes an important fix to avoid empty passwd files on full system disks (c93face).