Add reaper process to handle job execution and reaping of childs

[mariux]

Add reaper process to handle job execution and reaping of childs.

• spawn reaper to handle job execution
• store finished child status in file instead of database to be able to collect data on fails
• when recovering from fail read status files and store data n database (late finish jobs)
• when recovering and jobs are still running from previous mxqd runs "reattach" those jobs to mxqd
  • mark memory, thread/cpuset to be occupied by those jobs.
• catch signals from server to kill user jobs

new mxqd -> reaper interactions may include:

• send signal SIGUSR1 kill children slowly with SIGTERM!?
• send signal SIGUSR2 kill children now SIGKILL!?

other issues:

• log returned childs.

review: bugs to be fixed:

• date_start and date_endmissing in reaper stat file
• handle LOADED jobs
• save status from MAIN pid to database (atm it seems to be last PID ripped?)
• reaper: when MAINPID exits kill all left over children
  • scan proctree and kill all with PPID == PID(reaper)

[donald]

If we keep both alternatives to detect finished user jobs, the reaper had to reflect the exit status of the user process including signals. This is possible but ugly.
Perhaps we should go for files only.

[mariux]

it's not that ugly reset all signal handlers and kill(getpid(), SIGNAL) or exit(STATUS) ;)

but true... so main process should only reap the pids and should only perform action if no file is available while in the normal case fspool_scan should be the only function to update database entries.... sounds good ;)

[donald]

In that case... If a reaper terminates without leaving a finished job file, it would always be an error and a rather serious and unexpected one like spool file system full (which the reaper could handle by waiting) or, of course, a coding error or hardware error. In that case it might be an idea for mxqd to terminate or at least stop to start new jobs, to prevent more jobs going into the same sink.

[mariux]

so reaper should try to write the file forever - if recoverable? ;)

[donald]

because "terminate daemon but leave jobs running" is now a valid option, what meaning do we want to have for signals to mxqd ?

a) exit (leave children running)
b) wait for children to complete and exit
c) kill children, wait for children to complete and exit

SIGINT for a , SIGTERM for c, we don't need b ?

[mariux]

• a) SIGQUIT because it quits the daemon?
• b) SIGTERM because it (slowly) terminates all actions?
• c) SIGINT because it interrupts all actions and exits as fast as possible?
• maybe SIGTERM does not have to wait forever in future releases and also kills processes that are not scheduled to finish within the (configurable) 24h limit? (e.g. where runtime_left >= 24h and/or where current_runtime >= 24h)
• but for all signals it should be possible to first TERM and later INT or kill all leftovers if QUIT was used?

what do you think? kreuze an ;))))) hehehe

[mariux]

I just remembered and checked again and found that SIGTERM and SIGINT are already implemented that way:

• TERM stops starting new, waits for running and exits.
• INT stop stating new, kills all running, waits for killed and exits.

So the only new signal to handle is SIGQUIT or any other signal that should just quit daemon.

[donald]

Hmmm. I don't like the idea of a "signal escalation order". This isn't defined anywhere and is not really useful.

We have a more or less defined meaning for "TERM" in signal(7), which is the termination signal: The process should cleanup and exit. IMO in our case it should leave the user jobs running, because with the reaper thats a perfect legal state and there is no reason to wait for the processes to finish. A system shutdown would send signals to the user processes anyway.

"QUIT" and "INT" are defined as keyboard signals. If I have a daemon running in the foreground on a terminal, I'd expect ^C to terminate it (the same as TERM in the above paragraph). A detached daemon has no terminal so the signals have no predefined meaning in that context. We could reuse the signals for other purposes, if we run out of SIGUSR1,SIGUSR2.

What do we want to ask the daemon besides "Terminate" ?

We might want a way to ask the daemon to stop accepting new jobs. But we could get to the same state by killing the daemon and restarting it with -m 1 or whatever. So this would just be a convenience.

We might want a way to ask the daemon to kill all running jobs. But we could do the same manually. So again, this would just be a convenience.

We might want a way to ask the daemon to exit, when the jobs are finished and their cleanup work is done, which only makes sense, when we don't accept jobs while waiting. On the other hand, we could leave the daemon running, because it won't do anything when not accepting new jobs or finishing old ones. So a third time just convenience.

Hmmm....

[mariux]

it does not matter how a signal is sent because there is no difference in sending SIGQUIT/SIGINT via keyboard or via kill. When using keyboard shortcuts for signals you want the foreground process to end now: and both signals end the main mxqd - only sending SIGTERM will not exit immediately but wait for a clean shutdown.

And for SIGTERM signal(7) defines no "cleanup and exit." action. Yes, it's a termination signal and that is what currently happens directly after waiting for all jobs to finish in a clean way.

Stopping a daemon and restarting it with different (less) resources (-m 1) is currently undefined behavior.
And removing current SIGTERM action and replacing it with something like SIGQUIT+mxqd -m 1 should not be an option.

But after all: The only new action is to quit the daemon immediately and leave all reaper running. There is no need to change or remove existing behavior and signal actions.

[donald]

Completed (2015) by 9e9c3a5 ("Merge remote-tracking branch 'mariux/issues/issue30')