Grid Engine Tools (gtools)

gtools is a collection of bash / awk scripts that make your life easy when running jobs with the Sun Grid Engine (SGE).

The three most important features are:

• transparent ssh to a submit host;
• detection of failed jobs;
• running Matlab compiled code.

The first feature is important because every interaction with the SGE happens via qsub, qstat, and other q* commands that must be executed on a submit host. Instead, gtools automatically ssh to and run the respective command on the submit host, so that you can interact with the cluster directly from your workstation.

The second feature is probably the most important. When running a large number of array jobs where each job contains tens or even hundreds of tasks, it is quite common that a few of those tasks may fail. The problem is then to identify and resubmit only these few failed tasks rather than the whole array job. gtools do that by detecting a non-zero exit code of a failed command. When that happens, a wrapper script from gtools returns a special exit code to the SGE daemon, which then puts that one given task into an error state.

You get an overview of running / pending / failed jobs by running

gt st

If you want to see details like which tasks are running or failed, run

gt st -j <job_id>

And if you want to simply resubmit the failed tasks, just run

gt re

Finally, the third main feature makes it easy to run Matlab compiled code. gtools set all the necessary environment variables, just like the run_*.sh files that are generated by the Matlab compiler, so that you can run the compiled code as an ordinary binary.

There is a number of other features like grouping of tasks, custom user options, and setting the resource limits conveniently.

Just try gt to see a list of commands and gt help <command> to learn more about any given command.

Installation (GNU/Linux)

• Clone the repository:

git clone https://github.com/mlapin/gtools.git

• Make sure the gt script is in your $PATH.

Hint: either add the gtools folder to your $PATH or create a symlink to gt in a folder that is already in the $PATH, like so:

ln -s /path/to/gtools/gt

• Run gt check to see if everything has been setup correctly.

Minimal Working Example

• Prepare a text file with commands

for i in `seq 5`; do echo "echo 'line $i:'" '"running job $JOB_ID, task $SGE_TASK_ID"' >> cmd.txt; done

• Submit a job

gt file cmd.txt

• Check job status

gt st

Hint: use gt check to verify the logs directory.

Troubleshooting

• run gt check -v
• check gt-setup.sh
• create a user config (gt config) and modify it according to your needs.