SGE / Grid Engine

Sun Grid Engine (SGE) and its descendants — Open Grid Scheduler, Son of Grid Engine, Univa/Altair Grid Engine — remain common on institutional clusters. Jobs are submitted with qsub, monitored with qstat, and cancelled with qdel. Directives live in #$ comment lines.

A basic job script

#!/bin/bash
#$ -N analysis                # job name
#$ -cwd                       # run from the submission directory
#$ -j y                       # merge stderr into stdout
#$ -o logs/                   # output log location
#$ -l h_rt=02:00:00           # hard walltime limit
#$ -l h_vmem=4G               # memory PER SLOT (core), not total
#$ -pe smp 4                  # parallel environment: 4 slots on one node

module load python/3.12
python analyze.py input.csv

qsub job.sh
# Your job 123456 ("analysis") has been submitted

Two SGE quirks to remember: without -cwd jobs run from your home directory, and memory requests like h_vmem are usually per slot, so the script above gets 4 × 4G = 16G total.

Monitoring and controlling jobs

qstat                    # your jobs (some clusters: all jobs; use -u $USER)
qstat -j 123456          # full details, including why a job is pending
qdel 123456              # cancel one job
qdel -u $USER            # cancel all your jobs
qhost                    # node list with load and memory
qstat -g c               # queue summary (slots used/available)

Job states in qstat: qw queued/waiting, r running, Eqw error while queued — inspect with qstat -j ID | grep error, fix the cause, then qmod -cj ID to clear or qdel and resubmit.

Interactive sessions

qrsh -l h_rt=01:00:00 -l h_vmem=4G -pe smp 4

qrsh requests an interactive shell on a compute node with the same -l/-pe resource syntax as qsub. Some clusters offer qlogin as an alternative.

Job arrays

#!/bin/bash
#$ -N array-demo
#$ -cwd
#$ -t 1-100                   # task indices 1..100
#$ -tc 10                     # at most 10 tasks running at once
#$ -l h_rt=00:30:00
#$ -l h_vmem=4G

INPUT=$(sed -n "${SGE_TASK_ID}p" inputs.txt)
python process.py "$INPUT"

Each task gets $SGE_TASK_ID. Output files are suffixed with the task ID (array-demo.o123456.1, .2, …).

Parallel environments

Multi-core and multi-node jobs go through a parallel environment (PE) requested with -pe NAME SLOTS. Names vary by cluster — smp, omp, sharedmem for single-node multithreading; mpi, orte for MPI across nodes. List what exists:

qconf -spl               # list parallel environments
qconf -sp smp            # show settings of one PE
qconf -sql               # list queues

Inside the job, $NSLOTS holds the slot count — pass it to your tools (e.g. --threads $NSLOTS).

Dependencies and chaining

qsub -N prep prep.sh
qsub -N train -hold_jid prep train.sh
qsub -hold_jid train summarize.sh

-hold_jid accepts job IDs or names; the job stays in hqw (hold) state until the dependencies finish. For array-to-array chaining, -hold_jid_ad waits per-task.

Accounting: what did my job actually use?

qacct -j 123456

Look at maxvmem vs your h_vmem request and wallclock vs h_rt to calibrate future requests. failed / exit_status distinguish scheduler kills (e.g. over memory) from your program's own errors.

Useful environment variables

• $JOB_ID — the job's ID.
• $SGE_TASK_ID — index within an array job (undefined for non-array jobs).
• $NSLOTS — slots allocated via -pe.
• $SGE_O_WORKDIR — directory the job was submitted from.
• $TMPDIR — per-job scratch directory on the compute node, cleaned up automatically.