PBS / OpenPBS / Torque

The PBS family — OpenPBS, PBS Professional, and the older Torque — uses qsub/qstat/qdel like SGE, but with its own directive syntax in #PBS comment lines and resources requested via -l select=... (PBS Pro/OpenPBS) or -l nodes=... (Torque).

A basic job script

#!/bin/bash
#PBS -N analysis                       # job name
#PBS -q workq                          # queue; check your cluster's names
#PBS -l select=1:ncpus=4:mem=16gb      # 1 chunk: 4 cores, 16 GB (PBS Pro/OpenPBS)
#PBS -l walltime=02:00:00
#PBS -j oe                             # merge stderr into stdout
#PBS -o logs/

cd "$PBS_O_WORKDIR"                    # jobs start in $HOME by default

module load python/3.12
python analyze.py input.csv

qsub job.sh
# 123456.pbsserver

On Torque-era clusters the resource line is written as:

#PBS -l nodes=1:ppn=4
#PBS -l mem=16gb

Note the cd "$PBS_O_WORKDIR" — unlike Slurm, PBS starts jobs in your home directory, so almost every script begins with this line.

Monitoring and controlling jobs

qstat -u $USER           # your jobs
qstat -f 123456          # full details of a job
qstat -x 123456          # include finished jobs (PBS Pro/OpenPBS)
qstat -Q                 # queue limits and states
qdel 123456              # cancel a job
pbsnodes -a              # node states and resources

Job states in qstat: Q queued, R running, H held, E exiting, F finished (only visible with -x).

Interactive sessions

qsub -I -l select=1:ncpus=4:mem=16gb -l walltime=01:00:00

-I queues an interactive job and connects you to a shell on the compute node when it starts. Add -X for X11 forwarding if you need graphics.

Job arrays

#!/bin/bash
#PBS -N array-demo
#PBS -J 1-100                          # task indices (PBS Pro/OpenPBS)
#PBS -l select=1:ncpus=1:mem=4gb
#PBS -l walltime=00:30:00

cd "$PBS_O_WORKDIR"
INPUT=$(sed -n "${PBS_ARRAY_INDEX}p" inputs.txt)
python process.py "$INPUT"

Each subjob gets $PBS_ARRAY_INDEX. On Torque the flag is -t 1-100 and the variable is $PBS_ARRAYID. Query the whole array with qstat -t 123456[], or one subjob with qstat 123456[7].

Multi-node and MPI jobs

select requests chunks of resources; multiply chunks for multi-node jobs:

#PBS -l select=4:ncpus=32:mpiprocs=32   # 4 nodes × 32 cores, 128 MPI ranks
#PBS -l place=scatter                   # spread chunks across distinct nodes

The assigned nodes are listed in the file $PBS_NODEFILE, which MPI launchers read automatically on most clusters (mpirun -np 128 ./app).

GPUs

#PBS -q gpuq
#PBS -l select=1:ncpus=8:ngpus=1:mem=32gb

Queue and resource names (ngpus, gpu_type, etc.) vary by site — check your cluster's docs.

Dependencies and chaining

jid1=$(qsub prep.sh)
jid2=$(qsub -W depend=afterok:$jid1 train.sh)
qsub -W depend=afterok:$jid2 summarize.sh

qsub prints the job ID, so it can be captured directly. As in Slurm, afterok runs on success, afterany regardless, afternotok on failure.

Accounting: what did my job actually use?

qstat -x -f 123456 | grep -E 'resources_used|Exit_status'

Compare resources_used.mem and resources_used.walltime against your requests to calibrate the next submission. Exit_status of 0 means success; large values (≥ 128 + N) usually mean the job was killed by signal N (e.g. over its limits).

Useful environment variables

• $PBS_JOBID — the job's ID.
• $PBS_ARRAY_INDEX — index within an array job (Torque: $PBS_ARRAYID).
• $PBS_O_WORKDIR — directory the job was submitted from; cd there first.
• $PBS_NODEFILE — file listing assigned nodes, one line per MPI rank.
• $NCPUS — cores allocated to the job (PBS Pro/OpenPBS).
• $TMPDIR — per-job scratch directory on the compute node.