Slurm

Slurm is the most widely used scheduler on modern academic and national HPC systems. Jobs are submitted with sbatch, monitored with squeue, and cancelled with scancel. Directives live in #SBATCH comment lines.

A basic job script

#!/bin/bash
#SBATCH --job-name=analysis
#SBATCH --partition=general        # a.k.a. queue; check your cluster's names
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4          # cores for multithreaded code
#SBATCH --mem=16G                  # total memory (or --mem-per-cpu=4G)
#SBATCH --time=02:00:00            # walltime limit HH:MM:SS
#SBATCH --output=logs/%x_%j.out    # %x = job name, %j = job ID
#SBATCH --error=logs/%x_%j.err

module load python/3.12
python analyze.py input.csv

sbatch job.sh
# Submitted batch job 123456

Any #SBATCH option can also be passed on the command line, which overrides the script:

sbatch --time=08:00:00 --mem=64G job.sh

Monitoring and controlling jobs

squeue --me                 # your queued and running jobs
squeue -j 123456 --start    # estimated start time for a pending job
scontrol show job 123456    # full details of a job
scancel 123456              # cancel one job
scancel --me                # cancel all your jobs
sinfo                       # partitions and node states

The ST column in squeue shows job state: PD pending, R running, CG completing. The NODELIST(REASON) column explains why a job is pending — Priority (others are ahead of you), Resources (waiting for nodes), QOSMaxWallDurationPerJobLimit (asked for more than the partition allows), etc.

Interactive sessions

srun --pty --cpus-per-task=4 --mem=16G --time=01:00:00 bash

This queues like any job, then drops you into a shell on a compute node. On clusters with newer Slurm, salloc followed by srun inside the allocation does the same.

Job arrays

#!/bin/bash
#SBATCH --job-name=array-demo
#SBATCH --array=1-100%10        # indices 1..100, at most 10 running at once
#SBATCH --time=00:30:00
#SBATCH --mem=4G

INPUT=$(sed -n "${SLURM_ARRAY_TASK_ID}p" inputs.txt)
python process.py "$INPUT"

Each task gets $SLURM_ARRAY_TASK_ID; a common pattern (above) is reading the Nth line of a file listing inputs. In --output patterns, %A is the array's job ID and %a the task index (e.g. --output=logs/%A_%a.out).

GPUs

#SBATCH --partition=gpu
#SBATCH --gres=gpu:1            # one GPU of any type
#SBATCH --gres=gpu:a100:2       # two A100s (type names vary by cluster)

Inside the job, $CUDA_VISIBLE_DEVICES is set automatically.

Dependencies and chaining

Run jobs in sequence (e.g. preprocess → train → summarize):

jid1=$(sbatch --parsable prep.sh)
jid2=$(sbatch --parsable --dependency=afterok:$jid1 train.sh)
sbatch --dependency=afterok:$jid2 summarize.sh

afterok runs only if the dependency succeeded; afterany runs regardless; afternotok runs only on failure (useful for cleanup or notification jobs).

Accounting: what did my job actually use?

sacct -j 123456 --format=JobID,JobName,Elapsed,TotalCPU,MaxRSS,State
seff 123456      # human-readable efficiency summary (if installed)

MaxRSS vs requested memory and TotalCPU vs Elapsed × cores tell you whether to request less (queue faster) or more (avoid OUT_OF_MEMORY / TIMEOUT states) next time.

Useful environment variables

• $SLURM_JOB_ID — the job's ID.
• $SLURM_ARRAY_TASK_ID — index within an array job.
• $SLURM_CPUS_PER_TASK — cores allocated; pass to tools, e.g. --threads $SLURM_CPUS_PER_TASK.
• $SLURM_SUBMIT_DIR — directory sbatch was run from (jobs start there by default).
• $SLURM_JOB_NODELIST — nodes assigned to the job.