Skip to content
Contact Support

Batch Computing Guide

Batch jobs can be submitted via several methods. The most basic is a simple Slurm job. Slurm can also run jobs arrays. We also provide access to a Globus Compute endpoint which can be used to submit jobs.

The Slurm scheduler utilises Fair Share to help with job prioritisation. We also impose general limits on the size and number of jobs submitted by any user.

Depending on the needs of your batch jobs, you may need to specify the partition you want the job to run on. Please see the Hardware page for specifics about the system. If you need to use GPUs, the Using GPUs page will provide generic information to get started.

Slurm job basics

Please see Submitting your first job for a detailed tutorial with instructions and examples. We also have a Slurm reference sheet.

Batch scripts

Jobs on the HPC are submitted in the form of a batch script (.sl) containing the code you want to run and a header of information needed by our job scheduler Slurm.

The following is a template batch script with both the minimum requirements and some additional best practice options included.

#!/bin/bash -e

#SBATCH --account       <projectcode>   # needed if you are in multiple projects
#SBATCH --job-name      BatchJob        # shows up in the queue
#SBATCH --time          00:01:00        # Walltime limit (minutes or HH:MM:SS)
#SBATCH --mem           512MB           # Memory in MB or GB
#SBATCH --cpus-per-task 1               # CPUs
#SBATCH --output        log/%x.%j.out   # send output to the file <job-id>.<job-name>.out

# print the contents of the batch script at the top of the output file for reference
cat $0

# purge and load needed modules
module purge
module load <module-name>

<code to be run goes here>

Submitting

Jobs are submitted to the scheduler using:

sbatch myjob.sl

You should receive an output:

Submitted batch job 1234567

sbatch options can be given on the command line or (as in the example above) in #SBATCH pragmas.

You can find details on its use in the sbatch manual.

Managing and reviewing your Slurm jobs

Job Queue

The whole job queue can be seen using

squeue

Or you can filter to check just your jobs using

squeue --me

You can find details on its use in the squeue manual.

Completed jobs

You can check all jobs submitted by you in the past day using:

sacct

Or since a specified date using:

sacct -S YYYY-MM-DD

Each job will show as multiple lines, one line for the job and then additional lines for each job step within it.

Tip

  • sacct -X Only shows the jobs, not the job steps.
  • sacct --state=PENDING/RUNNING/FAILED/CANCELLED/TIMEOUT Filter jobs by state.
  • sacct --format... changes which fields are displayed out of the 120+ available.

You can find details on its use in the sacct manual.

Cancelling

scancel <jobid> will cancel the job described by <jobid>. You can obtain the job ID by using sacct or squeue.

Tip

  • scancel --me Cancel all jobs submitted by you.
  • scancel {[n1]..[n2]} Cancel all jobs with an id between [n1] and [n2].

You can find details on its use in the scancel manual.

Checking job efficiency

After a job has completed you can get basic usage information with seff <job-id>, which examines the full details about the job from sacct to produce a comparison of the actual resource usage against what was requested:

Cluster: hpc
Job ID: 1234567
State: FAILED
Cores: 48
Tasks: 1
Nodes: 1
Job Wall-time:     0.6%  00:00:04 of 00:12:00 time limit
CPU Utilisation:   1.0%  00:00:02 of 00:03:12 core-walltime
Mem Utilisation:   0.0%  0.00 MB of 260.00 GB

The "CPU Utilisation" represents the average utilisation over the course of the job. The "Mem Utilisation" represents the maximum memory utilisation over the course of the job.

To get a more detailed sense of how your job uses the resources allocated over time, you can use Slurm Native Profiling. Add the following to your batch script before running:

#SBATCH --profile           task

After the job finishes running you can get plots of the resource utilisation by running profile_plot <jobid>, or the raw profile data by running profile_data <jobid>. Both programs have formatting options shown by their --help option.