Skip to content
Contact Support

Tutorial

Debugging Failed Jobs

45 Minutes

Objectives

  • Find why a job failed, using its state, exit code and output files.
  • Tell apart a script error, running out of memory, and running out of time.
  • Test a fix quickly, without wasting resources or misusing the debug QoS.

Prerequisite

  • You can pass arguments to a batch script and override its options when you submit. See Reusable Job Scripts.
Skipped ahead? Set up here

Where to look

Common failures

Diagnose the failure

Solution
#

Job states

sacct shows your job's state as OUT_OF_MEMORY. What should you change?

Test a fix

Keypoints

  • A job's output files, state and exit code together usually show why it failed.
  • OUT_OF_MEMORY and TIMEOUT mean the job needs more memory or time, not a change to the script.
  • Check inputs at the start of a script, so it fails early with a clear message.
  • Test fixes on a small input. Only use the debug QoS for one-off tests.

What next