Tutorial

Splitting and Chaining Jobs

45 Minutes

Objectives

• Split a job into parts that each request only the resources they need.
• Make a job wait for another job with --dependency.
• Queue a whole workflow from one submission script.

Prerequisite

• You can find why a job failed. See Debugging Failed Jobs.
• The profile from Measuring Job Efficiency is the starting point for the split.

Skipped ahead? Set up here

Split by resource needs

Read and write less

Make a job wait for another

#

Dependencies

Job 1001 is running and job 1002 has failed. Which of these jobs can start once their dependency is checked?

--dependency after:1001

after only needs job 1001 to have started.

--dependency afterok:1002

Job 1002 failed, so this job will never start.

--dependency afternotok:1002

afternotok waits for the job to fail, which it has.

--dependency afterok:1001

Job 1001 is still running. This job waits until it finishes successfully.

Submit

Queue a whole workflow

Why queue every stage at once?

Solution

Keypoints

• Split a job where its parts need different resources.
• --dependency afterok:JOB_ID makes a job wait for another to finish successfully.
• sbatch --parsable gives the job ID, so a script can queue a whole workflow.

What next

• Next lesson: Job Arrays.
• For reference: Job Dependencies and Temporary Directories.

Quiz Progress

• Answered: 0 / 0 (0%)
• Correct: 0 / 0 (0%) Reset