Tutorial

Putting It Together

30 Minutes

Objectives

• Run a whole pipeline, from preparation to a summary, from one submission script.
• Compare the pipeline's resource use with the job you started with.
• Recognise when a workflow manager would serve you better than Slurm scripts.

Prerequisite

• You can run a job array and chain jobs with dependencies. See Job Arrays and Splitting and Chaining Jobs.

Skipped ahead? Set up here

The finished pipeline

Compare with where you started

Where did the savings come from?

Solution

When to use a workflow manager

#

Choosing a tool

Which of these suggests it is time to move to a workflow manager?

You often need to rerun only the stages whose inputs changed.

Your submission script has grown to many stages, with different dependencies between them.

You need to run the same command on 50 input files.

A single job array handles that well.

Submit

Keypoints

• A short submission script can queue a whole pipeline, with each stage sized for its own needs.
• Measure before and after: the same work can use far fewer resources.
• Workflow managers take over when pipelines outgrow scripts.

What next

• Snakemake and Nextflow on Mahuika.
• For reference: Job Dependencies and Job Arrays.

Quiz Progress

• Answered: 0 / 0 (0%)
• Correct: 0 / 0 (0%) Reset