Learning path 04 · Intermediate
Build workflows that travel
Workflow engineering is shared across scientific tracks. Learn one engine first, then use a familiar analysis to understand dependencies, portability, and failure recovery.
Put it into practiceOne analysis, two workflow engines →IntermediateFrom scripts to reproducible workflows
Learn the common ideas before committing to a particular engine.
Before you begin: Shell basics, a working analysis script, and software environments.
What to understand
- Tasks, input/output contracts, dependency graphs, and configuration.
- Versioned software, containers, sample sheets, resource declarations, and logs.
- Small fixtures, expected outputs, restart behavior, and provenance.
What you’ll be able to do
Draw and implement a small analysis as a dependency graph.
Try this
Run the workflow-basics project as a plain Python script, then reproduce its expected output with one engine.
Keep in mind
A workflow engine cannot repair unclear scientific assumptions or an incorrect analysis step. Validate the underlying computation first.
A few good resources
- Nextflow documentation ↗
Reference processes, channels, modules, profiles, executors, and dependency handling while building a small pipeline.
Documentation · Intermediate · Free; compute may cost · English - Snakemake documentation ↗
Understand rules, file dependencies, software environments, execution, and reporting.
Documentation · Intermediate · Free; compute may cost · English - nf-core documentation ↗
Learn conventions for using and developing community pipelines, including configuration and contribution guidance.
Documentation · Intermediate · Free · English - Apptainer user guide ↗
Learn container execution and reproducibility concepts relevant to shared research computing systems.
Documentation · Intermediate · Free; infrastructure may cost · English
IntermediateNextflow and nf-core
Run, configure, and develop portable pipelines using dataflow concepts.
Before you begin: Workflow concepts and command-line experience.
What to understand
- Processes, channels, workflow composition, and modules.
- Parameters, profiles, containers, executors, and resume behavior.
- Using nf-core pipelines and understanding their documentation and quality conventions.
What you’ll be able to do
Run a small Nextflow workflow, change a parameter, and explain its task dependencies.
Try this
Complete the official introductory training, then run main.nf in the workflow-basics project and compare its output with the reference file.
Keep in mind
Start with a small local test profile. Check the data, storage, and compute requirements before running a production pipeline.
A few good resources
- Nextflow community training ↗
A guided route from running a first pipeline to composing and configuring workflows.
Course · Intermediate · Free · English - Nextflow documentation ↗
Reference processes, channels, modules, profiles, executors, and dependency handling while building a small pipeline.
Documentation · Intermediate · Free; compute may cost · English - nf-core documentation ↗
Learn conventions for using and developing community pipelines, including configuration and contribution guidance.
Documentation · Intermediate · Free · English - nf-core/rnaseq ↗
A production workflow to examine after a smaller tutorial; learn its inputs, profiles, QC outputs, and documented options.
Workflow · Intermediate · Code free; compute and storage may cost · English
IntermediateSnakemake
Express analysis steps as rules connected through their input and output files.
Before you begin: Workflow concepts and basic Python.
What to understand
- Rules, targets, wildcards, configuration, and dependency graphs.
- Software environments, containers, resource declarations, and execution profiles.
- Dry runs, logs, reports, and testing small examples.
What you’ll be able to do
Write a small rule-based workflow and inspect what will run before executing it.
Try this
Run a dry run and then the Snakefile in the workflow-basics project. Change the input FASTA and predict which output should be recomputed.
Keep in mind
Make all dependencies explicit. Hidden files, undeclared parameters, and changing environments undermine reproducibility.
A few good resources
- Snakemake documentation ↗
Understand rules, file dependencies, software environments, execution, and reporting.
Documentation · Intermediate · Free; compute may cost · English - Snakemake general-use tutorial ↗
Build and extend an example workflow while learning dependency handling, configuration, and execution.
Tutorial · Intermediate · Free · English - Apptainer user guide ↗
Learn container execution and reproducibility concepts relevant to shared research computing systems.
Documentation · Intermediate · Free; infrastructure may cost · English