Nextflow

Nextflow is a flexible, powerful workflow manager used extensively in Bioinformatics. While Nextflow can be configured to submit Slurm jobs for individual tasks within the workflow, we recommend you NOT use the 'slurm' executor but instead use the 'local' executor.

Nextflow has many customization options that can be adjusted to best use the resources on the cluster. You may wish to examine the workflow itself to see if there are specific adjustments made for the individual processes within the workflow.

Nextflow within a job

We generally prefer you run Nextflow within the resources requested by a single job. To do this, you will need to create a configuration file that lists the same resources as the job and then provide that to Nextflow when it runs.

For example, we can run the wf-transcriptome workflow tutorial data from the Oxford Nanopore group, see Workflow Transcriptomes for more information.

The recommended CPUs and memory on that page is high for demo data, so we'll modify the minimum requirements of 8 CPUs and 16 GB memory. You should create a configuration file, which for this example we'll call nextflow.config that contains the following text.

process {
    executor = 'local'
    memory = '16G'
    cpus = '8'
}

In the job request, it might be

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --mem=18gb
#SBATCH --cpus-per-task=8

When running nextflow, the configuration file is specified with the -c option.

Nextflow formats output with colors and dynamic status when run interactively; however, when output is redirected to a file, those can make the output very long and basically unreadable. Use the -ansi-log false option to suppress colorization. Combining those,

nextflow run epi2me-labs/wf-transcriptomes \
    -ansi-log false \
    -c ./nextflow.profile
    . . . . [ other options ] . . . .