phac-nml/submitdatairidanext: Output

Introduction

This document describes the output produced by the pipeline.

The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory.

• assembly: very small mock assembly files for each sample
• generate: intermediate files used in generating the IRIDA Next JSON output
• pipeline_info: information about the pipeline's execution
• simplify: simplified intermediate files used in generating the IRIDA Next JSON output
• summary: summary report about the pipeline's execution and results

The IRIDA Next-compliant JSON output file will be named iridanext.output.json.gz and will be written to the top-level of the results directory. This file is compressed using GZIP and conforms to the IRIDA Next JSON output specifications.

Pipeline overview

The pipeline includes two subworkflows, one for submitting to SRA and one for submitting to ENA.

Submit to SRA

• Create SRA AddFiles XML - Creates the <AddFiles> section of the SRA submission.xml file for each sample
• Create SRA Submission XML - Creates the complete submission.xml file by combining the AddFiles XML for each sample.
• Upload to SRA - Uploads reads and submission.xml file to the SRA using FTP.

Submit to ENA

• Create ENA Upload Manifest - Creates an ENA upload_manifest.json file for each sample
• Combine ENA Upload Manifests - Combines all upload_manifest.json files into a single file to facilitate uploading multiple samples from one process.
• Upload to ENA - Uploads reads and upload_manifest.json files to the ENA using the ena-webin-cli tool.

Common

• Simplify IRIDA JSON - Simplifies the sample JSONs by limiting nesting depth
• IRIDA Next Output - Generates a JSON output file that is compliant with IRIDA Next
• Pipeline information - Report metrics generated during the workflow execution

Submit to SRA

Create SRA Addfiles XML

Output files

• create_sra_addfiles_xml/
  • AddFiles XML: ID_sra_addfiles.xml

Create SRA Submission XML

Output files

• create_sra_submission_xml/
  • Submission XML: submission.xml

Upload Reads to SRA

Output files

• upload_reads_to_sra/
  • Upload Log: ID_sra_upload_log.txt
  • Upload Metadata: ID_upload_metadata.csv

Submit to ENA

Create ENA Upload Manifest

Output files

• create_ena_upload_manifest/
  • Upload Manifest JSON: ID_ena_upload_manifest.json

Upload Reads to ENA

Output files

• upload_reads_to_ena/
  • Upload Log: ID_ena_upload_log.txt
  • Upload Metadata: ID_upload_metadata.csv

Common

Upload Checker

Output files

• upload_checker/
  • Error Report: errors.csv

IRIDA Next Output

Output files

• /
  • IRIDA Next-compliant JSON output: iridanext.output.json.gz

Pipeline information

Output files

• pipeline_info/
  • Reports generated by Nextflow: execution_report.html, execution_timeline.html, execution_trace.txt and pipeline_dag.dot/pipeline_dag.svg.
  • Reports generated by the pipeline: pipeline_report.html, pipeline_report.txt and software_versions.yml. The pipeline_report* files will only be present if the --email / --email_on_fail parameter's are used when running the pipeline.
  • Reformatted samplesheet files used as input to the pipeline: samplesheet.valid.csv.
  • Parameters used by the pipeline run: params.json.

Nextflow provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.