ConnectedReads Quick Start on Microsoft Azure

February 14, 2020 · View on GitHub

This is an explanation of how to launch ConnectedReads in Microsoft Azure.

Background

Microsoft Azure is a cloud computing service. Throught AZTK (Azure Distributed Data Engineering Toolkit) cloud-based managed Spark and Hadoop service offered on Microsoft Azure Platform.

Preliminaries

install ansible: pip install --user ansible
install aztk: pip install aztk
build ConnectedReads package
build Adam package (forked and modified by Atgenomix)

Launch ConnectedReads via aztk

Step 1. Create Azure Batch account and Storage account

Please create your Azure Batch account and Storage account first.

Step 2. Initalize aztk package

aztk spark init

Step 3. Modify configurations for Spark cluster

.aztk/secrets.yaml, .aztk/core-site.xml, .aztk/spark-defaults.conf

Please specify your JAR files and the FASTQ files you would like to process.

Step 5. Upload your FASTQ to Microsoft Data Lake Storage

Please upload your data to Microsoft Data Lake Storage.

Step 6. submit job by aztk

Please edit the folder information on scripts/run_on_azure.sh and then run it.

cd scripts
sh ./run_on_azure.sh