Welcome to the 2019 ENCODE Users' Meeting Pipeline Workshop

July 8, 2019 ยท View on GitHub

Do this before the workshop

  1. Register by following instructions in the email you received with the subject "Welcome to Using ENCODE in the Cloud".

  2. Open a web browser (Chrome, Safari, or Edge - Firefox is not supported) and go to our workshop server instance on Google Cloud Platform console.

  3. Click on the SSH button under Remote Access. It may sake several seconds to open a connection to the server instance.

WARNING: If it takes too long (>2 minutes) to log in, then switch to a "Cloud Shell" method. Click on the inverse triangle next to "SSH" button and choose "View gcloud command". Click on "RUN IN CLOUD SHELL" button in the bottom-right corner. Push Enter to execute the copied command line. Answer "Y" to the question. Push Enter twice to pass two questions.

  1. Set up your server account: Soft-link a shared configuration file.
$ mkdir -p ~/.caper && cd ~/.caper && rm -f ~/.caper/default.conf
$ ln -s /opt/code/default.conf default.conf
  1. Authenticate yourself to get access to buckets. After running each command, follow the link and copy and paste the authentication key into the console.
$ gcloud auth login --no-launch-browser
$ gcloud auth application-default login --no-launch-browser

To do together during workshop

WARNING: USERS SHOULD NOT FOLLOW THE BELOW STEPS BEFORE THE WORKSHOP.

  1. Submit a workflow to Caper server.
$ caper submit /opt/code/rna-seq-pipeline/rna-seq-pipeline.wdl -i gs://encode-workshop-samples/rna-seq-pipeline/input_workshop_example_SSD.json
# you will see the following message. make sure to remember the workflow_id
# in this example, the workflow_id is f7094621-3d38-48a6-b877-1da2b0cec931
[Caper] submit:  {'id': 'f7094621-3d38-48a6-b877-1da2b0cec931', 'status': 'Submitted'}
  1. Make sure to remember workflow_id of your submitted workflow. You can monitor workflows with:
$ caper list [WORKFLOW_ID]

# you can also find it by your username

$ caper list | grep $USER
  1. Once your workflow is done (marked as Succeeded). Retrieve a metadata.json with the following command:
$ caper metadata [WORKFLOW_ID] > metadata.json
  1. Run Croo with the retrieved metadata.json to organized outputs on --out-dir.
$ croo metadata.json --out-dir gs://encode-workshop-croo/$USER --out-def-json /opt/code/rna-seq-pipeline/output_definition.json

  1. Open a web browser and go to Google Cloud Storage console.

  2. Navigate to your organized output directory under your username. For example, gs://encode-workshop-croo/[YOUR_USER_NAME]/. Click on an HTML file then you will see a nice file table summarizing all outputs with description. Find any bigwig file in it and take a URL for it. That URL will be public so you can use it to visualize the track with your preferred genome browser (for example, you can use this one).

To be done by admins

  1. Run Croo with the retrieved metadata.json to organized outputs locally.
$ cd /srv/scratch
$ mkdir -p test_croo && cd test_croo

$ caper metadata [WORKFLOW_ID] > metadata.json
$ croo metadata.json --out-def-json /opt/code/rna-seq-pipeline/output_definition.json

Setting up a Caper server instance (ADMIN ONLY)

This example is to set up a server instance for the ENCODE workshop 2019 at Seattle. However, this example should also be helpful to set up your own server instance.

WARNING: This section is for admins only. USERS SHOULD NOT FOLLOW THE BELOW STEPS ON THE INSTANCE.

  1. Create an instance with Debian-based Linux (e.g. Ubuntu). Minimum requirements for the server is CPU >=4, Memorsy > 16GB.

  2. Install softwares. Install Caper (Cromwell wrapper) and Croo (Cromwell output organizer).

$ sudo apt-get update && sudo apt-get install -y default-jdk acl python3 python3-pip git wget curl htop
$ sudo pip3 install caper croo
  1. Clone pipeline codes and share them with users. This example will install ENCODE RNA-Seq and Demo pipelines on /opt/code.
$ sudo mkdir /opt/code
$ sudo chown $USER:$USER /opt/code
$ cd /opt/code
$ git clone https://github.com/ENCODE-DCC/rna-seq-pipeline
$ git clone https://github.com/ENCODE-DCC/demo-pipeline
  1. Authenticate yourself.
$ gcloud auth login --no-launch-browser
$ gcloud auth application-default login --no-launch-browser
  1. Create a scratch directory for Caper. Any subdirectories under /srv/scratch will inherit permissions from their parent directory.
$ sudo mkdir /srv/scratch
$ sudo chown $USER:$USER /srv/scratch
$ sudo chmod 777 /srv/scratch
$ sudo setfacl -d -m u::rwx /srv/scratch
$ sudo setfacl -d -m g::rwx /srv/scratch
$ sudo setfacl -d -m o::rwx /srv/scratch
  1. Create a Caper configuration file, which will be shared with all users.
$ touch /opt/code/default.conf
  1. Edit the shared configuration file /opt/code/default.conf. You can comment settings for the ENCODE workshop 2019 and uncomment/define your own gcp-prj, tmp-gcs-bucket and out-gcs-bucket.
[defaults]
cromwell=/opt/code/cromwell-42.jar
java-heap-server=8G

backend=gcp

out-dir=/srv/scratch/caper_out
tmp-dir=/srv/scratch/caper_tmp

#gcp-prj=[YOUR_GOOGLE_PROJECT]
gcp-prj=encode-workshop

#out-gcs-bucket=[YOUR_OUTPUT_BUCKET_FOR_CAPER]
#tmp-gcs-bucket=[YOUR_TMP_BUCKET_FOR_CAPER]
out-gcs-bucket=gs://encode-workshop-outputs/caper_out
tmp-gcs-bucket=gs://encode-workshop-outputs/caper_tmp

max-concurrent-workflows=100
  1. Download Cromwell 42 JAR and share it with all users.
$ cd /opt/code
$ wget https://github.com/broadinstitute/cromwell/releases/download/42/cromwell-42.jar
  1. Soft-link a shared configuration file.
$ mkdir -p ~/.caper && cd ~/.caper
$ ln -s /opt/code/default.conf default.conf

  1. Create Caper's output bucket gs://encode-workshop-outputs.

  2. Make the bucket public by adding a Storage Object Viewer role for allUsers to the bucket. This will allow public HTTP access to all files on the bucket, which will be used to visualize some of pipeline outputs (e.g. bigwigs) on a genome browser.

  3. Give write permission to *ALL WORKSHOP PARTICIPANTS (not for all public users). Add Storage Object Creator role to all participants. This is to give all participants write access to Caper tmp directory gs://encode-workshop-outputs/caper_tmp so that --deepcopy does not make duplicate files on the shared bucket. This will also give them write access to gs://encode-workshop-outputs/croo so that their organized outputs generates from Croo will be write on that bucket directory.

  4. Run a Caper server.

$ caper server

  1. Make all buckets public (Read access to anyone).

  2. Give users the following IAM Roles:

  1. For the whole project

    • Compute Engine > Compute Instance Admin (v1)
    • Compute Engine > Compute OS Login
    • Service Account > Service Account User

  2. For the croo bucket (gs://encode-workshop-croo)

    • Storage Object Admin