Guide to Installing Requirements for a Local Instance of IRIDA Next

February 18, 2025 · View on GitHub

Links

Before you begin ensure your user account has sudo privileges configured.

# Replace $USER with desired or existing username to grant and set up sudo privledges
$ sudo useradd $USER
$ sudo passwd $USER
$ sudo usermod -aG sudo $USER

Installing IRIDA Next

asdf is a tool version manager that uses a single .tool-versions file to ensure consistency. It simplifies development by replacing multiple CLI version managers with a unified interface and configuration file, supporting various tools through an easy plugin interface.

Required dependencies: git and curl

1. Download asdf

# replace `v0.14.0` with the latest release tag available on the asdf GitHub release page
git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.14.0

2. Add asdf to your shell configuation

To integrate asdf with your shell you need to add the following lines to your shell configuration file (~/.bashrc, ~/.bash_profile, etc.)

Edit bash profile using a text editor

vi ~/.bashrc

Add the following lines at the end of the file

# set up .asdf directory and file paths
export ASDF_DIR="$HOME/.asdf"

. ~/.asdf/asdf.sh

. ~/.asdf/completions/asdf.bash

Source the shell configuration

source ~/.bashrc

Verify the installation

asdf --version
# you should see the version number of asdf printed if it has been installed correctly

Installing the required IRIDA Next plugins

To maintain consistency between IRIDA Next and your global installation, verify the .tool-versions file in the irida-next repository before installing the necessary plugins.

1. Install IRIDA Next and Verify Plugin Versions

cd /path/to/git/repos
git clone https://github.com/phac-nml/irida-next
# Check .tool-versions and document plugin versions for installations
cd irida-next
less .tool-versions

2. Install the Node.js Plugin

node.js allows developers to run JavaScript on the server-side, facilitating scalable and efficient web applications with its event-driven architecture and extensive module ecosystem.

Required dependencies: apt-get install dirmngr gpg curl and gawk

Install the plugin

asdf plugin add nodejs

Install and set a global version

# To check the available versions of the tool and find the version used by IRIDA Next.
asdf list all nodejs
# Install the desired $VERSION
# You can substitute the desired version with 'latest' to install the newest release of the plugin. This may cause incompatibility with IRIDA Next and is not recommended.
asdf install nodejs $VERSION
# Verify the install
which node
# Set global version for projects to use by default
asdf global nodejs $VERSION
# Verify that node.js was added to the .tools-versions file
less /home/$USER/.tools-versions
node --version

3. Install the pnpm Plugin

pnpm is a fast, disk space-efficient package manager for JavaScript.

Required dependencies: curl grep cut sort xargs tar and ln

Install the plugin

asdf plugin add pnpm

Install and set a global version

# To install the desired $VERSION
asdf install pnpm $VERSION
which pnpm
# Set global version for projects to use by default
asdf global pnpm $VERSION
# Verify that pnpm was added to the .tools-versions file
less /home/$USER/.tools-versions
pnpm --version

4. Install the ruby Plugin

ruby is a high-level interpreted programming language known for its simplicity, readability, and productivity.

Required dependencies: apt-get and install

Install the ruby-build dependencies

sudo apt-get install autoconf patch build-essential rustc libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libgmp-dev libncurses5-dev libffi-dev libgdbm6 libgdbm-dev libdb-dev uuid-dev
# Depending on your OS version, libgdbm6 may not be available. In that case, try an earlier version, such as libgdbm5

Install the plugin

asdf plugin add ruby

Install and set a global version

# To install the desired $VERSION
asdf install ruby $VERSION
which ruby
# Set global version for projects to use by default
asdf global ruby $VERSION
# Verify that ruby was added to the .tools-versions file
less /home/$USER/.tools-versions
ruby --version

5. Install the postgres Plugin

postgres is an open-source, powerful, and advanced relational database management system (RDBMS) known for its robustness, extensibility, and compliance with SQL standards.

Required dependencies: apt-get install and curl

Install postgres specific dependencies

# Ubuntu
sudo apt-get install linux-headers-$(uname -r) build-essential libssl-dev libreadline-dev zlib1g-dev libcurl4-openssl-dev uuid-dev icu-devtools libicu-dev
# Ubuntu (WSL)
sudo apt-get install build-essential libssl-dev libreadline-dev zlib1g-dev libcurl4-openssl-dev uuid-dev icu-devtools libicu-dev

Install the plugin

asdf plugin add postgres

Install and set a global version

# To install the desired $VERSION
asdf install postgres $VERSION
which postgres
# Set global version for projects to use by default
asdf global postgres $VERSION
# Verify that ruby was added to the .tools-versions file
less /home/$USER/.tools-versions
postgres --version

6. Ensure bundler and remaining dependencies are installed

# Ensure you are in the irida-next directory
cd /path/to/irida-next
# Ensure bundler installed
gem install bundler
# Install remaining dependencies
bundle && pnpm install

7. Setup Postgres Server

Start by running:

# Set correct $USER and postgres #VERSION
/home/$USER/.asdf/installs/postgres/$VERSION/bin/pg_ctl -D /home/$USER/.asdf/installs/postgres/$VERSION/data -l logfile start

Create a new role (user) and set the password

createuser -s test -P -U postgres
# When prompted for a password for the `test` role above, set the password as `test`. These are the credentials used by the development and test databases.

8. Generate and configure the environment credentials

Refer to GA4GH WES Sapporo Setup for how to setup and configure the IRIDA Next environment credentials for development purposes.

9. Initialize the databases

Run:

cd /path/to/irida-next
bin/rails db:create db:migrate db:seed

10. Register pipelines

Refer to Registering pipelines in IRIDA Next for how to register Nextflow pipelines.

11. Restarting

Rebuild and restart the sapporo docker container, restart postgres, and start Irida Next following the steps found here: Running a local instance of IRIDA Next.

GA4GH WES Sapporo Setup for development

Prerequisites

You will need to have a Docker Compose plugin installed to manage Sapporo.

Install Docker Compose plugin on Ubuntu

sudo apt-get update
sudo apt-get install docker-compose-plugin
# Verify installation
docker compose version
# Should be version v2.26.1 or newer

Add yourself to the docker group to run docker commands

sudo groupadd docker
sudo usermod -aG docker $USER
# Reboot, log out/log in, or run the following command
newgrp docker
#If you still have issues with running docker, you may need to change the permissions of the docker socket using the following command
sudo chmod 666 /var/run/docker.sock

Configure IRIDA Next environment developer credentials

Check that your active storage service is set to :local in config/environments/development.rb

# check if the active storage service is set to :local
grep -q "config.active_storage.service = :local" config/environments/development.rb && echo "Active storage service is set to :local" || echo "Active storage service is not set to :local"

Configure environment developer credentials

EDITOR="vim --nofork" bin/rails credentials:edit --environment development

# Add the following to the credentials file and save in the text editor
ga4gh_wes:
  server_url_endpoint: 'http://localhost:1122/'

Make sure that there's no comment characters (#) preceding these lines!
You may replace vim with any text editor of your preference.

Setup Sapporo (WES implementation)

Download and run the PHAC-NML Sapporo fork in dev docker mode. If docker permissions are setup correctly you should not have to use sudo when running these commands.

cd ~/path/to/git/repos
git clone https://github.com/phac-nml/sapporo-service
cd sapporo-service
# This branch has a custom docker compose script for IRIDA Next
git checkout irida-next
# Replace /PATH/TO/IRIDA/NEXT/REPO with your irida next repo path.
# This allows the docker container read/write access to the repo
# This is needed for it to read the input files, and write the output files back to the blob directories
IRIDA_NEXT_PATH=/PATH/TO/IRIDA/NEXT/REPO MYUID="$(id -u)" MYGID="$(id -g)" docker compose -f compose.irida-next.yml up -d --build
IRIDA_NEXT_PATH=/PATH/TO/IRIDA/NEXT/REPO MYUID="$(id -u)" MYGID="$(id -g)" docker compose -f compose.irida-next.yml exec app bash
# Within docker container, start sapporo
sapporo

In a new terminal confirm Sapporo is running

# This should output all the service information for this ga4gh wes instance
curl -X GET http://localhost:1122/service-info

Registering pipelines in IRIDA Next

Pipelines are registered at postgres server startup, requiring restarts to register new pipelines added to the configuration file.

Currently only Nextflow pipelines are supported and they must have a Github repository.
- Ensure there is a nextflow_schema.json file at the top level of the repository and a schema_input.json file located in an assets directory.

Setup the configuration file to register pipelines

The pipelines.json configuration file to register pipelines is stored in the /home/$USER/irida-next/config/pipelines/ directory.

The pipelines.json file should be in the format below and must contain the following:
- URL of the pipeline Github repository
- name of the pipeline
- description of the pipeline
- versions of the pipeline that should be available to launch. Note: The name in the versions refers to the -r flag used by nextflow.

E.g.

[
  {
    "url": "https://github.com/phac-nml/gasnomenclature",
    "name": "phac-nml/gasnomenclature",
    "description": "Gas Nomenclature assignment pipeline",
    "versions": [
      {
        "name": "dev"
      }
    ]
  },
  {
    "url": "https://github.com/phac-nml/speciesabundance",
    "name": "phac-nml/speciesabundance",
    "description": "SpeciesAbundance Pipeline",
    "versions": [
      {
        "name": "2.1.1"
      }
    ]
  }
]

Running a local instance of IRIDA Next

1. Run the PHAC-NML Sapporo fork in dev docker mode

cd /path/to/sapporo-service
git checkout irida-next
IRIDA_NEXT_PATH=/PATH/TO/IRIDA/NEXT/REPO MYUID="$(id -u)" MYGID="$(id -g)" docker compose -f compose.irida-next.yml up -d --build
IRIDA_NEXT_PATH=/PATH/TO/IRIDA/NEXT/REPO MYUID="$(id -u)" MYGID="$(id -g)" docker compose -f compose.irida-next.yml exec app bash
sapporo

Open a new terminal to confirm that Sapporo is running and then continue to step 2

curl -X GET http://localhost:1122/service-info

2. Start the postgres server

/home/$USER/.asdf/installs/postgres/14.6/bin/pg_ctl -D /home/$USER/.asdf/installs/postgres/14.6/data -l logfile start

3. Serve the local instance of the IRIDA Next web application

cd /path/to/irida-next
bin/dev

Navigate to your browser to http://localhost:3000

The first time you serve IRIDA Next, you will need to Register a local account. Once completed, you can use IRIDA Next as usual in the web browser.

Additional Installation and Usage Details

Using Another IP Address

If you want to use another IP address than localhost, you'll need to make the following changes:

Developer Credentials

EDITOR="vim --nofork" bin/rails credentials:edit --environment development

ga4gh_wes:
  server_url_endpoint: 'http://IP:1122/'

Make sure that there's no comment characters (#) preceding these lines!

Docker Conatiner

https://github.com/phac-nml/sapporo-service/blob/irida-next/compose.irida-next.yml#L32

    ports:
      - ip:1122:1122

Make sure to rebuild the docker container, relaunch, and start sapporo.

Irida Next

Procfile.dev

web: bin/rails server -b ip -p 3000

Restarting

To be safe, rebuild and restart the sapporo docker container, restart postgres, and restart Irida Next.

Pipelines that Require Databases

When selecting databases remember that Sapporo operates within a Docker container and does not have full access to the host filesystem.
- To address this, you can provide the full path to the location where the GitHub repository codes are downloaded by the Sapporo service:

/PATH/TO/sapporo-service/run/.nextflow/assets/phac-nml/...

Finding Error Logs from a Workflow Execution in Local IRIDA Next Instance

If a workflow execution stalls with an error state and a RUN ID is provided, you can find the error logs at:

# Replace `[RUN ID]` with the run ID from IRIDA Next

#Remote error logs
http://localhost:1122/runs/[RUN ID]

# Local error logs
# Replace [##] with the first two digits of the RUNID
sapporo-service/run/[##}/[RUN ID]/.nextflow.log

Updating IRIDA Next

In order to update IRIDA Next the following steps can be taken:

cd path/to/irida-next
git fetch origin
git pull origin main
# Resolve any conflicts that arise with the pull, such as changes made to the `pipelines.json` file

You may need to repeat the installation of IRIDA Next dependencies and/or update the existing asdf plugins.

To update IRIDA Next dependencies:

# Ensure newest bundler is installed
gem install bundler
# Check remaining dependencies
bundle && pnpm install
# Re-initialize the database
bin/rails db:create db:migrate db:seed

If prompted to update plugins to new versions, check the irida-next/.tool-versions file and refer to the section Installing the required IRIDA Next plugins

E.g.

# May need to update the plugin in order to install the required version
asdf plugin update ruby
asdf install ruby $VERSION
asdf global ruby $VERSION
ruby --version
# If you no longer need the old version of the plugin you can remove it
asdf uninstall ruby <old-version>

Make sure to rebuild and restart the Sapporo Docker container, restart the postgres server, and start IRIDA Next.

Installng asdf core