sourmash-queue

July 9, 2021 · View on GitHub

A queue using celery to run https://sourmash.readthedocs.io/ commands.

Initially only gather supported.

It requires python 3.6+

Setup

  1. Once you have cloned this repo, create a virtual environment for it

    python -m venv virtualenv
source virtualenv/bin/activate

  2. Install the dependencies:

    pip install -r requirements.txt

  3. [Optional] Install redis. For example:

    wget https://download.redis.io/releases/redis-6.2.4.tar.gz
tar xvfs redis-6.2.4.tar.gz
cd redis-6.2.4
make

    Edit its config to make it available from a different VM and disable protected-mode.

    And run the service:

    ./redis-6.2.4/src/redis-server redis-6.2.4/redis.conf

  4. Edit the settings.py file to point to the right paths.

  5. Start the celery worker

    nohup celery -A tasks worker --loglevel=INFO > celery.log &

    ⚠️TODO: Setup this as a systemd or something more reliable

  6. [Optional] Setup the web monitoring tool flower. You can use to see the running/pending tasks and workers on http://[host]:5555

    nohup celery -A tasks flower &

    ⚠️TODO: Setup this as a systemd or something more reliable