Run a list of commands on a set of SSH nodes. With a bit of optional parametrization.

Demo

Features

• Passwordless SSH is all you need.
• Simple config for simple use cases, flexible config for advanced ones.
• Two modes:
  • Broadcast mode runs each command on every node.
  • Queue mode runs each command once on the next free node.
    • Hosts can expose integer slots and jobs can request integer slots needed to run (good for GPU node slicing).
• Modify the file-based queue (queue.yaml) while Pegasus is running.
• Parametrize hosts and commands.

Getting Started with Examples

To use Pegasus,

1. Install Pegasus, either from GitHub Release or cargo install pegasus-ssh.
2. Setup passwordless SSH for your nodes.
3. Populate hosts.yaml and queue.yaml, and run Pegasus.

Pegasus will remove one entry at a time from the top of queue.yaml and move it to consumed.yaml as it begins to execute it.

Queue Mode: Getting a Bag of Jobs Done

Run four Python commands using two nodes.

# hosts.yaml
- node-1
- node-2

# queue.yaml
- . /opt/miniconda3/etc/profile.d/conda.sh; python train.py --bs 8
- . /opt/miniconda3/etc/profile.d/conda.sh; python train.py --bs 16
- . /opt/miniconda3/etc/profile.d/conda.sh; python train.py --bs 32
- . /opt/miniconda3/etc/profile.d/conda.sh; python train.py --bs 64

$ pegasus q  # stands for Queue

Broadcast Mode: Terraforming Nodes

Run identical commands for multiple nodes.

# queue.yaml
- mkdir workspace
- cd workspace && git clone https://github.com/jaywonchung/dotfiles.git
- . workspace/dotfiles/install.sh

$ pegasus b  # stands for Broadcast

Parallelizing Execution with Node Parameters

Split nodes into sub-nodes that run commands in parallel. Below, four SSH connections are kept, and four commands run in parallel.

# hosts.yaml
- hostname:
    - node-1
    - node-2
  container:
    - gpu0
    - gpu1

When parametrizing nodes, just make sure you specify the hostname key.

You can use these parameters in your commands. By the way, the templating engine is Handlebars.

# queue.yaml
- docker exec {{ container }} python train.py --bs 8
- docker exec {{ container }} python train.py --bs 16
- docker exec {{ container }} python train.py --bs 32
- docker exec {{ container }} python train.py --bs 64

Four sub-nodes and four jobs. So all jobs will start executing at the same time.

Resource-Aware Scheduling with Slots

When running GPU jobs, you often want multiple jobs to share a single node. For example, an 8-GPU node can run four 2-GPU jobs concurrently. Pegasus supports this with slots.

# hosts.yaml
- hostname:
    - gpu-node-1
    - gpu-node-2
  slots: 8

Each host now has 8 slots (e.g., 8 GPUs). Jobs can declare how many slots they need:

# queue.yaml
# 8-GPU job: uses all GPUs on one node
- command:
    - CUDA_VISIBLE_DEVICES={{slots}} python train.py --model llama-70b
  slots:
    - 8

# 4-GPU jobs: two can run concurrently on an 8-GPU node
- command:
    - CUDA_VISIBLE_DEVICES={{slots}} python train.py --model llama-7b
    - CUDA_VISIBLE_DEVICES={{slots}} python train.py --model mistral-7b
  slots:
    - 4

# 2-GPU jobs: four can run concurrently on an 8-GPU node
- command:
    - CUDA_VISIBLE_DEVICES={{slots}} python train.py --model {{model}}
  model:
    - bert
    - roberta
    - gpt2
    - t5
  slots:
    - 2

The {{slots}} variable is automatically injected with the allocated slot indices (e.g., 0,1,2,3 for a 4-slot job). Use it with CUDA_VISIBLE_DEVICES to control which GPUs your job uses.

Notes:

• Jobs without slots default to 1 slot.
• Hosts without slots default to 1 slot.
• Jobs are scheduled in FIFO order. If the next job doesn't fit, Pegasus waits for slots to free up. Therefore, ordering your jobs by descending slot count is recommended lower makespan.
• All jobs are still expected to fit in a single host. A job requiring more slots than any single host can provide is skipped and reported as an error at the end.

Allocation Policies

By default, Pegasus uses first-fit allocation: it tries to find a contiguous block of slots (helpful for nodes with only NVLink bridges), but falls back to any available slots if needed. For workloads that strictly require aligned GPU blocks, you can use buddy allocation policy.

# queue.yaml
- command:
    - CUDA_VISIBLE_DEVICES={{slots}} python train_distributed.py
  slots: 4
  allocation_policy: buddy

Two policies are supported:

• first_fit (default): Tries even-aligned contiguous blocks, then any contiguous block, then falls back to any available slots.
• buddy: Strict buddy allocation. Requires power-of-2 aligned blocks with no fallback. A 2-slot job gets 0-1, 2-3, 4-5, or 6-7 (never 1-2 or 3-4). A 4-slot job gets 0-3 or 4-7 (never 2-5). If no aligned block is available, the job waits.

Notes:

• allocation_policy in the queue file takes a single value, not a list of values.
• buddy allocation requires the slot count to be a power of 2 (1, 2, 4, 8). Non-power-of-2 slot counts with buddy are skipped and reported as errors.
• The {{allocation_policy}} template variable is available in commands if needed.

Parametrizing Commands for Conciseness

If you can parametrize nodes, why not commands?

# queue.yaml
- command:
    - docker exec {{ container }} python train.py --bs {{ bs }}
  bs: [8, 16, 32, 64]

This results in the exact same jobs with the example above. When parametrizing commands, just make sure you specify the command key.

Quiz

How many commands will execute in Queue mode?

# hosts.yaml
- hostname:
    - node-1
    - node-2
  laziness:
    - 1
- hostname:
    - node-3
  laziness:
    - 2

# queue.yaml
- echo hi from {{ hostname }}
- command:
    - for i in $(seq {{ low }} {{ high }}); do echo $i; sleep {{ laziness }}; done
    - echo bye from {{ hostname }}
  low:
    - 1
    - 2
  high:
    - 3
    - 4

Note that although echo bye from {{ hostname }} doesn't really use the low or high parameters, it will run 2 * 2 = 4 times regardless.

The answer is 1 + 2 * 2 * 2.

Lock Mode: Modifying the Queue

queue.yaml is actually the queue.

Pegasus removes the first entry in queue.yaml whenver there's a free host available. If you delete entries before Pegasus pulls it, they will not execute. If you add entreis to queue.yaml, they will execute.

Q. Why do I need this?

Think about when the number of remaining commands is less than the number of free nodes. Without a way to submit more jobs to Pegasus, those free nodes will stay idle until all the commands finish and you start a fresh new instance of Pegasus.

By providing a way to add to the queue while commands are still running, users may achieve higher node utilization. Being able to delete from the queue is just a byproduct; adding to the queue is the key feature.

Q. But that's a race condition on queue.yaml.

Lock mode will lock queue.yaml and launch a command line editor for you.

$ pegasus l --editor nvim  # l stands for Lock

Editor priority is --editor > $EDITOR > vim. When you save and exit, the queue lock is released and Pegasus is allowed access to queue.yaml.

Q. What if Pegasus terminates before I add to queue.yaml?

Enable daemon mode, and Pegasus will not terminate even if queue.yaml is empty. It will stand waiting for you to populate queue.yaml again, and execute them.

$ pegasus q --daemon

Advanced templating with Handlebars

Handlebars is a templating engine, and Pegasus uses Handlebars to fill in parameters into hostnames and commands.

Look at this example to see how this can be useful:

# queue.yaml
- command:
  - python main.py --model-path {{ model }} --output-path {{ replace model "/" "--" }}.json
- model:
  - facebook/opt-13b
  - facebook/opt-30b
  - facebook/opt-66b

The commands above expand to:

# queue.yaml
- python main.py --model-path facebook/opt-13b --output-path facebook--opt-13b.json
- python main.py --model-path facebook/opt-30b --output-path facebook--opt-30b.json
- python main.py --model-path facebook/opt-66b --output-path facebook--opt-66b.json

Here, facebook/opt-13b is the name of the model in one chunk (and you need it to have the / so that Hugging Face understands it), but if you just tell your script to output results in facebook/opt-13b.json, it'll create a directory called facebook and save opt-13b.json inside it. That's not good. Instead, we just used the replace helper from the String transformation section of handlebars_misc_helpers to pretty much run model.replace("/", "--").

Details

Pegasus uses sh -c to run commands

The commands you put in queue.yaml are wrapped individually inside sh -c and executed via SSH. In your hosts, sh may be symlinked to bash, dash, or something else, and certain syntax may or may not be allowed (e.g., double brackets).

queue.yaml

This is the queue file. Entries in queue.yaml are consumed from the top, one by one. Also, entries are consumed only when a new host is available to execute new commands. Consumed entries are immediately appended to consumed.yaml in "canonical form", where every entry has a command key. Thus, you might do something like tail -n 2 consumed.yaml > queue.yaml to re-execute your previous single-line command.

As mentioned earlier, always use the Lock Mode when you need to modify queue.yaml.

Broadcast Mode

In broadcast mode, hosts are kept in sync with each other. That is, the next command is fetched from queue.yaml and executed on all hosts when all the hosts are done executing the previous command.

Consider the following situation:

              fast-host   slow-host
- command1     success     success
- command2     success      fail!
- command3     success
- command4     running

In this case, we would want to prepend a undo command for command2 (e.g., rm -rf repo || true) and restart from that, but fast-host is already far ahead, making things complicated. Thus, especially when you're terraforming nodes with Pegasus, keeping hosts in sync should be beneficial.

There is also a -e or --error-aborts flag in Broadcast Mode, which aborts Pegasus automatically when a host fails on a command.

Cancelling and killing

For the special case when you are running jobs on localhost, a Ctrl-c will be propagated to all running jobs and also stop Pegasus itself.

However, it is very difficult to find a generic way to cancel commands that started running via SSH (See also #11). Therefore, the caveat of Pegasus at the moment is that it works very well when things go well, but it's difficult to cancel and kill when things go not quite as planned. You need to walk into every node and manually kill commands. That said, you can still use Broadcast mode to automate that.

A work-in-progress idea in the jw-remote-cancel branch is to wrap the command in a shell script that checks the parent PID of the process, and if it becomes 1, it means that the SSH session was killed, and the command needs to be killed as well.

Development

bash scripts/check.sh