Wordle

In this example, we demonstrate how to train Qwen3-1.7B to play Wordle. This will require a SFT warmup to learn the format and, finally, multi-turn RL against the wordle taskset to improve performance.

The commands in this example were designed to be run on 2 GPUs (one trainer and one inference GPU). It is possible to run on less or more GPUs using different deployment strategies. If you run on a different setup, you may need to adjust the start commands.

Setup

The taskset is included through the Verifiers workspace. After syncing the repository, verify it with:

uv run python -c "import wordle"

Start the pre-layouted tmux session which we will use to run all experiments and view logs conveniently

bash scripts/tmux.sh

Before training, we want to get a baseline score and test how well Qwen3-1.7B does out-of-the-box in the wordle environment so that we quantify our training effect. To do so, first start a local inference server to serve Qwen3-1.7B.

# Run this in the `Inference` pane
uv run inference --vllm.model Qwen/Qwen3-1.7B

Then, use the eval entrypoint to evaluate the model in the wordle environment. We evaluate on the 20 evaluation examples which are distinct words not appearing in the training set of the environment. We constrain the response length to 1024 tokens as this should be more than enough to play a full game.

# Run this in the `Trainer` pane
uv run eval wordle --harness.id null -m Qwen/Qwen3-1.7B --client.base-url http://localhost:8000/v1 -n 20 -r 3 --sampling.max-tokens 1024 --no-push

We got an average reward of ~0.2 across the 20x3 rollouts. From the summary, we can see that the most of the reward is coming from format and partial rewards. In fact, the model does not guess the correct word within any game, leading to a win rate of 0%. Looking at some samples, it is evident repeatedly submitting guesses in the wrong format and is not able to revise its strategy from the environment feedback. Let's do some SFT warmup to get the model to learn the format of the environment.

SFT

We will fine-tune PrimeIntellect/Qwen3-1.7B (HF), which is a clone of Qwen/Qwen3-1.7B (HF) with a chat template suitable for multi-turn RL, on willcb/V3-wordle (HF). It is a multi-SFT dataset with traces of Wordle games from the environment. A few steps should be enough to get the model to learn the expected response format of the environment.

Check out the logs of the SFT run on W&B.

To train on a single GPU, run

# In the `Trainer` pane
uv run sft @ examples/basic/wordle/sft.toml \
  --run.name sft \
  --monitors.wandb.project ... \
  --monitors.wandb.name ...

To train on multiple GPUs, run

# In the `Trainer` pane
uv run torchrun \
  --local-ranks-filter 0 \
  --nproc-per-node ... \
  src/prime_rl/trainer/sft/train.py @ examples/basic/wordle/sft.toml \
  --monitors.wandb.project ... \
  --monitors.wandb.name ... 

After training completes, you will find the final DCP checkpoint in outputs/sft/checkpoints/step_20.

We have uploaded the final model as PrimeIntellect/Qwen3-1.7B-Wordle-SFT.

RL

Finally, we will do multi-turn RL against the wordle environment using the model PrimeIntellect/Qwen3-1.7B-Wordle-SFT (HF), we obtained from the SFT warmup. We will do sizable RL training, with 100 training steps, each generating training batches of 32x16 rollouts, for a total batch size of 512, at a context length of 4096.

Check out the logs of the RL run on W&B.

# Run this in the `Trainer` pane
uv run rl @ examples/basic/wordle/rl.toml \
  --model.name ... \
  --run.name rl \
  --monitors.wandb.project ... \
  --monitors.wandb.name ... 

This will write a DCP checkpoint in outputs/rl/checkpoints/step_100.

We have uploaded the final model as PrimeIntellect/Qwen3-1.7B-Wordle-RL.

Evals

Let's see how our final RL checkpoint performs on the eval set.

# Run this in the `Inference` pane
uv run inference --vllm.model PrimeIntellect/Qwen3-1.7B-Wordle-RL

# Run this in the `Trainer` pane
uv run eval wordle --harness.id null -m PrimeIntellect/Qwen3-1.7B-Wordle-RL --client.base-url http://localhost:8000/v1 -n 20 -r 3 --sampling.max-tokens 1024 --no-push

Way better! Our model now wins ~60% and gets an average reward of ~1.5.