Experiments

• Create synthetic dataset
• Create benchmarking dataset for evaluation
• Manual data vs synthetic data
• Model size comparison
• Data pipeline ablation

Setup for rollout trajectories using Moatless

For experiments require rollout trajectories using Moatless, you need to install Moatless as follows:

bash swesynth/lib/moatless/install-moatless-fork.sh

then run conda activate swesynth-moatless before generating rollout trajectories.

Setup for training models

For experiments require training models, you need to install LLaMA Factory as follows:

bash swesynth/lib/llama_factory/install-llama-factory.sh

then run conda activate swesynth-llama-factory before training models using llamafactory-cli.