Mind2Web Evaluation

Preliminary

Data files

To reproduce Mind2Web evaluation with ScribeAgent. You should first download the Mind2Web test set here and unzip it with the password mind2web. You should also download the scores obtained from the tuned DeBERTa-v3-base model used in Mind2Web.

Model directory

After finetuning, run finetuning/prepare_model.py to create the compiled vllm model for the inference script. Next, specify in run_inference.sh the model path using --vllm_model "YOUR MODEL PATH". By default this points to 7b_vllm. If you have any LoRA adapters, please merge them before saving.

Install vllm

Make sure you have vllm installed on top of the Mind2Web required libraries:

pip install vllm

.env

Create a .env file with your Huggingface token

HF_TOKEN="YOUR HUGGINGFACE TOKEN"

If your directory looks like this youre ready to go!

Mind2Web
├── .env
├── 7b_vllm
├── README.md
├── concat_response.py
├── create_dataset.py
├── data
│   ├── test_domain
│   │   └── test_domain_*.json
│   ├── test_task
│   │   └── test_task_*.json
│   └── test_website
│       └── test_website_*.json
├── eval.sh
├── inference.py
├── metrics.py
├── preprocess.py
├── refined_evals
│   └── ***
├── run_eval.py
├── run_inference.sh
└── scores_all_data.pkl

Data preparation

First, we need to preprocess and format the test files in a way that is ingestible by our inference code.

python3 create_dataset.py --task "test_domain" --rope 4

Set rope parameter according to your GPU memory capacity. We use H100s and use rope scaling factor 4 with 64 samples at inference.

Inference

We now run inference using the prepared model

./run_inference.sh

Computing Results

Lastly, we gather the results of our model and compute them.

./eval.sh

If you have any question regarding our implementation, please raise issues! We will try our best to get back asap!