Benchmarking LLaMA-3 with LLMBox

This tutorial demonstrates how to benchmark the LLaMA-3 model using LLMBox.

Meta has published its new LLaMA-3 pre-trained models and instruct models. The official model card includes a benchmark results across 15 datasets under differenct settings.

For all the evaluations, meta uses their internal evaluations library.

Here, we detail how you can reproduce these results using LLMBox in the same setup.

Step 1: Install LLMBox

Begin by installing LLMBox. Execute the following commands:

git clone https://github.com/RUCAIBox/LLMBox.git && cd LLMBox
pip install -r requirements.txt

Alternatively, utilize the faster Python package installer uv to expedite the pip install process.

LLMBox is a comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation. For our purpose, we will use the evaluation component (inference.py and utilization) of LLMBox to conduct our benchmarks.

Step 2: Download the LLaMA-3 Model

Visit the Huggingface model repository (meta-llama/Meta-Llama-3-8B and meta-llama/Meta-Llama-3-8B-Instruct for example) and follow the instructions to download the LLaMA-3 model in Huggingface format.

After accepting the terms, utilize the Huggingface CLI to log into your account (huggingface-cli login). Then, with the same account, you can manually download the model or load directly from Huggingface Hub.

For manual download, you can enable the hf-transfer to accelerate. The original model weights can also be excluded from download:

mkdir meta-llama
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download \
 --resume-download meta-llama/Meta-Llama-3-8B \
 --local-dir meta-llama/Meta-Llama-3-8B \
 --exclude "original/*" \
 --local-dir-use-symlinks False

Step 3: Benchmark LLaMA-3 Base Pre-trained Model with LLMBox

TOC:

• GenerationDataset with few-shot settings: SQuAD v2, QuAC, DROP, and TriviaQA
• Perplexity over the Options: BoolQ, CommonSenseQA, and WinoGrande
• Probability over Choice Characters: MMLU and ARC-Challenge
• AGIEval English
• Chain-of-Thought strategy: BIG-Bench Hard

Then we can benchmark the LLaMA-3 base pretrained model on 11 datasets with different settings.

By default, LLMBox will use all visible GPUs. The CUDA_VISIBLE_DEVICES environment variable or the --cuda flag allows you to specify particular GPUs to utilize.

View the full test scripts at here.

GenerationDataset with few-shot settings

To streamline the process, evaluate multiple datasets with a similar setup concurrently by grouping them by the number of shots:

python inference.py \
  --model meta-llama/Meta-Llama-3-8B \
  --dataset squad_v2 quac \
  --num_shots 1 \
  --max_example_tokens 4096

This command will evaluate the model on the SQuAD v2 and QuAC datasets with 1-shot setting. Since LLaMA-3 has a context lenght of 8192, we can set the maximal length of examples to 4096.

The DROP dataset is a challenging dataset used to evaluate the capabilities of natural language processing models in discrete reasoning and text comprehension: given a passage and a question, the model is required to accurately answer the question based on the information in the passage.

DROP (3-shots) randomly draws 3 examples from example_set = "train" for each instance:

python inference.py \
  --model meta-llama/Meta-Llama-3-8B \
  --dataset drop \
  --num_shots 3

The DROP dataset uses two evaluation metrics:

1. EM (Exact Match): Calculates the proportion of predictions that match the ground truth answer exactly. This metric measures the accuracy of the model, awarding points only when the model's prediction exactly matches the standard answer.

2. F1 Score: Considers both precision and recall. For the model's predicted answer and the ground truth answer, the F1 score calculates the overlap between them, which is particularly useful for answers that contain multiple words. The F1 score is more lenient, allowing partially correct answers to receive some credit.

Meta reports F1 scores for the DROP dataset.

In LLMBox, TriviaQA by default uses the same subset as LLaMA-3, i.e. the WIKI validation set rc.wikipedia.nocontext.

python inference.py \
  --model meta-llama/Meta-Llama-3-8B \
  --dataset triviaqa \
  --num_shots 5

Perplexity over the Options

For WinoGrande, LLaMA-3 uses a choice based setup for evaluation. The missing blank is filled with the two possible choices and then compute log-likelihood over the suffix. LLMBox also implements WinoGrande as a MultipleChoiceDataset.

python inference.py \
  --model meta-llama/Meta-Llama-3-8B \
  --batch_size 128:auto \
  --dataset winogrande \
  --num_shots 5

LLMBox uses ppl_no_option ranking type by default, where one instance is splitted into sub-instances by the choices. In each sub-instance, only the corresponding choice is provided and we calculate the perplexity over it. We choose the option with the lowest perplexity as the answer.

{
  "question":
    "What is the ...?",
  "choices": [
    "The first",
    "The second",
    ...
  ],
  "answer": 3
}

Q: What is ...?
A: The first
   └--ppl--┘

Q: What is ...?
A. The first
B. The second
C. ...
A: A. The first
   └----ppl---┘

Q: What is ...?
A. The first
B. The second
C. ...
A: _
   └→ [A B C D]

Q: What is ...?
A. The first
B. The second
C. ...
A: _ (max_tokens=1)

BoolQ (0-shot) dataset uses the same settings as LLaMA-1 and LLaMA-2:

python inference.py \
  --model meta-llama/Meta-Llama-3-8B \
  --batch_size 128:auto \
  --dataset boolq  # --ranking_type ppl_no_option

CommonSenseQA (7-shots):

python inference.py \
  --model meta-llama/Meta-Llama-3-8B \
  --batch_size 128:auto \
  --dataset commonsenseqa \
  --num_shots 7  # --ranking_type ppl_no_option

Notes: CoT currently not supported in CSQA.

Probability over Choice Characters

For ARC-Challenge, LLaMA-3 uses the MMLU setup for evaluation where they provide all the choices in the prompt and calculate likelihood over choice characters (A, B, C, D, ...).

In LLMBox, you can use the --ranking_type prob option to calculate in the same way. You can see the difference between it and other ranking types in our documentation.

MMLU (5-shot):

python inference.py \
  --model meta-llama/Meta-Llama-3-8B \
  --batch_size 128:auto \
  --dataset mmlu \
  --num_shots 5 \
  --ranking_type prob \
  --max_example_tokens 4096

ARC-Challenge (25-shot):

python inference.py \
  --model meta-llama/Meta-Llama-3-8B \
  --batch_size 32:auto \
  --dataset arc:ARC-Challenge \
  --num_shots 25 \
  --ranking_type prob \
  --max_example_tokens 4096

AGIEval English

Then we can evaluate on the English subcollection (includes 8 English subsets) of AGIEval dataset following the default settings.

Notes that the --num_shots option is used to specify the maximum number of few-shots examples, and for subsets with fewer examples (like some subsets in AGIEval), the actual number of shots will be less than the specified number (3-5 shots in the case of AGIEval).

python inference.py \
  --model meta-llama/Meta-Llama-3-8B \
  --dataset agieval:[English] \
  --num_shots 5 \
  --batch_size 16:auto \
  --max_example_tokens 2560

You might see the following warning message:

2024-04-20 21:54:41 WARNING The example data of agieval:lsat-rc only has 3 instances, but the few-shot number is set to 5. Setting the few-shot number to 3.
2024-04-20 21:54:41 WARNING The example data of agieval:lsat-lr only has 3 instances, but the few-shot number is set to 5. Setting the few-shot number to 3.
...
2024-04-20 21:56:21 INFO Evaluating get_ppl on agieval 8 subsets (model_attr={'type': 'base', 'model_backend': 'huggingface', 'model_max_input': 8191, 'model_max_input_and_output': 8192}, ppl_kwargs=None, num_shots=3-5, len=11447, num_instances=2546, use_cache=False)

Chain-of-Thought strategy

LLaMA-3 uses the Chain-of-Thought strategy in BIG-Bench Hard (BBH) dataset. In LLMBox, you can use the --cot base option to evaluate in the same way.

python inference.py \
  --model meta-llama/Meta-Llama-3-8B \
  --dataset bbh \
  --cot base \
  --num_shots 3 \
  --vllm

Then we report the Exact Match (EM) scores for it.

Step 4: Benchmark LLaMA-3 Instruct Model with LLMBox

TOC:

• LLaMA-3 Instruct model: MMLU and GPQA
• Pass@K Metrics: HumanEval
• Self-consistency: MATH and GSM8K

The LLaMA-3 Instruct model is evaluated on 5 datasets with different settings.

LLaMA-3 Instruct model

In LLMBox, you can use --model_type to specify the model type. For an instruction tuned model, you can use --model_type chat. If model_type is not provided, LLMBox will automatically detect from the model name.

LLMBox also loads the chat_template automatically. You can specify the --chat_template option to use a custom chat template.

MMLU (5-shots):

python inference.py \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --batch_size 128:auto \
  --dataset mmlu \
  --max_example_tokens 4096 \
  --ranking_type prob \
  --num_shots 5

For GPQA (0-shot), we use the default settings:

python inference.py \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset gpqa

You can also see gpqa for additional settings.

Pass@K Metrics

For HumanEval dataset, LLaMA-3 uses the Pass@1 metrics for evaluation.

python inference.py \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset humaneval \
  --pass_at_k 1 \
  --sample_num 20 \
  --temperature 0.1 \
  --max_example_tokens 2560

The core idea of Pass@K is the probability that among the $k$ code outputs generated by a computational model for a single problem input, at least one code can pass the verification. However, directly calculating this metric requires repeatedly testing a single problem and generating $k$ codes in each test, leading to high computational costs. To reduce the computational complexity of the evaluation, an unbiased estimation approach is commonly used to approximate the value of Pass@K. The estimation formula is as follows:

$\text{Pass@$k$} = \mathbb{E}\left(1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\right)$

where $n$ represents the number of codes synthesized for each test problem, and $c$ represents the number of codes that pass the test. According to the law of large numbers, as the total sample size n increases, the accuracy of this estimation also improves.

We evaluate the pass@1 metric by setting the --sample_num parameter to synthesize $20 solutions for each test problem. The temperature should be set to greater than \0$ to ensure that different answers can be generated for the same problem.

Self-consistency

Both MATH and GSM8K use maj@1 metric (majority at 1) for evaluation, which is equivalent to the exact match (EM) scores with self-consistency --majority 1 option.

For GSM8K (8-shots, CoT), we use PAL CoT strategy:

python inference.py \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset gsm8k \
  --num_shots 8 \
  --max_example_tokens 2560 \
  --cot pal

MATH (4-shots, CoT):

python inference.py \
  --model meta-llama/Meta-Llama-3-8B-Instruct \
  --dataset math \
  --num_shots 4 \
  --max_example_tokens 2560 \
  --cot base

You can further the CoT strategy by adjusting the instruction template, which in MATH datasets is --instruction "Solve the following math problem.\n\nQ: {problem}\nA:".

Step 5: Report the Results

We are done with the evaluation now. You can collect the results and report them. Check the logs directory for the logs of each evaluation and evaluation_results directory for model raw inputs and outputs for each evaluation instance.

View the full test scripts at here.

LLaMA-3 (8B) Pre-trained Model

LLaMA-3 (8B) Instruct Model