Data

Directory tree

This folder contains two folders lfqa and vicuna80 for LFQA and Vicuna80 respectively. The directory tree for both folders are listed below.

.
|-- lfqa
|   |-- experts_pairwise_human_preferences.jsonl
|-- vicuna80
    |-- vicuna_benchmark_human_annotations.csv
    |-- clean_battle_20230522.json
    |-- generations
    |   |-- answer_bard.jsonl
    |   |-- answer_claude.jsonl
    |   |-- answer_gpt35.jsonl
    |   |-- answer_gpt4.jsonl
    |   |-- answer_vicuna-13b.jsonl
    |   |-- answer_zephyr.jsonl
    |-- prompt
    |   |-- reviewer.jsonl
    |   |-- vicuna_prompt_threeclass.jsonl
    |   |-- vicuna_questions.jsonl
    |-- ratings-{reviewer}
    |   |-- answer_{model1}-vs-answer_{model2}-{reviewer}-reviewer_threeclass.jsonl

LFQA

All data of LFQA is stored in the experts_pairwise_human_preferences.jsonl file.

Vicuna80

We pulish the Vicuna80 dataset here. For more details, please refer to our paper. Explanations for files in the Vicuna80 folder are as follows:

1. vicuna_benchmark_human_annotations.csv
   • This file stores human preference for all human annotations on all questions.
2. clean_battle_20230522.json
   • This file is for Chatbot Arena for comparison.
3. generations
   • This folder stores generated answers of all models (Bard, Claude, GPT-3.5, GPT-4, and Vicuna-13b).
4. prompt
   • This folder stores reviewers' information, questions, and prompts to generate reviews.
5. ratings-{reviewer}
   • {reviewer} includes Bard, Claude, GPT-3.5, GPT-4, human, and Vicuna-13b.
   • Both {model1} and {model2} refer to models in the generations folder.