FormulaEval: Benchmarking Scientific Formula Vocalization in Large Speech Language Models Toward Accessible Learning

March 29, 2026 · View on GitHub

🤗 Datasets | 🖥️ Quick Start | ©️ Citation

This repository provides the official implementation of FormulaEval, a benchmark for evaluating the ability of Large Speech Language Models (LSLMs) to vocalize scientific formulas — spanning physics, chemistry, and mathematics — toward accessible learning.

Highlights

Three curated datasets covering Physics, Chemistry, and Mathematics formula vocalization.
Comprehensive evaluation framework supporting multiple state-of-the-art LSLMs out of the box.
Flexible evaluation pipeline with support for both local models and API-based judge models.
Reproducible benchmarking with configurable metrics including WER and instruction-following evaluation.

Datasets

All datasets are publicly available on HuggingFace 🤗:

Dataset	Description	Link
Physics700	Physics formulas and equations	🤗 Physics700
ChemEquation	Chemical equations and formulas	🤗 ChemEquation
MixMath	Mixed mathematical expressions	🤗 MixMath

Project Structure

FormulaEval/
├── README.md                           # This file
├── requirements.txt                    # Python dependencies
├── config.yaml                         # Configuration file
├── run_audio.py                        # Main evaluation script
├── run_audio_online.sh                 # Online evaluation shell script
├── almeval/                            # Core evaluation framework
│   ├── datasets/                       # Dataset processing modules
│   │   ├── ds_tts.py                   # TTS dataset handler
│   │   └── utils.py                    # Utility functions
│   ├── models/                         # Model implementations
│   │   ├── baichuan_audio.py           # Baichuan Audio models
│   │   ├── glm4_voice.py               # GLM-4 Voice model
│   │   ├── kimi_audio.py               # Kimi Audio model
│   │   ├── qwen_omni.py                # Qwen-Omni model
│   │   └── ...                         # Other model implementations
│   ├── judge_models/                   # Judge models for evaluation
│   │   └── api/                        # API-based judge models
│   ├── metrics/                        # Evaluation metrics
│   │   ├── wer.py                      # Word Error Rate
│   │   ├── cn_tn.py                    # Chinese text normalization
│   │   └── ifeval.py                   # Instruction following evaluation
│   └── utils/                          # Utility modules
│       ├── config_manager.py           # Configuration management
│       └── misc.py                     # Miscellaneous utilities
├── data/                               # Dataset storage
│   └── downloaded_datasets/            # Local dataset cache
├── examples/                           # Example scripts
│   └── run_baichuan_eval.sh            # Standalone evaluation script
└── results/                            # Evaluation results output

Quick Start

1. Environment Setup

# Clone the repository
git clone https://github.com/<your-username>/FormulaEval.git
cd FormulaEval

# Create conda environment
conda create -n formula-eval python=3.11
conda activate formula-eval

# Install dependencies
pip install -r requirements.txt

2. Download Datasets

Datasets are hosted on HuggingFace and can be downloaded automatically or manually:

from datasets import load_dataset

# Load Physics700
physics = load_dataset("Stephen-Lee/FormulaEval_datasets", "Physics700")

# Load ChemEquation
chem = load_dataset("Stephen-Lee/FormulaEval_datasets", "ChemEquation")

# Load MixMath
mixmath = load_dataset("Stephen-Lee/FormulaEval_datasets", "MixMath")

3. Configuration

Important: Before running any evaluation, configure config.yaml with your model paths and API endpoints.

cp config.yaml config_local.yaml

Edit config_local.yaml:

MODEL_PATHS:
  # ASR Models
  asr_zh_model_path: "/path/to/your/paraformer-zh"

  # Audio Language Models
  baichuan_audio_base_path: "/path/to/your/Baichuan-Audio-Base"
  baichuan_audio_instruct_path: "/path/to/your/Baichuan-Audio-Instruct"

API_ENDPOINTS:
  asr_en_service_url: "http://your-whisper-service-url/v1"
  qwen3_api_url: "http://your-qwen3-url/v1/chat/completions"
  openai_api_url: "https://api.openai.com/v1/chat/completions"

API_KEYS:
  openai_api_key: "your-openai-api-key"

# Point to your local config
export PROJECT_CONFIG_PATH="/path/to/your/config_local.yaml"

4. Run Evaluation

Option A: Standalone Script (Recommended)

bash examples/run_baichuan_eval.sh \
    --model "Baichuan-Audio-Chat" \
    --data "Physics700" \
    --work-dir "./results/my_evaluation"

Option B: Python Script

python run_audio.py \
    --model "Baichuan-Audio-Chat" \
    --data "Physics700" "ChemEquation" "MixMath" \
    --work-dir "./results/baichuan_eval" \
    --eval-method "default"

Debug Mode (Quick Test with 10 Samples)

python run_audio.py \
    --model "Baichuan-Audio-Chat" \
    --data "Physics700" \
    --debug \
    --work-dir "./results/debug_test"

Command Line Reference

`run_audio.py`

python run_audio.py [OPTIONS]

Required:
  --data DATASET [DATASET ...]   Dataset name(s): Physics700, ChemEquation, MixMath
  --model MODEL                  Model name to evaluate

Optional:
  --work-dir DIR                 Output directory (default: ./eval_result)
  --rank INT                     Current GPU rank (default: 0)
  --world-size INT               Total number of GPUs (default: 1)
  --eval-method METHOD           Evaluation method (default: default)
  --model-path PATH              Custom model path override
  --force-reinfer                Force re-run inference
  --reeval                       Re-evaluate existing predictions only
  --skip-eval                    Skip evaluation, inference only
  --debug                        Debug mode (10 samples per dataset)

Output Structure

work_dir/
└── ModelName/
    ├── DatasetName/
    │   ├── ModelName_DatasetName.jsonl                         # Raw predictions
    │   ├── ModelName_DatasetName_default_performance.json      # Metrics summary
    │   └── logs/
    │       └── rank0.log                                       # Execution log
    └── output_audio/                                           # Generated audio (TTS)

Supported Models

Model	Type	Notes
Baichuan-Audio-Base	Open-source	Requires local download
Baichuan-Audio-Chat	Open-source	Requires local download
GLM-4-Voice	Open-source	Requires local download
Kimi Audio	API	Requires API key
Qwen-Omni	Open-source	Requires local download

Troubleshooting

Config file not found

Error: config.yaml not found in project root!

Ensure config.yaml exists and PROJECT_CONFIG_PATH is set correctly.

Model path not accessible

Error: Model path not accessible

Check model paths in config.yaml and ensure models are downloaded.

CUDA out of memory Reduce batch size or use a smaller model. Some models (e.g., StepAudio) require 4×80G GPUs.

NLTK data missing The system will automatically download required NLTK data on first run.

Citation

If you find FormulaEval useful in your research, please cite our paper:

@inproceedings{li2026benchmarking,
  title={Benchmarking Scientific Formula Vocalization in Large Speech Language Models Toward Accessible Learning},
  author={Li, Xueyi and Liu, Tianqiao and Zheng, Jiaqi and Liu, Zitao and Wu, Yongdong and Luo, Weiqi},
  booktitle={Proceedings of the 27th International Conference on Artificial Intelligence in Education},
  month={June},
  year={2026},
  address={Seoul, Republic of Korea}
}

Acknowledgements

This codebase is built with reference to the following excellent open-source projects. We sincerely thank the authors for their contributions:

License

This project is licensed under the Apache-2.0 License. See the LICENSE file for details.