Large-Language-Model-for-Multilingual-Vulnerability-Detection

Various deep learning-based approaches utilizing pre-trained language models (PLMs) have been proposed for automated vulnerability detection. With recent advancements in large language models (LLMs), several studies have begun exploring their application to vulnerability detection tasks. However, existing studies primarily focus on specific programming languages (e.g., C/C++) and function-level detection, leaving the strengths and weaknesses of PLMs and LLMs in multilingual and multi-granularity scenarios largely unexplored. In this study, we empirically investigate the effectiveness of PLMs and LLMs on detecting multilingual vulnerability at both function-level and line-level.

This paper is under-review right now. Here is the pre-print version: https://arxiv.org/pdf/2506.07503

Envirnoment Requirement

• Python 3.7.7
• PyTorch 1.13.1+cu117
• Sciki-learn 1.2.2
• Transformers 4.37.0.dev0
• TRL 0.7.11
• Numpy 1.18.1
• Pandas 1.3.0
• Matplotlib 3.4.2
• Openai 1.2.3
• Tiktoken 0.7.0
• Tqdm 4.66.5

Dataset

The dataset for function-level and line-level detection can be found as this link: https://drive.google.com/drive/folders/1lEhcs_IzW1fukldybtDZYey4WtMJNZxA?usp=sharing
How to use the dataset: Please download it and save the CSV files on the Data folder.

Experiment Replication

For running all language models (both PLMs and LLMs), we recommend using GPU with 48 GB up memory for training and testing since many of them are computing intensive.

For runing the close-source LLMs, please prepare your own api_key for executing the code.

Demo

Here is the Demo for running PLMs and LLMs.

For using PLMs

let's take the pre-trained UniXCoder on detecting line-level vulnerability as an example. Please ensure you had download and save the CSV data file on the Data folder.

(1) Training phase

You can train the model through the following commands:

cd ./Function-level/PLMs/
./train_unixcoder.sh

(2) Inference phase

You can test the model through the following commands:

cd ./Function-level/PLMs/
./test_unixcoder.sh

For using LLMs

Regarding to close-source LLMs

• Function-level detection
  Regarding apply zero-shot prompting/few-shot prompting/instruction tuning, please use the scripts function-level detection.
• Line-level detection
  Regarding apply zero-shot prompting/few-shot prompting/instruction tuning, please use the scripts line-level detection.

Regarding to open-source LLMs

The script use Code Llama (7B) as an example. You can change the base model according your need. Regarding the dataset for instruction tuning, please re-format the CSV data to the Jsonl format according to Huggingface turtorial.

• Function-level detection
  Regarding apply zero-shot prompting/few-shot prompting/instruction tuning, please use the scripts function-level detection.
• Line-level detection
  Regarding apply zero-shot prompting/few-shot prompting/instruction tuning, please use the scripts line-level detection.

Results

1) The performance of PLMs and LLMs on function-level detection

---

2) The performance of PLMs and LLMs on line-level detection

---

3) The unique correct/incorrect detection

---

4) The effectiveness of vulnerability detection on Top-25 CWE

---

5) The effectiveness of vulnerability detection on CVSS severity

---