Awesome LLM-generated Text Detection
December 30, 2024 · View on GitHub
The powerful ability of large language models (LLMs) to understand, follow, and generate complex languages has enabled LLM-generated texts to flood many areas of our daily lives at an incredible rate, with potentially negative impacts and risks on society and academia. As LLMs continue to expand, how can we detect LLM-generated texts to help minimize the threat posed by the misuse of LLMs?
¹ Junchao Wu, ¹ Shu Yang, ¹ Runzhe Zhan, ¹ ² Yulin Yuan, ¹ Derek Fai Wong, ¹ Lidia Sam Chao
¹ University of Macau, ² Peking University
📢 News
- [2024.11.28] ✨ Our survey paper is accepted by Computational Linguistics Journal. This survey offers a comprehensive overview of the latest advancements in LLM-generated text detection, highlighting the urgent need for more robust methods. It reviews mainstream approaches, addresses key challenges, and outlines promising future research directions. The paper serves as both a clear introduction for newcomers and a valuable resource for experts seeking updates in the field. Please refer to arXiv: A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions and Github Repo LLM-generated-Text-Detection for details.
- [2024.09.26] ✨ Our benchmark paper is accepted by NeurIPS 2024 D&B track. We released DetectRL, a benchmark for real-world LLM-generated text detection, provide real utility to researchers on the topic and practitioners looking for consistent evaluation methods. Please refer to arXiv: DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios and Github Repo DetectRL for details.
- [2023.10.24] Our survey paper is now available on arXiv: A Survey on LLM-generated Text Detection: Necessity, Methods, and Future Directions.
- [2023.05.01] : We began to explore the topic of LLM-generated Text Detection.
🔍 Table of Contents
📃 Papers
Overview
A survey and reflection on the latest research breakthroughs in LLM-generated Text detection, including data, detectors, metrics, current issues and future directions. Please refer to our article/paper for more details.
Datasets
Benchmarks
| Benchmarks / Datasets | Use | Human | LLMs |
|---|---|---|---|
| HC3 | train | 58k | 26k |
| HC3-Chinese | train | 22k | 17k |
| CHEAT | train | 15k | 35k |
| GROVER Dataset | train valid test | 5k 2k 8k | 5k 1k 4k |
| TweepFake | train | 12k | 12k |
| GPT-2 Output Dataset | train | 250k | 250k |
| TuringBench | train | 10k | 190k |
| MGTBench | train test | 2k 563 | 13k 3k |
| ArguGPT | train valid test | 3k 350 350 | 3k 350 350 |
| DeepfakeText-Dataset | train valid test | 95k 29k 29k | 236k, 29k 28k |
| M4 | train valid test | 122k 500 500 | 122k 500 500 |
| GPABenchmark | train | 600k | 600k |
| Scientific-articles Benchmark | train test | 8k 4k | 8k 4k |
Potential Datasets
| Tasks | Datasets |
|---|---|
| Questions Answering | PubMedQA, Children book corpus (CBT), ELI5, TruthfulQA, NarrativeQA |
| Scientific writing | Peer Read, arXiv, TOEFL11 |
| Story generation | WritingPrompts |
| News Article writing | XSum |
| Web Text | Wiki40b, WebText, Avax tweets dataset, Climate Change Tweets Ids |
| Opinion statements | r/ChangeMyView (CMV) Reddit subcommunity, Yelp , IMDB Dataset |
| Comprehension and Reasoning | SciGen, ROCStories Corpora, HellaSwag, SQuAD |
Detectors
Watermark Technology
| Paper | Link |
|---|---|
| A watermark for large language models. |   |
| On the Reliability of Watermarks for Large Language Models | |
| A Private Watermark for Large Language Models | |
| Distillation-Resistant Watermarking for Model Protection in NLP | |
| Watermarking Pre-trained Language Models with Backdooring | |
Zero-shot Methods
| Paper | Link |
|---|---|
| DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature | |
| Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature | |
| Efficient Detection of LLM-generated Texts with a Bayesian Surrogate Model | |
| DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text | |
| GLTR: Statistical Detection and Visualization of Generated Text | |
| HowkGPT: Investigating the Detection of ChatGPT-generated University Student Homework through Context-Aware Perplexity Analysis | |
| Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts | |
Fine-tuning LMs Methods
| Paper | Link |
|---|---|
| How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection | |
| Multiscale Positive-Unlabeled Detection of AI-Generated Texts | |
| Real or fake? Learning to discriminate machine from human generated text | |
| Automatic Detection of Generated Text is Easiest when Humans are Fooled | |
| Stylometric Detection of AI-Generated Text in Twitter Timelines | |
| TweepFake: about Detecting Deepfake Tweets | |
| Towards a Robust Detection of Language Model Generated Text: Is ChatGPT that Easy to Detect? | |
| Deepfake Text Detection in the Wild | |
| ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models | |
| Check Me If You Can: Detecting ChatGPT-Generated Academic Writing using CheckGPT | |
| GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content | |
| Neural Deepfake Detection with Factual Structure of Text | |
| ConDA: Contrastive Domain Adaptation for AI-generated Text Detection | |
Adversarial Learning Methods
| Paper | Link |
|---|---|
| RADAR: Robust AI-Text Detection via Adversarial Learning | |
| OUTFOX: LLM-generated Essay Detection through In-context Learning with Adversarially Generated Examples | |
| Red Teaming Language Model Detectors with Language Models | |
| Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect ChatGPT-Generated Text | |
LLMs as Detector
| Paper | Link |
|---|---|
| Fighting fire with fire: Can chatgpt detect ai-generated text? | |
| OUTFOX: LLM-generated Essay Detection through In-context Learning with Adversarially Generated Examples | |
| GPT Paternity Test: GPT Generated Text Detection with GPT Genetic Inheritance | |
Related Works
Other Surveys
| Paper | Link |
|---|---|
| Automatic Detection of Machine Generated Text: A Critical Survey | |
| The Science of Detecting LLM-Generated Texts | |
| Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods | |
| Computer-Generated Text Detection Using Machine Learning: A Systematic Review | |
| Attribution and Obfuscation of Neural Text Authorship: A Data Mining Perspective | |
🚩 Citation
If our research helps you, please kindly cite our paper.
@article{wu2023survey,
title={A Survey on LLM-gernerated Text Detection: Necessity, Methods, and Future Directions},
author={Junchao Wu and Shu Yang and Runzhe Zhan and Yulin Yuan and Derek F. Wong and Lidia S. Chao},
journal = {CoRR},
volume = {abs/2310.14724},
year = {2023},
url = {https://arxiv.org/abs/2310.14724},
eprinttype = {arXiv},
eprint = {2310.14724},
Contributing
Contributions are welcome! If you have any ideas, suggestions, or bug reports, please open an issue or submit a pull request. We appreciate your contributions to making LLM-generated Text Detection work even better.