🎭 HER: Hierarchical Emotion Reasoning

HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playing

HER introduces dual-layer thinking that distinguishes characters' first-person thinking from LLMs' third-person thinking for cognitive-level persona simulation.

📖 Overview

LLM role-playing—using LLMs to simulate specific personas—has emerged as a key capability in companionship, content creation, and digital games. While current models effectively capture character tones and knowledge, simulating the inner thoughts behind their behaviors remains a challenge.

HER (Hierarchical Emotion Reasoning) is a unified framework for cognitive-level persona simulation that introduces:

• 🧠 Dual-Layer Thinking: Distinguishes characters' first-person thinking from LLMs' third-person thinking
• 📚 Reasoning-Augmented Data: High-quality role-playing data via reverse engineering
• 🎯 Human-Aligned Rewards: Principle-based reward models aligned with human preferences

🏆 Results

HER achieves +30.26% improvement on CoSER and +14.97% on MiniMax Role-Play Bench.

🚀 Quick Start

Installation

git clone https://github.com/xxx/HER.git
cd HER
pip install -r requirements.txt

Try the Demo

Chat with AI characters from classic literature:

cd chat_demo

# Interactive chat with 200 classic book scenarios
python chat_demo.py

# Show the model's thinking process
python chat_demo.py --show-think --show-rolethink

📖 Pride and Prejudice
👥 AI plays: Elizabeth Bennet | You play: Mr. Darcy

════════════════════════════════════════════════════════════════════════════════
🎭 【Elizabeth Bennet's Response】
════════════════════════════════════════════════════════════════════════════════
[His tone is light, but the air feels heavy. I cannot let him see how much
Lady Catherine's intrusion still stings.]
(takes a steadying breath, smoothing the folds of her dress)
I believe I can manage, Father. Though I must admit, I am curious about
what this letter contains.
════════════════════════════════════════════════════════════════════════════════

📁 Project Structure

HER/
├── 📂 chat_demo/              # 🎮 Interactive demo (try this first!)
│   └── chat_demo.py           # Chat with book characters
│
├── 📂 data_process_code/      # 🔧 Data synthesis pipeline
│   ├── step1_data_process/    # Data cleaning & format conversion
│   ├── step2_gen_rolethinking/# Role thinking enhancement
│   ├── step3_gen_systhinking/ # System thinking generation
│   └── step4_setting_completion/ # Character profile enrichment
│
├── 📂 training_code/          # 🎯 Model training pipeline
│   ├── step1_roleplay_sft/    # Supervised fine-tuning
│   ├── step2_reward_sft/      # Reward model data
│   ├── step3_reward_rl/       # RM training
│   └── step4_roleplay_rl/     # RL training
│
└── 📂 eval_code/              # 📊 Evaluation framework
    ├── benchmarks/coser/      # CoSER multi-turn benchmark
    ├── models/                # Model adapters (vLLM, API)
    └── configs/               # Evaluation configs

🧠 Dual-Layer Thinking Architecture

HER introduces a hierarchical thinking mechanism:

┌─────────────────────────────────────────────────────────────────┐
│                    HER Response Structure                        │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ 🔍 System Thinking (Third-Person Analysis)              │   │
│  │ "Elizabeth should respond with wit but maintain dignity. │   │
│  │  Consider her pride and her evolving feelings..."       │   │
│  └─────────────────────────────────────────────────────────┘   │
│                              ↓                                   │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ 💭 Role Thinking (First-Person Inner Thoughts)          │   │
│  │ [His manner is so different now. Can I trust this       │   │
│  │  change, or is it merely another game?]                 │   │
│  └─────────────────────────────────────────────────────────┘   │
│                              ↓                                   │
│  ┌─────────────────────────────────────────────────────────┐   │
│  │ 🎭 Role Response (Speech + Action)                      │   │
│  │ (lifts her chin slightly) "Mr. Darcy, I find your       │   │
│  │ sudden interest in conversation rather remarkable."     │   │
│  └─────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

📊 Training Pipeline

Raw Dialogues (760 books)
       │
       ▼
┌──────────────────┐
│ Data Processing  │──► Enhanced with dual-layer thinking
└──────────────────┘
       │
       ▼
┌──────────────────┐
│ Roleplay SFT     │──► Base roleplay capability
└──────────────────┘
       │
       ▼
┌──────────────────┐
│ Reward Model     │──► Principle-based quality assessment
└──────────────────┘
       │
       ▼
┌──────────────────┐
│ Roleplay RL      │──► Aligned with human preferences
└──────────────────┘
       │
       ▼
   Final Model (HER-RL)

Run Training

# Step 1: Data Processing
cd data_process_code
# See DATA_PIPELINE.md for detailed workflow

# Step 2: SFT Training
cd training_code/step1_roleplay_sft
python convert_to_sft.py

# Step 3-4: Reward Model & RL
# See training_code/PIPELINE.md

📈 Evaluation

CoSER Benchmark

Multi-agent group conversation evaluation with 4 dimensions:

cd eval_code

# Run evaluation
python run_coser.py \
    --actor your-model \
    --judge qwen \
    --max-rounds 20

📚 Documentation

Contact

For questions or feedback, please open an issue in the repository.

🎓 Citation

@article{her2025,
  title={HER: Human-like Reasoning and Reinforcement Learning for LLM Role-playing},
  author={Chengyu Du, Xintao Wang, Aili Chen, Weiyuan Li, Rui Xu, Junteng Liu, Zishan Huang, Rong Tian, Zijun Sun, Yuhao Li, Liheng Feng, Deming Ding, Pengyu Zhao, Yanghua Xiao},
  journal={arXiv preprint arXiv:2026.21459},
  year={2026}
}

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Acknowledgments

• CoSER for the evaluation benchmark
• MiniMax for the evaluation benchmark