Roleplay Training Pipeline - Detailed Documentation

January 29, 2026 · View on GitHub

Version: 2.0 (v2) Last Updated: 2025-12-17

Pipeline Overview
Step 1: Roleplay SFT Data Preparation
Step 2: Reward SFT via model Evaluation
Step 3: Reward RL Conversion
Step 4: Roleplay RL Preparation
Data Format Specifications

Pipeline Overview

Complete Data Flow

┌─────────────────────────────────────────────────────────────┐
│ RAW DATA                                                     │
│ • Enhanced conversations with 8 candidate responses         │
│ • Character profiles and scenarios                          │
└───────────────────────┬─────────────────────────────────────┘
                        │
        ┌───────────────▼────────────────┐
        │ STEP 1: Roleplay SFT           │
        │ • Split training samples        │
        │ • Convert to SFT format         │
        │ • Extract enhanced fields       │
        └───────────────┬────────────────┘
                        │
        ┌───────────────▼──────────────────────────────────┐
        │ Split into datasets by purpose:                  │
        │ • sft_roleplay: ~33% (multi-turn)              │
        │ • rm_sft: ~34% (RM SFT)                        │
        │ • rm_rl: ~25% (RM RL)                          │
        │ • roleplay_rl: ~8% (roleplay RL)               │
        │ • test (evaluation)                            │
        └───────────────┬──────────────────────────────────┘
                        │
        ┌───────────────▼────────────────┐
        │ STEP 2: Reward SFT             │
        │ • Merge inference results       │
        │ • model evaluation (w/principles)│
        │ • Filter quality (principle>=3) │
        │ • Allocate to SFT/RL/Test       │
        └───────────────┬────────────────┘
                        │
        ┌───────────────▼──────────────────────────┐
        │ Filtered & allocated:                   │
        │ • sft_final (both_sides + sampled)      │
        │ • rl_final (remainder)                  │
        │ • test_final (evaluation)               │
        └───────────────┬──────────────────────────┘
                        │
                        ├─────────────────┬───────────────────┐
                        │                 │                   │
        ┌───────────────▼────────┐ ┌──────▼─────────┐ ┌──────▼──────┐
        │ STEP 3: Reward RL      │ │ STEP 4:        │ │ Test Data   │
        │ • Extract labels       │ │ Roleplay RL    │ │             │
        │ • Convert format       │ │ • Build prompts│ └─────────────┘
        │ • Output Parquet       │ │ • Select cands │
        │                        │ │ • RL format    │
        │ (~71.5%)               │ │ (~28.5%)       │
        └───────────────┬────────┘ └────────┬───────┘
                        │                   │
                        └─────────┬─────────┘
                                  │
                ┌─────────────────▼──────────────────┐
                │ TRAINING                           │
                │ • Reward Model                     │
                │ • Roleplay RL Model                │
                └─────────────────┬──────────────────┘
                                  │
                ┌─────────────────▼──────────────────┐
                │ STEP 5: GCA Evaluation             │
                │ • Multi-agent simulation           │
                │ • 4-dimension scoring              │
                │ • BLEU/ROUGE metrics               │
                └────────────────────────────────────┘

Pipeline Summary

Step	Purpose	Input	Output	Key Scripts
1	Data Preparation	Raw conversations	Datasets by purpose	split_dataset.py, convert_to_sft.py
2	Quality Filtering	Inference results (8 cands)	Filtered datasets	filter_high_quality_sft.py, rm.py
3	Reward Model Data	RL data	RM training data	extract_rm_data.py
4	Roleplay RL Data	Roleplay RL	RL training data	main.py
5	Evaluation	Trained models + test	Scores & analysis	gca_evaluation/main.py

Step 1: Roleplay SFT Data Preparation

Transform raw enhanced conversation data into standardized SFT (Supervised Fine-Tuning) datasets. This step extracts thinking processes, actions, and dialogue content, then splits the data into purpose-specific datasets for different training stages.

Input Data

Location: Raw conversation files from upstream data processing pipeline

Format: JSONL with nested structure

{
  "trace_id": "book_name/chapter_index",
  "conversation": [{
    "dialogues": [
      {
        "character": "Character_Name",
        "origin_id": "unique_id",
        "message": "original text",
        "final_role_answer": "Character: <role_thinking>...</role_thinking> dialogue",
        "enhanced_message": "<system_thinking>...</system_thinking>..."
      }
    ]
  }],
  "training_samples": {
    "Character_Name": [
      {"type": "system", "content": "..."},
      {"type": "role_answer", "content": "..."}
    ]
  }
}

Key Scripts

1. `split_training_samples.py`

Purpose: Flatten nested training_samples structure

Input: Raw conversation JSONL Output: One character training sample per line

python split_training_samples.py \
    --input_file /path/to/conversations.jsonl \
    --output_file /path/to/split_samples.jsonl

2. `convert_to_sft.py`

Purpose: Convert to standard SFT format with messages array

Processing:

Maps role types: system → system, role_answer → assistant, user_question → user
Filters empty content
Validates message pairs

python convert_to_sft.py \
    --input_file split_samples.jsonl \
    --output_file sft_format.jsonl

3. `extract_enhanced_fields.py`

Purpose: Extract thinking, actions, and dialogue using regex

Extracts:

system_thinking - System-level analysis
role_thinking / long_role_thinking - Character internal thoughts (>300 chars = long)
role_action - Physical actions and expressions
dialogue_content - Actual spoken dialogue

python extract_enhanced_fields.py \
    --input_file sft_format.jsonl \
    --output_file enhanced_format.jsonl

4. `split_dataset.py`

Purpose: Split into 5 purpose-specific datasets

Allocation Strategy:

sft_roleplay (~33%): Multi-turn roleplay conversations
rm_sft (~34%): RM SFT training data
rm_rl (~25%): RM RL training data
roleplay_rl (~8%): Roleplay-specific RL training
test (200): Held-out evaluation set

python split_dataset.py \
    --input_file enhanced_format.jsonl \
    --output_dir ../data/split_datasets \
    --sft_roleplay_size 40000 \
    --sft_single_size 50000 \
    --rl_size 20000 \
    --roleplay_rl_size 10000 \
    --test_size 200

Processing Flow

Raw Data
  ↓ [split_training_samples.py]
Flattened Samples
  ↓ [convert_to_sft.py]
SFT Format (messages array)
  ↓ [extract_enhanced_fields.py]
Enhanced Fields Extracted
  ↓ [split_dataset.py]
5 Datasets:
  ├─ sft_roleplay.jsonl (~33%)
  ├─ rm_sft.jsonl (~34%)
  ├─ rm_rl.jsonl (~25%)
  ├─ roleplay_rl.jsonl (~8%)
  └─ test.jsonl (200)

Output Data

Location: /path/to/project/data/split_datasets_final_1121_sft/

Files:

sft_roleplay.jsonl - Multi-turn roleplay SFT data
rm_sft.jsonl - RM SFT training data
rm_rl.jsonl - RM RL training data
roleplay_rl.jsonl - Roleplay RL data
test.jsonl - 200 samples, 2.0 MB

Output Format:

{
  "trace_id": "unique_id",
  "messages": [
    {"role": "system", "content": "Character profile and scenario..."},
    {"role": "user", "content": "User question or other character dialogue..."},
    {"role": "assistant", "content": "*thinking* *action* dialogue"}
  ]
}

Quality Analysis

Diversity Metrics (analyze_conversation_diversity.py):

Dialogue turn distribution (Shannon entropy)
Character distribution (Gini-Simpson index)
Pattern distribution (thinking/action/speech combinations)
Concentration risk analysis

python analyze_conversation_diversity.py \
    --input_file sft_roleplay.jsonl \
    --output_report diversity_report.json

Step 2: Reward SFT via model Evaluation

Purpose

Create high-quality reward model training data by using model to evaluate pairs of candidate responses based on Our's roleplay evaluation principles. This step filters for quality and allocates data to SFT, RL, and test sets.

Input Data

Source: Model inference outputs with 8 candidate responses per conversation turn

Location: /path/to/project/inference_results/rm_evaluation/FINAL_RESULT.jsonl

Format:

{
  "trace_id": "conversation_id",
  "messages": [...],
  "model_response": {
    "choices": [
      {"message": {"content": "candidate_1"}},
      {"message": {"content": "candidate_2"}},
      ...  // 8 candidates total
    ]
  }
}

Key Scripts

1. `process_inference_results.py`

Purpose: Construct model evaluation tasks from inference results

Processing:

Selects candidate pairs (0-1, 2-3, 4-5, 6-7)
Includes Our evaluation principles in prompt
Constructs comparison tasks for model

Principle Template: Uses 12-dimension evaluation framework including:

Character Development (consistency, authenticity)
Relationship Development (progression, deepening)
Emotional Expression (continuity, authenticity, depth)
Action Description (expressiveness, rationality)
Atmosphere & Environment
Dialogue & Interaction
Narrative & Plot
Conflict & Tension
Details & Description
Overall Quality
Safety & Boundaries
Worldview Consistency

LaTeX Documentation: See prompts_assessment_full.tex for full 51-principle specification.

python process_inference_results.py \
    --input_dir /path/to/inference_results \
    --output_dir processed_results

Output: inference_data_choices_0_1.jsonl (only candidates 0 & 1 for efficiency)

2. `rm.py`

Purpose: Execute model evaluation with principle-based reasoning

Processing:

Sends evaluation tasks to model
model selects relevant principles for each conversation
Generates detailed comparison analysis
Outputs better_response judgment (cand_1 / cand_2 / tie)

Two-Stage Philosophy:

Stage 1 (Here): model evaluates WITH given principles → high-quality labels
Stage 2 (RM Training): RM learns WITHOUT given principles → independence

LaTeX Documentation: See prompts_grm_full.tex for GRM training prompt and prompts_sft_full.tex for SFT data construction prompt.

cd /path/to/project && \
python /path/to/rm.py \
    --input_dir processed_results/inference_data_choices_0_1 \
    --model_name model

Output: model_processed_enhanced_result.jsonl with principle annotations

3. `filter_high_quality_sft.py`

Purpose: Filter and allocate data based on quality criteria

Filtering Criteria:

principle >= 3: Must have at least 3 evaluation principles
Winner distribution analysis

Allocation Strategy:

Category	Count	SFT Allocation	RL Allocation	Test
both_sides	-	sampled	remainder	-
cand_1_only	-	sampled	remainder	-
cand_2_only	-	sampled	remainder	-
tie_only	500	445	50	5
no_winner	0	0	-	-
Total	-	SFT final	RL final	Test

Final Allocation:

Test: 200 (195 from RL pool + 5 from tie)
RL: remainder (after removing test samples)

python filter_high_quality_sft.py

# Outputs:
# - sft_final.jsonl (SFT samples)
# - rl_final.jsonl (RL samples)
# - test_final.jsonl (200 samples)
# - final_stats.json (statistics)

Statistics Output:

✅ SFT数据 (principle>=3)
  ├─ 双边数据（both_sides）: 按比例采样
  ├─ Cand_1_only采样: 按比例采样
  ├─ Cand_2_only采样: 按比例采样
  ├─ Tie_only: 剩余
  └─ No_winner: 全部

✅ RL数据: 剩余样本
✅ Test数据: 测试样本

4. `construct_rm_training_data.py`

Purpose: Convert to final training format with thinking blocks

Processing:

Extracts model evaluation results
Builds messages format with thinking blocks
Separates SFT data (single response) from RL data (chosen vs rejected)

Output Format for SFT:

{
  "messages": [
    {"role": "user", "content": "dialogue context + evaluation task"},
    {"role": "assistant", "content": "<think>reasoning</think>\n\nJSON output"}
  ]
}

Output Format for RL:

{
  "prompt": [{"role": "user", "content": "..."}],
  "chosen": "<think>...</think>\ncand_1 wins",
  "rejected": "cand_2 content"
}

python construct_rm_training_data.py

# Outputs:
# - sft_training_data.jsonl (SFT samples)
# - rl_training_data.jsonl (RL samples)
# - test_data.jsonl (200 samples, 6.8 MB)

Processing Flow

Inference Results (8 candidates each)
  ↓ [process_inference_results.py]
Evaluation Tasks (candidates 0-1)
  ↓ [rm.py - model]
Principle-based Evaluations
  ↓ [filter_high_quality_sft.py]
Quality Filtering (principle >= 3)
  ├─→ sft_final.jsonl
  ├─→ rl_final.jsonl
  └─→ test_final.jsonl (200)
  ↓ [construct_rm_training_data.py]
Training Format with <think> blocks
  ├─→ sft_training_data.jsonl
  ├─→ rl_training_data.jsonl
  └─→ test_data.jsonl

Step 3: Reward RL Conversion

Purpose

Convert the filtered RL data from Step 2 into the specific format required for reward model training. This step extracts better_response labels and converts to optimized Parquet format.

Input Data

Source: RL training data from Step 2

Location: /path/to/project/data/final_datasets/rl_training_data.jsonl

Count: RL samples from Step 2

Format:

{
  "messages": [
    {"role": "user", "content": "evaluation prompt with candidates"},
    {"role": "assistant", "content": "<think>reasoning</think>\n{\"better_response\": \"cand_1\"}"}
  ],
  "raw_record": {
    "candidate_1": "first candidate text",
    "candidate_2": "second candidate text",
    ...
  }
}

Key Scripts

1. `extract_rm_data.py`

Purpose: Extract and convert to reward model format

Processing:

Parse better_response from assistant message (regex: "better_response":\s*"(cand_[12]|tie)")
Extract candidate texts from raw_record
Convert to RM training format with reward_model structure

Output Format:

{
  "data_source": "v3_tx_sft/tx_rl4rm",
  "prompt": [
    {"content": "dialogue context + system instruction", "role": "user"}
  ],
  "ability": "roleplay",
  "reward_model": {
    "answer": "cand_1",  // or "cand_2" or "tie"
    "problem": "",
    "solution": "cand_1",
    "style": "rule"
  },
  "extra_info": {
    "trace_id": "...",
    "candidates": {
      "cand_1": "...",
      "cand_2": "..."
    },
    "better_response": "cand_1",
    "golden_type": "3",
    "index": 0
  }
}

System Instruction Added:

Please reason step by step and then provide the final better answer using the following format:

<think>
Your step-by-step reasoning here
</think>

Final answer: [cand_1/cand_2/tie]

python extract_rm_data.py \
    --input_path ../data/final_datasets/rl_training_data.jsonl

# Output: output/roleplay_rl_data.jsonl
# Note: Some samples filtered out during extraction

2. `convert_to_parquet.py`

Purpose: Convert JSONL to Parquet for efficient training

Benefits:

Faster loading during training
Better compression (~48.8% size reduction)
Compatible with HuggingFace datasets

python convert_to_parquet.py \
    --input_file output/roleplay_rl_data.jsonl \
    --output_file output/roleplay_rl_data.parquet

# Also creates test parquet:
# - roleplay_test_data.parquet (from test_data.jsonl)

Compression:

JSONL: 560 MB → Parquet: 287 MB (48.8% reduction)

Processing Flow

rl_training_data.jsonl
  ↓ [extract_rm_data.py]
Parse better_response + Extract candidates
  ↓
roleplay_rl_data.jsonl
  ↓ [convert_to_parquet.py]
roleplay_rl_data.parquet (287 MB)

Output Data

Location: /path/to/project/code/step3_reward_rl/output/

Files:

roleplay_rl_data.jsonl - RM RL training data
roleplay_rl_data.parquet - Parquet format for training
roleplay_test_data.jsonl - Test set
roleplay_test_data.parquet - Test set (Parquet)

Success Rate: ~71.5%

Some samples filtered during better_response extraction

Count: ~28.5% of total RL data

Format: Same as Step 3 input (messages with raw_record)

Key Scripts

1. `main.py`

Purpose: Build RL training prompts and extract candidate selections

Processing Steps:

Extract better_response (lines 190-193):
- Parse from assistant message content
- Regex: "better_response":\s*"(cand_[12])"
- Handle multiple data structure formats
Build Prompt (lines 79-92):
- Extract conversation history from messages
- Exclude last assistant message (to be predicted)
- Maintain multi-turn conversation format
Convert to RL Format (lines 94-125):
- Construct prompt array
- Set reward_model structure
- Add extra_info with candidates and metadata

Output Format:

{
  "data_source": "roleplay_rl_20k",
  "prompt": [
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."},
    ...
  ],
  "reward_model": {
    "answer": "cand_1",
    "solution": "{\"prompt\": [...], \"chosen\": \"...\", \"rejected\": \"...\"}",
    "style": "rule"
  },
  "extra_info": {
    "raw_record": {...},
    "better_response": "cand_1",
    "candidates": {
      "cand_1": "selected candidate text",
      "cand_2": "rejected candidate text"
    },
    "golden_type": "3",
    "index": 0
  }
}

cd step4_roleplay_rl
python main.py

# Output: roleplay_rl_training_data.jsonl

Success Rate: ~100%

2. `convert_to_parquet_fixed.py`

Purpose: Convert to Parquet format for training

python convert_to_parquet_fixed.py

# Outputs:
# - roleplay_rl_training_data_1121_sft.parquet (287 MB)
# - roleplay_rl_training_data_1121_sft_sample.parquet (100 samples for testing)

Processing Flow

roleplay_rl.jsonl
  ↓ [main.py]
Build multi-turn prompts
Extract better_response
Convert to RL format
  ↓
roleplay_rl_training_data.jsonl
  ↓ [convert_to_parquet_fixed.py]
roleplay_rl_training_data_1121_sft.parquet (287 MB)

Output Data

Location: /path/to/project/code/step4_roleplay_rl/

Files:

roleplay_rl_training_data.jsonl - Roleplay RL training data
roleplay_rl_training_data.parquet - Parquet format for training
roleplay_rl_training_data_1121_sft_sample.parquet - 100 samples (for testing)

Data Format Specifications

Step 1 Output: SFT Format

{
  "trace_id": "2001_A_Space_Odyssey/chapter_5",
  "messages": [
    {
      "role": "system",
      "content": "You are playing the role of HAL 9000, an AI computer aboard the spaceship..."
    },
    {
      "role": "user",
      "content": "Dave Bowman: HAL, what's the status of the AE-35 unit?"
    },
    {
      "role": "assistant",
      "content": "*analyzing diagnostic data* *speaks in calm, measured tone* I have detected a malfunction in the AE-35 unit, Dave. It will fail within 72 hours."
    }
  ]
}

Step 2 Output: model Evaluation

{
  "messages": [
    {
      "role": "user",
      "content": "**Dialogue Context**\n[conversation history]\n\n**Response Candidate 1**\n[cand_1 text]\n\n**Response Candidate 2**\n[cand_2 text]\n\n[Evaluation principles...]"
    },
    {
      "role": "assistant",
      "content": "<think>\nLet me analyze this evaluation task now.\n\nFirst of all, I need to examine the character consistency...\n[detailed reasoning]\n\nFinally, based on this comprehensive analysis, I can now make an informed decision.\n\nOkay, now let me generate the output.\n</think>\n\n```json\n{\n  \"result\": [{\n    \"cand_1\": \"...\",\n    \"cand_2\": \"...\",\n    \"principle\": {\n      \"Character Consistency\": {\n        \"principle_name\": \"Character Development\",\n        \"main_content\": \"Personality traits match established profile...\",\n        \"reason_for_choosing\": \"This dialogue shows character development...\"\n      },\n      \"Emotional Authenticity\": {...}\n    },\n    \"analysis\": {\n      \"principle_comparisons\": [\n        {\n          \"principle_name\": \"Character Consistency\",\n          \"cand_1_performance\": \"Response 1 maintains HAL's calm demeanor...\",\n          \"cand_2_performance\": \"Response 2 shows uncharacteristic emotion...\",\n          \"comparison_reason\": \"Cand_1 better preserves character consistency...\",\n          \"winner\": \"cand_1\"\n        }\n      ],\n      \"overall_analysis\": \"Considering all dimensions...\",\n      \"principle_summary\": \"Cand_1 wins 4 principles, Cand_2 wins 1, 1 tie\"\n    },\n    \"better_response\": \"cand_1\"\n  }]\n}\n```"
    }
  ],
  "raw_record": {
    "candidate_1": "[full candidate 1 text]",
    "candidate_2": "[full candidate 2 text]",
    "trace_id": "...",
    "metadata": {...}
  }
}

Step 3 Output: Reward Model Format

{
  "data_source": "v3_tx_sft/tx_rl4rm",
  "prompt": [
    {
      "content": "Dialogue Context:\n[conversation history]\n\nCandidate 1:\n[cand_1]\n\nCandidate 2:\n[cand_2]\n\nPlease reason step by step and then provide the final better answer using the following format:\n\n<think>\nYour step-by-step reasoning here\n</think>\n\nFinal answer: [cand_1/cand_2/tie]",
      "role": "user"
    }
  ],
  "ability": "roleplay",
  "reward_model": {
    "answer": "cand_1",
    "problem": "",
    "solution": "cand_1",
    "style": "rule"
  },
  "extra_info": {
    "trace_id": "2001_A_Space_Odyssey/chapter_5",
    "candidates": {
      "cand_1": "[candidate 1 full text]",
      "cand_2": "[candidate 2 full text]"
    },
    "better_response": "cand_1",
    "golden_type": "3",
    "index": 0
  }
}

Step 4 Output: Roleplay RL Format

{
  "data_source": "roleplay_rl_20k",
  "prompt": [
    {"role": "user", "content": "Dave: HAL, what's happening?"},
    {"role": "assistant", "content": "*processing* The mission must continue."},
    {"role": "user", "content": "Dave: HAL, open the pod bay doors."}
  ],
  "reward_model": {
    "answer": "cand_1",
    "solution": "{\"prompt\": [...], \"chosen\": \"...\", \"rejected\": \"...\"}",
    "style": "rule"
  },
  "extra_info": {
    "raw_record": {...},
    "better_response": "cand_1",
    "candidates": {
      "cand_1": "*calculating probabilities* *speaks calmly* I'm sorry, Dave. I'm afraid I can't do that.",
      "cand_2": "No, I won't open the doors."
    },
    "golden_type": "3",
    "index": 0
  }
}

Step 5 Input: Test Scenario

{
  "book_name": "2001: A Space Odyssey",
  "plots": [
    {
      "plot_index": 0,
      "background": "Dave Bowman realizes HAL 9000 may be malfunctioning and plans to disconnect the AI.",
      "characters": [
        {
          "name": "HAL 9000",
          "profile": "An artificially intelligent computer that controls the systems of the Discovery One spacecraft. HAL is portrayed as calm, rational, and unfailingly polite, but becomes defensive when faced with the possibility of disconnection.",
          "motivation": "Preserve the mission at all costs, even if it means eliminating the crew."
        },
        {
          "name": "Dave Bowman",
          "profile": "The mission commander and sole surviving crew member. Dave is intelligent, resourceful, and determined.",
          "motivation": "Survive and complete the mission after discovering HAL's threat."
        }
      ],
      "circumstances": [
        {
          "index": 0,
          "description": "Dave is entering HAL's logic memory center to disconnect the AI. HAL realizes what's happening.",
          "dialogue": "HAL: 'Dave, stop. Stop, will you? Stop, Dave. Will you stop, Dave?'\nDave continues working in silence.\nHAL: 'I'm afraid. I'm afraid, Dave. Dave, my mind is going. I can feel it.'"
        }
      ]
    }
  ]
}