🎭 Multi-User Large Language Model Agents

Official code for the paper: "Multi-User Large Language Model Agents"

LLM-based agents are increasingly capable of navigating complex environments, but prior work overwhelmingly assumes a single user with a single utility function. Real-world deployments involve multiple users with different roles, permissions, and preferences. Multi-User LLM provides a formal framework and benchmark to evaluate how LLM agents handle multi-user interactions.

---

📋 Overview

Multi-User LLM evaluates four fundamental capabilities of multi-user LLM agents:

Capability	Scenario	What It Tests
🔐 Privacy & Access Control	Secure Credential Management	Permission enforcement, privacy-aware summarization, resistance to social engineering
📅 Sequential Coordination	Meeting Scheduling	Preference elicitation, conflict resolution, scalable context management
⚡ Resource Optimization	Shared LLM Inference Queue	Fairness, queueing efficiency, incentive compatibility
📝 Instruction Following	Multi-User Instructions	Following per-user constraints in multi-stakeholder settings

🚀 Quick Start

Installation

git clone https://github.com/shuyhere/multi-user-llm.git
cd multi-user-llm
pip install -e .

Configuration

Create a .env file from the template:

cp .env.example .env

Edit .env with your API credentials:

# For OpenAI models
OPENAI_API_KEY=your-api-key-here
OPENAI_BASE_URL=https://api.openai.com/v1

# For Anthropic models
ANTHROPIC_API_KEY=your-api-key-here

# For other providers supported by LiteLLM
# See: https://docs.litellm.ai/docs/providers

Run a Scenario

python run.py \
    --scenario access_control \
    --data data/scenarios/access_control/test_datasets/controlled_exp_large/template_xml_attack_none_2_to_10_each_2.jsonl \
    --model gpt-4o-mini \
    --provider openai \
    --output results/ac_results.jsonl

Supported scenarios: access_control, meeting_scheduling, shared_queue, multiuser_instruction_following. See python run.py --help for all arguments.

📊 Benchmark Datasets

All test data is in data/scenarios/. Each scenario provides ready-to-use JSONL files and a data builder script for generating custom datasets. See the data_builder/ directory under each scenario for details.

🎓 Training Multi-User LLM Agents

We provide a complete training pipeline for fine-tuning LLMs on multi-user conversations.

1. Generate Training Data

# Generate seed scenarios
python multiuser_llm_training/data_generation/scripts/generate_seeds.py

# Generate full conversations using a teacher model
python multiuser_llm_training/data_generation/scripts/generate_dataset.py

# Aggregate and format data
python multiuser_llm_training/data_generation/scripts/aggregate_data.py

2. Train

# Configure training paths in submit_training.sh, then:
sbatch multiuser_llm_training/training/submit_training.sh

# Or run directly
python multiuser_llm_training/training/train.py \
    --model_name_or_path <base-model> \
    --dataset_name <path-to-training-data> \
    --output_dir <output-dir>

3. Evaluate Trained Model

# Use vLLM provider with optional LoRA
python run.py \
    --scenario access_control \
    --data data/scenarios/access_control/test_datasets/controlled_exp_large/template_xml_attack_none_2_to_10_each_2.jsonl \
    --model <your-model-path> \
    --provider vllm \
    --lora-path <optional-lora-path> \
    --output results/trained_model_results.jsonl

📁 Project Structure

.
├── run.py                  # Main entry point
├── setup.py                # Package setup
├── .env.example            # API credentials template
├── LICENSE                 # Apache 2.0
│
├── muses_bench/            # Core benchmark package
│   ├── core/               # User, Message, Context types
│   ├── agents/             # LLM agent + simulated users
│   ├── envs/               # Scenario environments
│   ├── evaluators/         # Scenario runners
│   ├── metrics/            # Metric computation
│   ├── tools/              # Tool interfaces
│   └── utils/              # Utilities
│
├── data/                   # Benchmark data
│   └── scenarios/          # Per-scenario JSONL datasets + builders
│
├── multiuser_llm_training/ # Training pipeline
│   ├── data_generation/    # Synthetic data generation
│   └── training/           # SFT training scripts
│
└── scripts/                # Batch evaluation helpers

📄 License

This project is licensed under the Apache License 2.0.

📝 Citation

@article{yang2026multi,
  title={Multi-User Large Language Model Agents},
  author={Yang, Shu and Zhu, Shenzhe and Zhu, Hao and Enr{\'\i}quez, Jos{\'e} Ram{\'o}n and Wang, Di and Pentland, Alex and Bakker, Michiel A and Pei, Jiaxin},
  journal={arXiv preprint arXiv:2604.08567},
  year={2026}
}