MCP (Model Context Protocol) Server

The MCP server is a lightweight HTTP server that provides a simple interface for interacting with the ALE-Bench toolkit. It allows you to run evaluations and manage sessions without needing to write Python code directly.

Setup

1. Install Node.js and npm

   # Install nvm (Node Version Manager) for easy Node.js management
   curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
   export NVM_DIR="$HOME/.nvm"
   [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
   [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion
   # Install the latest LTS version of Node.js
   nvm install --lts
   # Install the Model Context Protocol Inspector
   npm install -g @modelcontextprotocol/inspector
   

2. Install the MCP server dependencies using pip or uv:

   cd mcp
   uv sync
   uv sync --dev  # For development dependencies
   

Running the MCP Server

# Ensure you are in the mcp directory (e.g., cd mcp from the project root)
uv run mcp run server.py
uv run mcp dev server.py --with-editable .  # For development

Use with Claude Desktop

1. Open the claude_desktop_config.json file that configures the Claude Desktop. Add the following configuration to connect to the MCP server, ensuring you replace /path/to/ALE-Bench in the args with the actual absolute path to your cloned ALE-Bench repository directory:

   {
       "mcpServers": {
           "ALE-Bench MCP Server": {
               "command": "/bin/bash",
               "args": [
                   "-c",
                   "cd /path/to/ALE-Bench/mcp && uv run --with ale_bench --with mcp[cli] mcp run /path/to/ALE-Bench/mcp/server.py"
               ]
           }
       }
   }
   

2. Restart the Claude Desktop application to apply the changes.

Available Tools

The MCP server provides tools that wrap the core functionalities of the Session object. You can use these tools to perform actions like:

• Health Check: check_app
• Session Management: start_session, close_session, list_current_sessions, get_remaining_time, get_visualization_server_port
• Problem Information: list_problem_ids, get_problem, get_public_seeds, get_rust_tool_source, case_gen
• Code Execution: code_run
• Evaluation: case_eval, case_gen_eval, public_eval, private_eval
• Visualization: case_vis, case_gen_vis, local_visualization

For detailed information on the parameters and behavior of these functions, please refer to the Session Object documentation or the server implementation.