3. How OpenEnv environments scale

This section covers benchmarking and scaling OpenEnv environments.

Contents:

• Provider Scaling
• WebSocket-based Scaling
• Microservice Scaling
• Scaling Experiments

Provider Scaling

The easiest way to scale an OpenEnv environment is to use a provider these are abstractions based on runtimes like Uvicorn, Docker Swarm, or Kubernetes.

from openenv.providers import UVProvider, DockerSwarmProvider, LocalDockerProvider

docker_provider = LocalDockerProvider() # default
uvicorn_provider = UVProvider() # python only
swarm_provider = DockerSwarmProvider() 

with EchoEnv.from_hub(
    repo_id="openenv/echo-env", 
    provider=swarm_provider, 
    replicas=4,
) as env:
  result = env.reset()
  result = env.step(EchoAction(message="Hello"))

WebSocket-based Scaling

OpenEnv uses WebSocket connections (/ws) instead of stateless HTTP for environment interactions. This design enables efficient scaling within a single container.

What are WebSockets?

WebSocket is a communication protocol that provides a persistent, bidirectional connection between client and server. Unlike HTTP—where each request opens a new connection, sends data, receives a response, and closes—a WebSocket connection stays open for the duration of a session.

For RL environments, this matters because a typical episode involves dozens to thousands of sequential step() calls. With HTTP, each step incurs TCP handshake overhead (~10-50ms). With WebSocket, messages are sent as lightweight frames (~0.1ms overhead) over the existing connection.

Also, with HTTP, long running sessions require logic to manage session state, which is not necessary with WebSocket.

Multiple sessions per container

With HTTP, maintaining session state requires cookies or session IDs with every request. Each isolated environment instance typically needs its own container:

HTTP approach: N parallel episodes → N containers

With WebSocket, one container handles many isolated sessions. Each WebSocket connection gets its own environment instance server-side:

# Single container serving multiple concurrent sessions
# docker run -d -p 8000:8000 my-env:latest

# Each client gets an isolated environment instance
with MyEnv(base_url="http://localhost:8000") as env1:  # Session 1
    result = env1.reset()
    
with MyEnv(base_url="http://localhost:8000") as env2:  # Session 2
    result = env2.reset()
    
with MyEnv(base_url="http://localhost:8000") as env3:  # Session 3
    result = env3.reset()

Server-side session state

The server maintains environment state per WebSocket connection which means that the environment builder does not need to worry about session state.

• No session IDs because Connection itself is the session
• Automatic cleanup because Environment instance destroyed when connection closes
• Isolation guaranteed because Each connection has dedicated state

# Server creates new environment instance per WebSocket connection
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    env = MyEnvironment()  # Fresh instance per connection
    await websocket.accept()
    
    while True:
        data = await websocket.receive_json()
        if data["type"] == "reset":
            result = env.reset()
        elif data["type"] == "step":
            result = env.step(data["action"])
        await websocket.send_json(result)
``$

### \text{Resource} \text{efficiency}

| \text{Approach} | \text{Containers} | \text{Memory} | \text{Startup} | \text{Max} \text{parallel} |
|----------|------------|--------|---------|--------------|
| \text{HTTP} (1 \text{env} = 1 \text{container}) | \text{N} | \text{N}  \times  ~100\text{MB} | \text{N}  \times  ~5\text{s} | \text{Limited} \text{by} \text{containers} |
| \text{WebSocket} (\text{N} \text{sessions} = 1 \text{container}) | 1 | ~200\text{MB} | ~5\text{s} | \text{Limited} \text{by} $MAX_CONCURRENT_ENVS` |

Configure session limits via environment variable:

```bash
docker run -d -p 8000:8000 -e MAX_CONCURRENT_ENVS=100 registry.hf.space/openenv-echo-env:latest

Scaling a Single Container

Before adding more containers, maximize the capacity of a single deployment. The key parameters are workers (CPU parallelism) and MAX_CONCURRENT_ENVS (session limit).

Uvicorn workers

Each Uvicorn worker is a separate process that can handle requests independently. More workers = more CPU cores utilized.

# Clone and run locally
git clone https://huggingface.co/spaces/burtenshaw/openenv-benchmark
cd openenv-benchmark
pip install -e .

# Run with 8 workers
WORKERS=8 uvicorn benchmark.server.app:app --host 0.0.0.0 --port 8000 --workers 8

The above example will use 8 workers and each worker will be able to handle 100 concurrent sessions. For simple environments, like text games, it's possible to get to 2000 concurrent sessions with 8 workers.

Note: More workers consume more memory. Each worker loads a full copy of the environment code.

Docker with environment variables

Pass scaling parameters when starting the container:

# Pull from HF Spaces registry
docker pull registry.hf.space/burtenshaw-openenv-benchmark:latest

# Run with custom configuration
docker run -d -p 8000:8000 \
    -e WORKERS=8 \
    -e MAX_CONCURRENT_ENVS=400 \
    --name openenv-benchmark \
    registry.hf.space/burtenshaw-openenv-benchmark:latest

HF Spaces configuration

Now, let's deploy the environment to HF Spaces so that we can interact with the server from the client. Configure scaling via Space Settings > Variables:

1. Go to your Space settings page
2. Add environment variables:
   • WORKERS=4 (max 4 on free tier, 8 on CPU Upgrade)
   • MAX_CONCURRENT_ENVS=100
3. Restart the Space

Limitation: HF Spaces free users tier caps at ~128 concurrent sessions regardless of configuration. See Scaling Experiments for measured limits.

Scaling limits

The experiments below found that even on larger instances, a single container eventually fails to scale and we need multiple containers to handle the load. For example, on a CPU Upgrade instance with 8 workers, the max batch was 1024 concurrent sessions:

• Success rate drops to 92%
• P99 latency exceeds 2× the expected step time
• Connection errors increase under load

When this happens, we need to scale to multiple containers and use a load balancer.

For high-throughput workloads, scale horizontally by running multiple environment containers behind a load balancer.

We explored this in detail in the Scaling Experiments repository.

static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 8080
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                upgrade_configs:
                  - upgrade_type: websocket
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: openenv_service
                      domains: ["*"]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            cluster: openenv_cluster
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
    - name: openenv_cluster
      connect_timeout: 30s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: openenv_cluster
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: host.docker.internal
                      port_value: 8001
              - endpoint:
                  address:
                    socket_address:
                      address: host.docker.internal
                      port_value: 8002
              - endpoint:
                  address:
                    socket_address:
                      address: host.docker.internal
                      port_value: 8003
              - endpoint:
                  address:
                    socket_address:
                      address: host.docker.internal
                      port_value: 8004

docker run -d \
    -p 8080:8080 \
    -v $(pwd)/envoy.yaml:/etc/envoy/envoy.yaml \
    --add-host=host.docker.internal:host-gateway \
    envoyproxy/envoy:v1.28.0

# Clients connect to Envoy, which distributes to backend containers
with MyEnv(base_url="http://localhost:8080") as env:
    result = env.reset()

Scaling expectations

Note: Actual throughput depends on environment complexity and hardware. Benchmark your specific workload.

Experiments Results

This section documents experiments measuring OpenEnv scaling characteristics across five infrastructure configurations. Full experiment data and code available at burtenshaw/openenv-scaling.

Experiment setup

Benchmark environment: A minimal OpenEnv environment with configurable wait time (simulates computation). Each step() call sleeps for the specified duration, isolating infrastructure overhead from environment logic.

Infrastructure tested:

Protocol: WebSocket (/ws) and HTTP (/reset, /step) compared where available.

Metrics:

• Max batch: Largest concurrent request count with ≥95% success rate
• Batch/core: Max batch divided by available cores (efficiency metric)
• P99 latency: 99th percentile total request time
• RPS: Requests per second at max batch

Results summary

All results measured with wait=10.0s step duration.

Maximum batch size by infrastructure (95% success threshold)

Finding 1: Local deployments have highest per-core efficiency

Single instance of Python and Docker both achieve 256 concurrent sessions per core—the highest efficiency observed. With 8 workers, both reach 2,048 concurrent sessions before degradation begins.

This makes sense because the environment is running in a single process and the overhead of the environment is relatively low. But it's ideal for hackers and developers who want to test their environment quickly or train on a single machine.

Beyond 2,048 concurrent connections, success rate drops sharply. The failure mode is connection rejection, not timeout—the server saturates its connection pool.

Per-core efficiency comparison across infrastructures

Finding 2: HF Spaces works reliably up to 128 concurrent sessions

HF Spaces free tier (cpu-basic) provides 2 workers and achieves 128 concurrent WebSocket sessions with 100% success. This translates to 64 sessions per core.

HF Spaces scaling behavior (WebSocket):

At 256 concurrent connections, results become unstable. At 512+, connections fail entirely due to HF Spaces connection limits.

HTTP mode does not work on HF Spaces. The /reset and /step HTTP endpoints are not accessible on the deployed Space—all HTTP requests fail. Use WebSocket mode exclusively.

Finding 3: Multi-node scaling works

Multi-node SLURM (96 cores across 2 nodes) achieves 16,384 concurrent sessions with 100% success rate—the highest absolute throughput tested.

SLURM multi-node scaling behavior:

The batch/core ratio (170.7) is lower than local deployments (256) but provides the highest absolute capacity for large-scale workloads.

Multi-node vs single-node scaling behavior

Latency breakdown

At max load (wait=1.0s), latency breaks down as:

Observations:

• Step latency is consistent across infrastructures (~1.0s for 1.0s wait), confirming the benchmark measures infrastructure overhead accurately
• Connect latency varies significantly—local Docker shows higher connect time at load (1.38s), likely due to container networking
• Multi-node has high connect latency (17.5s) at 16,384 batch due to queuing at the load balancer; this is the cost of handling 16× more connections than single-node

P99 latency across configurations and batch sizes

Success rate vs batch size for all infrastructures

Test methodology

# Clone benchmark environment
git clone https://huggingface.co/spaces/burtenshaw/openenv-scaling
cd openenv-scaling

# Run scaling test
python tests/test_scaling.py \
    --url http://localhost:8000 \
    --requests-grid 32,128,512,2048,4096,8192,16384 \
    --wait-grid 1.0,5.0,10.0 \
    --reps 3 \
    --mode ws \
    --output-dir experiments/results/

Each configuration was tested with 3 repetitions. Max batch is defined as the largest batch size achieving ≥95% success rate across all repetitions.

Summary

Recommendations:

1. For development and moderate workloads (<2,000 concurrent): Use single node Uvicorn or Docker depending software environment. These provide the best per-core efficiency (256 sessions/core).

2. For demos, testing, and published environments: HF Spaces free tier works reliably up to 128 concurrent sessions.

3. For large-scale training (>2,000 concurrent): Deploy multi-node with proper load balancing. Expect ~170 sessions per core, but much higher absolute throughput.