Scaling

Overview

Running xyOps in live production with lots of servers and/or lots of running jobs? Please read these best practices for scaling your deployment. This guide complements Self-Hosting -- start there first: see Self-Hosting.

Upgrade Hardware

• CPU cores: xyOps is multi-process and highly concurrent. More cores help the scheduler, web server, storage I/O, and log compression run smoothly under load.
• RAM: Add headroom for the Node.js heap, in-process caches, storage engine caches, and OS page cache. RAM directly improves cache hit rates and reduces disk/remote I/O.
• Storage: Prefer fast SSD/NVMe for local Filesystem/SQLite and log archives. Ensure enough IOPS for parallel job logs, snapshots, and uploads.
• Network: For large fleets, ensure good NIC throughput and low latency between conductors, workers, and shared storage. For multi-conductor production, place conductors close to both the JSON data store (Redis or Postgres) and the file store (S3 or S3-compatible).
• OS limits: Increase file descriptor and process limits for busy nodes (e.g. ulimit -n, systemd Limits). Ensure swap is configured conservatively to avoid heap thrash.

Increase Node.js Memory

xyOps honors the NODE_MAX_MEMORY environment variable to set Node's old-space heap size (default 4096 MB).

• Example: export NODE_MAX_MEMORY=8192 before starting xyOps (or -e NODE_MAX_MEMORY=8192 for Docker).
• Leave headroom for the OS, filesystem cache, and any external daemons. On an instance with 16 GB RAM, an 8-12 GB heap is typical depending on other workloads.
• Monitor RSS vs. heap usage over time and adjust conservatively to avoid swapping.

Increase Storage RAM Cache

xyOps uses pixl-server-storage and most engines support an in-memory cache for JSON records. Larger caches reduce round-trips to disk or network backends.

• Defaults: The sample config enables caches with maxBytes ≈ 100 MB and maxItems ≈ 100k for Filesystem and SQLite.
• Recommendation: For large production installs, consider increasing 5-10× if you have RAM available, and then tune based on hit ratio and latency.
• Where to set:
  • SQLite: Storage.SQLite.cache.enabled, Storage.SQLite.cache.maxBytes, Storage.SQLite.cache.maxItems.
  • Filesystem: Storage.Filesystem.cache.enabled, ...maxBytes, ...maxItems.
  • S3: Storage.S3.cache.enabled, ...maxBytes, ...maxItems (useful to reduce S3 GETs).
• See Storage Engines for engine-specific details and considerations (e.g., what is cached, eviction policy, binary vs JSON behavior).

Disable QuickMon

QuickMon sends lightweight metrics every second from all satellites. At large scale, per-second telemetry can add up. To reduce ingestion and WebSocket traffic, disable it:

• Set satellite.config.quickmon_enabled to false in your config. The setting is distributed to all servers automatically when they connect.
• Minute-level monitoring remains enabled via satellite.config.monitoring_enabled.

Disable Job Network Monitoring

For Linux servers with a large amount of open network connections, you may want to disable real-time network monitoring while jobs are running. By default, xyOps Satellite will continuously monitor server resources including processes and network connections, while active jobs are running on the server. This may add extra load on servers with tens of thousands of network connections.

To disable network monitoring while jobs are running, set the disable_job_network_io property to true in the /opt/xyops/satellite/config.json file on your large servers:

"disable_job_network_io": true

Or, you can set it globally in the main satellite.config object on your xyOps primary conductor server, which will automatically propagate out to all servers the next time they connect.

Multi-Conductor Setups

Multi-conductor requires shared external storage so all conductors see the same state. For live production, use a Hybrid storage setup with Redis or Postgres for JSON data, plus S3 or an S3-compatible service for files. Redis and Postgres support native storage transactions, so the database owns the commit or rollback decision if a conductor fails during a transaction.

For details, see the Storage Setup Guide.

NFS Warning

While it is possible to use the Filesystem engine with an NFS mount, this is only recommended if used in conjunction with the Hybrid engine, where the Filesystem is set as the binaryEngine to store only files, and the docEngine is set to Redis or Postgres for JSON data.

The reason is, xyOps reads and writes thousands upon thousands of tiny key/value records as part of its database system, and this is extremely difficult for NFS to deal with at scale. Also, NFS does not provide native pixl-server-storage transactions, so it should not be the document store for multi-conductor production.

Automated Backups

• Use the nightly API export for critical data as described in Self-Hosting: Daily Backups. Schedule via cron and store off-host.
• SQLite engine: It can perform its own daily DB file backups during maintenance. Configure in Storage.SQLite.backups (defaults keep the most recent 7). Note backups lock the DB briefly while copying.

Critical Errors

For critical errors (i.e. crashes and failed upgrades) you can configure a global System Hook to send out an automated email for each one. Set this in your config.json file, in the hooks object:

"hooks": {
	"critical": {
		"email": "ops-oncall@yourcompany.com"
	}
}

Or you can configure the hook to create a ticket (which in turn will email all the assignees):

"hooks": {
	"critical": {
		"ticket": {
			"type": "issue",
			"assignees": ["admin"]
		}
	}
}

See System Hooks for more details.

Monitoring Alert Emails

For server monitor alerts, you may want to send out emails. This can be set up at three different levels:

• At the alert level: You can edit individual alert definitions, and configure an email action for the important ones (e.g. "Low Memory" is a good one).
• At the server group level: You can set default alert actions for all alerts in specific server groups (e.g. "Production Databases").
• At the global configuration level. See below...

You can add global "universal" alert actions in the alert_universal_actions configuration object. These will fire for all alerts. Example:

"alert_universal_actions": [
	{
		"enabled": true,
		"hidden": true,
		"condition": "alert_new",
		"type": "snapshot"
	},
	{
		"enabled": true,
		"condition": "alert_new",
		"type": "email",
		"email": "oncall-pager@mycompany.com"
	}
]

Security Checklist

Harden your web entry point and xyOps config before going live:

• Configure Plugins to run as underprivileged users and/or groups (see Plugin Credentials).
• Restrict inbound IPs using WebServer.whitelist (supports CIDR). Only allow your corporate ranges and load balancers.
• Limit valid Host headers/SNI via WebServer.allow_hosts to your production domains (e.g. xyops.yourcompany.com).
• HTTPS: Enable WebServer.https, set cert/key paths, and consider WebServer.https_force so HTTP redirects to HTTPS. If terminating TLS upstream, configure WebServer.https_header_detect.
• Upload limits: Reduce WebServer.max_upload_size from the default 1 GB to your expected maximums (also adjust per-feature limits in client.*_upload_settings).
• Connection limits: Tune WebServer.max_connections and WebServer.max_concurrent_requests to match instance capacity. Optionally set WebServer.max_queue_length and WebServer.max_queue_active to cap overload.
• Timeouts: Consider WebServer.socket_prelim_timeout, WebServer.timeout, WebServer.request_timeout, and WebServer.keep_alive_timeout to mitigate slow-loris patterns and bound request durations.
• Bind address: If running behind a proxy, set WebServer.bind_address appropriately and configure WebServer.public_ip_offset to select the correct client IP from proxy headers.
• Headers/CSP: Use WebServer.uri_response_headers to enforce CSP, HSTS, and other security headers for HTML routes.
• Access control: Use WebServer.default_acl for private handlers and verify API keys/SSO policies. Lock down admin endpoints behind SSO where applicable.
• Rotate your secret key every few months. See Secret Key Rotation for details.

Plugin Credentials

xyOps Plugins can be configured to run as any user and/or group, by specifying a UID / GID for each one. However, you may also want to specify a set of default users / groups via the default_plugin_credentials configuration object. Using this you can set defaults per each plugin type:

"default_plugin_credentials": {
	"action": { "uid": "xyops", "gid": "xyops" },
	"event": { "uid": "xyops", "gid": "xyops" },
	"monitor": { "uid": "xyops", "gid": "xyops" },
	"scheduler": { "uid": "xyops", "gid": "xyops" }
}

Note that individual plugins can still specify their own UID/GID, which will override the defaults. An exception is Marketplace Plugins, which explicitly cannot specify their own UID or GID, and will always use the default credentials you set in default_plugin_credentials.

It should be noted that Docker-based Plugins, including the built-in Docker Shell Plugin, require elevated privileges in order to launch their containers. If you plan on using Docker features in xyOps, please make sure your underprivileged user has read/write access to the Docker socket, or set those specific plugins to run as root.

Note that Microsoft Windows doesn't have the concept of UIDs or GIDs, so Plugins on that platform will always run as administrator unless you specifically script them not to. For example, you can launch a Powershell script as a different user given their credentials (which should be stored in a Secret Vault):

# Read credentials from environment variables (secret vault)
$username = $env:WIN_USERNAME
$password = $env:WIN_PASSWORD

# Convert password to SecureString
$secure = ConvertTo-SecureString $password -AsPlainText -Force

# Build credential object
$cred = New-Object System.Management.Automation.PSCredential ($username, $secure)

# Launch child script as target user
Start-Process powershell `
    -Credential $cred `
	-LoadUserProfile `
	-WorkingDirectory "C:\scripts" `
    -ArgumentList "-NoProfile -ExecutionPolicy Bypass -File C:\scripts\child.ps1" `
    -Wait

Rate Limiting

If you are using our Multi-Conductor with Nginx or Multi-Conductor with OAuth2-Proxy and TLS with Nginx setups, consider adding on a rate limiting configuration. To do this, add a new volume bind to the Nginx Docker container:

-v ./limits.conf:/etc/nginx/conf.d/limits.conf:ro

And in the limits.conf file on the host side, add a Nginx configuration like this:

limit_req_zone $binary_remote_addr zone=req_per_ip:20m rate=100r/s;
limit_req_status 429;

This would limit traffic to 100 requests/sec per IP, utilizing up to 20MB of IP cache (around 300K IPs). For more details see the ngx_http_limit_req_module.

Additional Tuning Ideas

• Job throughput: Increase max_jobs_per_min prudently and monitor worker CPU/RAM. Align with your per-category limits and workflow constraints.
• Data retention: Cap history sizes to prevent unbounded growth via the db_maint *.max_rows properties (jobs, alerts, snapshots, activity, servers). Adjust to fit your storage budget.
• Search concurrency: If you run frequent file searches, consider increasing search_file_threads carefully (I/O bound; test first).
• Logging: Disable verbose request or storage event logs in production unless actively debugging (WebServer.log_requests, Storage.log_event_types).

References

• xyOps Self-Hosting Guide
• xyOps Storage Setup Guide
• Web server documentation