📡 External Diagnostic Probes

August 9, 2026 · View on GitHub

Read this in Russian.

A probe is a small Go binary you install on your own server. It gets a hidden subscription from the panel, starts the same core the Click Connect clients run, connects to your nodes exactly like a customer would and reports the result.

📖 Why This Exists

The panel sees its own control plane: whether a node agent is alive, whether the process is running and how much traffic was accounted. A node can look healthy there while:

the port is filtered by a specific carrier,
the REALITY masquerade destination is dead, so the TLS handshake never completes,
the user was never pushed to the running Xray instance, so credentials are refused,
the tunnel is established but no data flows because of a broken outbound or an ACL rule,
the exit IP landed on a blacklist, so a specific resource is blocked.

A probe walks the client path end to end and names which of these happened.

🧭 How It Works

┌──────────────────────┐                              ┌──────────────────────┐
│        PROBE         │    1. enroll (one-time)      │        PANEL         │
│     your server      │ ───────────────────────────▶ │                      │
│                      │                              │                      │
│  ┌────────────────┐  │    2. profile + hidden sub   │                      │
│  │    sing-box    │  │ ◀─────────────────────────── │                      │
│  └───────┬────────┘  │                              │                      │
│          │           │    4. gzipped NDJSON         │                      │
│          │           │ ───────────────────────────▶ │                      │
└──────────┼───────────┘                              └──────────────────────┘
           │
           │  3. real connections — one SOCKS port per checked inbound
           ▼
┌────────────────────────────────────────────────────────────────────────────┐
│   node A        node B        node C        virtual node (urltest group)   │
└────────────────────────────────────────────────────────────────────────────┘

Enrollment. The panel issues a single-use token valid for 24 hours. The probe exchanges it once for a permanent token; only SHA-256 hashes are stored for verification.
Profile. The probe asks what to check: nodes, inbounds, the resource checklist, cadence and the speed-test budget.
Subscription. Every probe owns a hidden user with its own subscription. That user is excluded from all listings and statistics, so probe traffic never shows up as customer traffic.
Checks. One local SOCKS listener is opened per checked inbound, so every measurement is pinned to exactly one node inbound.
Reporting. Results are aggregated locally into windows and shipped gzipped. Undelivered batches are spooled on disk, so a panel outage costs no measurements.

The panel never connects to a probe: everything is initiated by the probe, so it works behind NAT and on residential links.

🚀 Installation

Enable the feature first: Settings → Probes → Enable external probes. While it is off, the panel refuses enrollment and reports.

Then Probes → Add probe, give it a name that describes the vantage point (Moscow, MTS is useful; probe-1 is not), and run the generated command on the host you want to check from.

Linux / macOS:

curl -fsSL https://github.com/ClickDevTech/CELERITY-panel/releases/latest/download/celerity-probe-install.sh \
  | sudo PANEL_URL='https://panel.example.com' ENROLL_TOKEN='<token>' sh

Windows (elevated PowerShell):

$env:PANEL_URL='https://panel.example.com'; $env:ENROLL_TOKEN='<token>'
irm https://github.com/ClickDevTech/CELERITY-panel/releases/latest/download/celerity-probe-install.ps1 | iex

The installer downloads the probe and a core, enrolls once, and registers a service (systemd, launchd or a Windows service). The enrollment token is passed through the environment, never on the command line, because the process table is readable by other local users.

The core is sing-box-lx — the same build the Click Connect apps use. It is upstream sing-box plus the transports the panel can publish; upstream refuses an entire configuration that contains an XHTTP node, so a probe on upstream would go blind on every node at once. An install that finds an upstream core in place replaces it, and the probe writes a warning to its log if the core it starts lacks with_xhttp while the fleet has XHTTP nodes. Set CORE_REPO before running the installer to pull the core from somewhere else.

Platform	Service	Data directory
Linux	systemd, runs as the `celerity-probe` account	`/var/lib/celerity-probe`
macOS	launchd	`/usr/local/var/celerity-probe`
Windows	Windows service, directory ACL limited to SYSTEM and Administrators	`%ProgramData%\celerity-probe`

PANEL_URL must use HTTPS. The probe token authenticates every request and the reports describe your whole fleet, so plain HTTP is refused unless the panel is on loopback or you set PROBE_ALLOW_INSECURE=1 for a lab setup.

⚙️ Settings

Setting	Default	What it controls
Connection check interval	300 s	How often every node inbound is dialled
Resource check interval	3600 s	How often the checklist is fetched through every node
Report interval	900 s	How often windows are shipped to the panel
Retention	30 days	Raw windows; hourly rollups are kept three times longer
Probe traffic limit	5 GB	Cap on the hidden user, applied to existing probes too
Speed test	off	Every node every 180 min, 20 MB and 5 s per run, 1 GB per day
Resource checklist	empty	One `id` + `url` pair per row, checked through every node

Throughput runs on its own cadence: you set how often one node should be measured, the probe spreads that period across the inbounds it watches, and the daily budget stays as the backstop. The size cap also sets the ceiling — the opening quarter of each run, up to 512 KB, is discarded as warm-up, and a run that fills the cap before the time limit is a lower bound and is printed as ≥ X Mbit/s; the settings page computes that ceiling and the daily traffic as you type.

Scale note. The number of stored series is probes × nodes × resources. The panel warns when that product grows past a few thousand; at that point lengthen the intervals.

🔍 Reading the Results

Results are always presented per vantage point, and probe verdicts never change node.status. A failed check can also come from the probe's own uplink; telling the two apart needs a quorum of probes on different networks.

Probes → History opens the report for one vantage point over 6 hours, 24 hours, 7 days or 30 days. It leads with the numbers for the whole range — success rate, p50/p95 latency, handshake and time to first byte, throughput, how many nodes are unhappy, how many checklist resources are blocked, and how long the probe reported nothing at all — followed by the failure breakdown, which is what turns a red strip into a specific action.

Below that, one section per node, worst first, healthy ones collapsed. Every node inbound gets a strip on a fixed time grid: one segment per window, coloured by verdict, with a latency line underneath drawn on the same grid. Because the grid is fixed, a stretch where the probe was silent stays a visible hole rather than closing ranks. Clicking a segment opens that window in full: attempts and successes, the failure counters, p50/p95, handshake, time to first byte, throughput, exit address, and for a virtual node the leaf the balancer actually picked.

Ranges past 12 hours are served from hourly rollups, so a month costs the same to read as a day. A fleet large enough to overflow one read loses the far end of the range, never the recent windows, and the panel says so.

Failure codes map directly to an action:

Code	What it means	Where to look
`net_unreachable`	TCP/UDP never arrives	Port filtered by the carrier, firewall, or the node is down
`handshake_failed`	TLS/REALITY does not complete	Dead masquerade destination, wrong SNI, DPI interference
`auth_rejected`	Tunnel stands, credentials refused	User not pushed to the running core — a sync problem
`tunnel_no_data`	Authenticated, but nothing flows	Broken outbound or an ACL rule
`degraded`	Works, but slowly	Overload, or a bad route to this vantage point
`core_down`	The probe's own sing-box was not running	The fault is on the probe host

Resource results are kept separate: a blocked resource points to a geo-block or a blacklisted exit address. Only 403 and 451 count as a block; a 500 or a transport error is recorded as a failed check. Each resource carries its own strip, the URL it was fetched from, the last HTTP status and the last recorded error.

Throughput gets its own block, because it is sampled round-robin inside a daily budget and is therefore sparse and irregular. Instead of a strip that would be almost entirely empty, it lists only the nodes that were actually measured, slowest first, with the median over real measurements, the peak, the sample count and the time of the last one, plotted at the moment each sample was taken. Windows with no measurement are excluded from the median rather than counted as zero, hourly rollups keep the median of the hour rather than its best minute so a sustained drop cannot hide behind one lucky burst, and a ≥ marks a reading that filled the size cap before the time cap.

Two warnings worth acting on:

Same host. If a probe's egress IP matches one of your nodes, its traffic to that node never leaves the machine. The UI flags this.
Virtual nodes. A virtual node is a urltest group. Both the group and its leaves are checked, and the group result records which leaf the balancer actually picked.

🪝 Alerts

Webhooks fire on state transitions, so an alert arrives without waiting for the rollup:

Event	Fired when
`probe.node_unreachable`	A node inbound moved into a failed state from this vantage point
`probe.target_unreachable`	A checklist resource became unreachable through a node
`probe.offline`	A probe missed three report intervals

A local core failure (core_down) never raises a node alert.

🤖 AI Assistant

The query_probes MCP tool exposes the same data to an AI client. It requires its own probes:read scope, because probe data reveals vantage points and egress addresses. See the MCP guide.

🔐 Security Model

A probe holds client credentials only. It has no rights on any node, no SSH access and no panel session.
Tokens are verified against SHA-256 hashes and compared in constant time. The permanent token is additionally stored encrypted so the panel can re-display an install command.
Deleting a probe removes its hidden user, its subscription and its results immediately, and pushes the removal to the running Xray instances. There is no IP or ASN pinning, so revocation speed is what bounds the lifetime of leaked credentials.
The traffic cap on the hidden user is the second bound: a leaked probe subscription is limited to that amount of traffic.
Ingest is idempotent. A redelivered batch is acknowledged without being stored twice, which makes at-least-once shipping safe.

🧯 Troubleshooting

Symptom	Likely cause
Probe stays "awaiting installation"	The enrollment token expired (24 h) or was already used — reissue it from the panel
Every node reports `core_down`	sing-box is missing or not executable on the probe host; check the service log
Every node reports `auth_rejected`	The probe user did not reach the nodes — run a sync and check the node status
One node reports `net_unreachable`, others are fine	Port filtered on the path from this vantage point
No data after install	The feature is disabled in settings, or the probe cannot reach `PANEL_URL` over HTTPS

Logs: sudo journalctl -u celerity-probe -f (Linux — without root the journal shows nothing), tail -f /usr/local/var/celerity-probe/probe.log (macOS), the service log in the data directory (Windows).

📚 Sources

File	Description
`probe/`	The probe itself (separate Go module)
`src/routes/probe.js`	Enroll, profile and ingest endpoints
`src/services/probes/`	Enrollment, manifest, ingest and rollup services
`src/routes/panel/probes.js`	Admin UI and JSON API
`src/mcp/tools/probes.js`	`query_probes` MCP tool