Pelorus API Reference

The primary endpoint is POST /api/run for agent execution. Other endpoints are utility/debug helpers used mainly by the web frontend.

---

POST /api/run - Execute agent task

URL: POST /api/run

Request body (application/json):

Field	Type	Default	Description
prompt	string	required	Task description sent to the LLM
server_id	string|null	null	Select a named server from config.toml
provider	string|null	null	Override: "ollama", "openai", "gemini"
endpoint	string|null	null	Override: LLM API endpoint URL
model	string|null	null	Override: model name
api_key	string|null	null	Override: API key
vision	bool|null	null	Override: enable vision (image inputs)
max_steps	integer	15	Max agent loop iterations
response_format	string	"default"	One of "minimal", "default", "thoughts", "verbose"
stream	bool	false	When true, returns SSE stream

Config resolution order:

1. If server_id is provided, that server is loaded from the TOML config
2. Otherwise the server marked default: true, or the first server
3. Any explicit provider / endpoint / model / api_key / vision overrides the loaded config
4. If nothing resolves: ollama, http://localhost:11434, gemma4:12b

Response formats

When stream: false, the response is JSON. The response_format controls verbosity:

Format	success	final_text	steps[]	steps[].tool_screenshot	initial_screenshot	final_screenshot	steps[].thinking
minimal	yes	-	-	-	-	-	-
default	yes	yes	yes	null	-	-	-
thoughts	yes	yes	yes	omitted	-	-	yes
verbose	yes	yes	yes	base64 PNG	base64 PNG	base64 PNG	yes

# Minimal - returns {"success": true}
curl -s localhost:5100/api/run -H 'Content-Type: application/json' \
  -d '{"prompt":"open kate","response_format":"minimal"}'

Failure: success: false when the agent exhausted max_steps while still making tool calls.

SSE streaming

When stream: true, the response is text/event-stream. Events are emitted as the agent works:

event: step_start
data: {"iteration": 1}

event: tool_call
data: {"name": "computer", "input": {"action": "desktop_state"}}

event: tool_output
data: {"text": "=== Desktop (4 windows, 1920x1080) ==="}

event: step_end
data: {"iteration": 1, "has_tool_calls": true, "tool_calls_count": 1, "text": "..."}

event: done
data: {"success": true, "final_text": "Kate has been opened."}

Full event reference:

Event	Payload	When
step_start	{"iteration": n}	Each agent loop iteration begins
think	{"delta": "..."}	Streaming chain-of-thought token
tool_call	{"name": "...", "input": {...}}	Model invoked a tool
tool_output	{"text": "..."}	Text output from tool execution
tool_error	{"text": "..."}	Error from tool execution
tool_screenshot	{"data": "base64..."}	Screenshot (verbose format only)
step_end	{"iteration": n, "has_tool_calls": bool, ...}	Step completed
screenshot	{"which": "initial"|"final", "data": "base64..."}	Before/after (verbose only)
done	{"success": bool, "final_text": "...", "final_screenshot": "..."}	Agent run finished
error	{"message": "..."}	An error occurred

---

Other endpoints

Method	Path	Description
GET	/	Serve the SPA frontend (static/index.html), or {"status":"pelorus","docs":"/docs"}
GET	/docs	FastAPI auto-generated Swagger documentation
GET	/api/servers	List configured LLM servers from config.toml
POST	/api/servers	Add a new server
GET	/api/servers/{id}	Get one server
PUT	/api/servers/{id}	Update server fields
DELETE	/api/servers/{id}	Delete a server
GET	/api/servers/{id}/models	Fetch available models from a saved server's provider
POST	/api/models/fetch	Fetch models from ad-hoc provider config
GET	/api/env	Return first server's config for frontend pre-population
GET	/api/windows	List windows
GET	/api/state	Return the desktop state text sent to the LLM
WS	/ws	WebSocket endpoint for browser UI (full-duplex agent interaction)

The server CRUD, model fetch, and WebSocket endpoints are used by the web frontend and are fully documented via the interactive /docs Swagger UI.

---

Low-Level Desktop Integration

These endpoints expose pelorus as a standalone desktop control API. External scripts, bots, or any HTTP client can read desktop state, explore windows, capture screenshots, and send input commands without going through the LLM agent loop.

All coordinate values returned are screen-absolute -- ready to pass directly to the computer-use backend or any other click/input system.

GET /api/desktop/explore/{pid}

Returns the AT-SPI accessibility tree for a window, enriched with screen-absolute coordinates. This is the same data the agent loop uses internally.

If the window has no accessibility tree (games, Electron apps, custom GL canvases), a base64 PNG screenshot of the window region is returned automatically. The caller checks for the screenshot field to know if vision-based inspection is needed.

Query parameters:

Param	Type	Default	Description
force_screenshot	bool	false	Always include a window screenshot alongside the tree

Response (AT-SPI available):

{
  "pid": 9728,
  "tree": "[frame] \"Welcome — Kate\" (1297, 518, 688, 400)\n  [button] \"New File\" (1384, 644, 121, 25)\n  ..."
}

Response (AT-SPI unavailable — screenshot auto-included):

{
  "pid": 9728,
  "error": "Error: No AT-SPI application found for PID 9728",
  "screenshot": "base64..."
}

Response (force_screenshot=true, AT-SPI available):

{
  "pid": 9728,
  "tree": "[frame] ...",
  "screenshot": "base64..."
}

curl -s localhost:5100/api/desktop/explore/9728 | python3 -m json.tool

GET /api/desktop/screenshot

Returns a raw base64-encoded PNG screenshot of the full desktop.

Response:

{
  "data": "base64..."
}

curl -s localhost:5100/api/desktop/screenshot | python3 -c "
import sys, json, base64
data = json.load(sys.stdin)['data']
open('screen.png', 'wb').write(base64.b64decode(data))
print('Saved screen.png')
"

GET /api/desktop/screenshot/{pid}

Returns a base64-encoded PNG screenshot of a specific window's screen region.

Response:

{
  "pid": 9728,
  "screenshot": "base64..."
}

curl -s localhost:5100/api/desktop/screenshot/9728 | python3 -c "
import sys, json, base64
data = json.load(sys.stdin)['screenshot']
open('kate.png', 'wb').write(base64.b64decode(data))
print('Saved kate.png')
"

POST /api/desktop/control

Send input commands directly to the desktop. This is a passthrough to the computer-use backend (PIXELFLUX_CU on port 5000).

Request body:

Field	Type	Required	Description
action	string	yes	One of the actions listed below
coordinate	[int, int]	for click/move	[x, y] screen position
text	string	for key/type	Key name or text to type
start_coordinate	[int, int]	for drag	[x, y] drag start
end_coordinate	[int, int]	for drag	[x, y] drag end
scroll_direction	string	for scroll	"up", "down", "left", "right"
scroll_amount	int	for scroll	Number of scroll units
duration	float	for wait	Seconds to wait
region	[int, int, int, int]	for zoom	[x1, y1, x2, y2] capture region

Available actions:

Action	Parameters	Description
left_click	coordinate	Click at screen position
right_click	coordinate	Right-click at screen position
double_click	coordinate	Double-click at screen position
middle_click	coordinate	Middle-click at screen position
triple_click	coordinate	Triple-click at screen position
mouse_move	coordinate	Move cursor to position
left_mouse_down	coordinate	Press left mouse button
left_mouse_up	coordinate	Release left mouse button
left_click_drag	start_coordinate, end_coordinate	Drag between two points
scroll	coordinate, scroll_direction, scroll_amount	Scroll at position
key	text	Press a key (e.g. "Return", "ctrl+a")
hold_key	text, duration	Hold a key for duration
type	text	Type a string of text
cursor_position	-	Returns current cursor position
wait	duration	Pause for N seconds
screenshot	-	Capture full desktop screenshot
zoom	region	Capture a specific screen region

Response: JSON passthrough from the CU backend, typically:

{
  "status": "ok",
  "data": "base64..."
}

# Click at (500, 300)
curl -s -X POST localhost:5100/api/desktop/control \
  -H 'Content-Type: application/json' \
  -d '{"action": "left_click", "coordinate": [500, 300]}'

# Type text
curl -s -X POST localhost:5100/api/desktop/control \
  -H 'Content-Type: application/json' \
  -d '{"action": "type", "text": "hello world"}'

# Press Enter
curl -s -X POST localhost:5100/api/desktop/control \
  -H 'Content-Type: application/json' \
  -d '{"action": "key", "text": "Return"}'

POST /api/desktop/close/{pid}

Terminate a window's process by PID (sends SIGTERM).

Response:

{
  "ok": true,
  "pid": 9728
}

curl -s -X POST localhost:5100/api/desktop/close/9728

Typical workflow

1. GET  /api/state                         → see open windows and PIDs
2. GET  /api/desktop/explore/{pid}         → get element tree with screen coords
3. POST /api/desktop/control               → click/type/scroll using those coords
4. GET  /api/desktop/explore/{pid}         → verify result, get next element
5. POST /api/desktop/close/{pid}           → close when done

---

Configuration

Servers are stored in /config/agent/config.toml. On first run, the file is seeded from environment variables:

[[servers]]
id = "svr_abc123def456"
name = "Default Server"
provider = "ollama"
endpoint = "http://localhost:11434"
model = "gemma4:12b"
api_key = ""
vision = false
default = true

Seeding env vars:

Env var	Default
PELORUS_PROVIDER	ollama
PELORUS_ENDPOINT	http://localhost:11434
PELORUS_MODEL	gemma4:12b
PELORUS_API_KEY	""

Other env vars:

| Env var | Default | Purpose | |---|---|---|---| | PELORUS_PORT | 5100 | HTTP server port | | PIXELFLUX_CU | 5000 | Computer-use backend port |

Inline override: instead of pre-configuring a TOML server, you can pass provider, endpoint, model, and api_key directly in the POST /api/run request body. These override any config file values for that single request.

Environment variables in TOML values are resolved via ${VAR_NAME} syntax.

---

Vision fallback

Pelorus primarily drives the desktop through AT-SPI accessibility data, giving the LLM structured text with precise element coordinates. This is far more reliable than pixel-based approaches, click targets are known exactly, not guessed from images.

Vision mode (vision: true) is used as a fallback for applications that lack accessibility tree support (games, Electron apps without a11y bridges, custom GL canvases). When enabled, the agent can request per-application screenshots to interpret non-text interfaces. This is inherently less reliable for coordinate targeting, even frontier vision models misjudge click positions, but it allows Pelorus to function in environments where AT-SPI alone is insufficient. Because the primary loop uses text-based state, even quantized local models can maintain accurate desktop control.