Driving the editor via MCP
July 27, 2026 · View on GitHub
The editor (packages/frontend/editor) can be
driven by any MCP-capable agent (Claude Code, Claude Desktop, Codex, …): insert
and transform nodes, author materials and edit WGSL, drive the animation
timeline, and read back editor state and viewport screenshots. This is the
reference for how it's wired and how to use it.
The editor was built for this: every mutation funnels through a serializable
EditorCommand, every read through a serializable EditorQuery, and both types
live in a shared crate the native server and the editor both depend on.
Architecture
agent (MCP client) ──HTTP /mcp──▶ awsm-renderer-scene-mcp ──WebSocket /editor──▶ editor (browser tab)
(packages/mcp) editor dials out → EditorController
• rmcp tool layer id-tagged req/resp • src/remote.rs
• /editor ws + link + push events • calls controller directly
• /png side-channel • uploads PNG to /png/<id>
The one hard constraint: a browser tab can't be a server. So the editor
dials out to the native server's /editor WebSocket and serves the server's
requests against its EditorController. The link is one ordered channel: the
server tags each Request with an id and the editor replies with a Response
carrying the same id (ids correlate request↔response). Frames are JSON text;
rendered PNGs never ride the link — the editor POSTs the bytes to a /png/<id>
HTTP side-channel and returns a small handle, keeping the control link byte-light.
Three pieces:
| Piece | Where | Role |
|---|---|---|
awsm-renderer-editor-protocol | packages/mcp/editor-protocol | The serializable wire vocabulary — EditorCommand / EditorQuery / EditorSnapshot / QueryResult + the Request / Response envelope and the WsServerMsg / WsClientMsg WebSocket frames. Compiles for both wasm and native. |
awsm-renderer-scene-mcp | packages/mcp | Native binary. rmcp tool layer over streamable-HTTP + the /editor WebSocket link + the /png side-channel. Per-tab isolation via pairing codes. publish = false. |
| editor remote module | packages/frontend/editor/src/remote.rs | The WebSocket client: parse ?mcp=/?pair=, dial ws://<origin>/editor, read Request frames → call EditorController → reply with Response frames; POST screenshots to /png/<id>. |
All editor mutation flows through EditorController (the editor's single
command/query authority), so an agent and a human watching the same tab stay in
sync, and undo/redo/coalescing all work as in the UI.
Quick start
-
Start the editor and the MCP server together:
task mcp-devService Address Editor (Trunk) http://localhost:9085MCP server (HTTP + WebSocket) http://127.0.0.1:9086—/mcp,/editor(ws),/png/<id>,/debug,/health,/boot-error(The single port lives in
taskfiles/config.yml:PORT_MCP_HTTP_DEV. Run the server alone withtask mcp:serve, or the installed binary withawsm-renderer-scene-mcp.) -
Attach the editor to the server, either way:
- Button — open the editor normally (
http://localhost:9085) and click the link icon in the top bar ("Connect to MCP server"). It connects to the default server (http://127.0.0.1:9086); click again to disconnect. - URL param —
http://localhost:9085/?mcp=http://127.0.0.1:9086auto-connects on load (and points the button at that origin).
Connect and disconnect show a toast, and the button reflects the live state (
Connecting…→MCP connected). The server logseditor attachedonce the WebSocket link is up. With neither, the editor runs normally with zero remote overhead. When more than one tab/agent is connected the server asks for a pairing code — the agent prints it (pairing_status); enter it in the MCP modal or append&pair=<code>to the editor URL. - Button — open the editor normally (
-
Point your agent at the MCP server. A ready-to-use
.mcp.jsonlives in the repo root:{ "mcpServers": { "awsm-renderer-scene": { "type": "http", "url": "http://127.0.0.1:9086/mcp" } } }- Claude Code / Claude Desktop — a project-root
.mcp.jsonis picked up automatically; just (re)start the agent in this directory. - Codex / other MCP clients — register a streamable-HTTP MCP server at
http://127.0.0.1:9086/mcp.
- Claude Code / Claude Desktop — a project-root
Tool catalog
~130 typed tools plus MCP resources (the docs below) and prompts (workflow
templates). Each tool is a thin wrapper that builds an EditorCommand /
EditorQuery from typed (schema'd) parameters and relays it to the editor. Node
and asset references are UUID strings — get them from get_snapshot. This catalog
groups the tools by area; it isn't exhaustive — the escape hatches (bottom) reach
every command/query, and each tool self-describes over the MCP schema.
New to driving this over MCP? Read the Agent Guide (
awsm://docs/agent-guide) first — it covers the mutate→settle→screenshot loop, an end-to-end scene walkthrough, lighting, batching, and troubleshooting. For custom materials see the recipes cookbook (awsm://docs/material-recipes); for animation see Animation Authoring (awsm://docs/animation).
Connection / health
ping— confirm an editor is attached (fails fast otherwise).pairing_status— this session's pairing state (paired? this session's code? how many tabs/agents connected?) without performing an editor op. Call it after aNo editor is pairederror to surface the code for the human.get_console_logs { limit? }— recent editor notices (toasts) + raw tracing (WARN/ERROR from the render loop / bridges) from a ring buffer; surfaces runtime errors otherwise stuck in the browser.get_memory_stats— JS-heap bytes (Chrome) + renderer object counts (meshes / transforms / materials / lines / compiled render+compute pipelines), for leak / soak observability (sample over time — flat = healthy).GET /health(plain HTTP, not an MCP tool) —{ editor_attached, last_boot_error }. Check this first when/debug/ tool calls go silent: it truthfully reports a detached/dead session (the relay drops a session on transport failure) and surfaces a renderer boot error the tab POSTed to/boot-errorbefore any attach.
Discover / observe
get_snapshot— scene tree (ids/names/kinds + visible/locked), selection, mode, undo/redo depth, animation library, custom materials (incl.compile_okerrors), textures, project coordinate-system metadata. Start here.
get_mode— current workspace (scene/material/animation).get_skin_data { nodes? }— rig discovery: joints as scene-node ids (+ name, live flag, current local TRS); pose joints withset_node_transform, animate them withadd_tracktransform targets.get_morph_data { nodes? }/set_morph_weight { node, index, value }— live morph weights (+ target names via glTFmesh.extras.targetNames); set is a transient preview, tracks own the weights during playback.get_node_transforms { nodes? }— local TRS + world matrix per node (empty = all).get_children { node }— direct children as a lightweight[{ id, name, kind }]list.get_subtree { node? }— the id/name/kind subtree rooted atnode(or EVERY scene root when omitted), with nestedchildren. Both avoid the heavy whole-sceneget_snapshotwhen you just need to navigate the hierarchy (e.g. find the descendants of a node you just created/duplicated).get_node_details { nodes? }— full per-kind config + material assignment.resolve_node_material { node }— the material a node actually RENDERS with (the direct answer, vs parsing theNodeKindblob).get_node_bounds { nodes? }— world-space AABB{ min, max }per node (for framing/sizing) + a facing hint{ forward, up, right }: the node's local axes (−Z / +Y / +X) in world space, derived from its world matrix.forwardis the project's −Z-forward convention — use it to place things relative to a node's orientation ("on the back" = −forward). NOTE: this is the node's transform orientation; an imported model's geometry may face a different way (the convention; verify visually).get_material_wgsl { asset }— a custom material's WGSL source.get_material_diagnostics { asset }—{ registered, ok, errors }(tell a compile failure from a successful-but-dark shader).get_material_contract { transparent? }— the WGSL authoring ABI + legal keys.get_track_data { clip, track }— a track's full keyframes/sampler/mute/solo.get_frame_globals— renderertime/delta_time/frame_count/resolution.canvas_stats { region? }— mean/min/max luma over a region or the whole canvas.wait_render_settled { max_ms? }— block until recompiles drain + a frame presents. Call between an edit and a screenshot. Note: bloom / SSR / decal / cluster pipelines compile lazily on first use, so the first settle after enabling one of those features takes longer than later ones. Loads (import_model_from_url/load_project_from_url/load_player_bundle) are settle-visible: one settle after the call observes the fully-populated scene.screenshot_scene { width?, height? }/screenshot_material { width?, height? }/screenshot_texture { asset }— PNG as an MCP image block.
Scene / nodes
insert_primitive { shape, parent? }(plane/box/sphere/cylinder/cone/torus),insert_empty,insert_camera,insert_light { kind, parent? },insert_particle { parent? }(CPU particle emitter),insert_decal { parent? }(projection decal; transform = oriented unit-cube volume, projects down local -Z) — return the new node id.insert_instancer { mesh?, parent? }— insert an explicit Instancer node (one node referencing a mesh ASSET, drawn as ONE GPU-instanced mesh; optionally wires themeshref in the same undo step), then author placements withset_instancer_transforms { node, transforms, per_instance_colors? }(bulk-replace the whole instance list in one call). Other node kinds (Line, Sprite, Curve, Sweep, Instances) are created viadispatch_command { command: { cmd: "insert", spec: "line" | "sprite" | "curve" | "sweep" | "instances", … } }(unit-variant specs are the bare string or{"<tag>":{}}; inlined fields error with "expected unit").set_particle_emitter { node, spawn_rate?, burst_count?, max_alive?, one_shot?, space?, shape?, initial_speed?, lifetime?, size?, forces?, color_over_life?, size_over_life?, blend?, texture? }— typed, patch-style emitter config (send any subset; only those change).texture= a billboard SPRITE asset id: author a soft radial-alpha disc withcreate_textureand bind it for disc-shaped (alpha-masked) particles instead of hard squares.shapeis{point}/{sphere:{radius}}/{cone:{ angle_radians, direction}}(conedirectionis in the emitter's local space);forcesis a list of{gravity:{acceleration:[x,y,z]}}/{linear_drag:{coefficient_x1000}};blend:trueroutes through the transparent-blend pass for true alpha fades (smoke/glows). Errors if the node isn't an emitter.set_mesh_shadow { node, cast, receive }— toggle a Mesh / SkinnedMesh / InstancesAlongCurve node's shadow casting / receiving (read-modify-write of itsshadowconfig viaSetKind).set_instance_colors { node, colors }— set an InstancesAlongCurve node's per-instance linear-RGBA tints (empty clears them).node_set_transform { node, translation, rotation, scale }(rotation is a local quaternion[x,y,z,w]), plus convenience:set_translation,translate_by,set_scale,set_rotation_euler { euler, order? }.rename_node,delete_node,duplicate_node(deep clone as a following sibling — returns the new clone's root node id; descendants get fresh ids, found viaget_children/get_subtree),reparent_node,set_node_visible,set_node_locked,set_selection,set_prefab(mark/clear a node as a prefab root).
Project / import / history
new_project(seeds a key light + the default three-slot environment),load_project_from_url { base_url },import_model_from_url { url },undo,redo. Loads are settle-visible: onewait_render_settledafter the call observes the fully-populated scene (noget_snapshotpolling loops).save_project— serialize the open project (project.toml+assets/*side files) to a server-side directory; the bytes ride a side-channel (never the tool result) and the tool returns the path + a file manifest.verify_roundtrip— destructive save→load losslessness self-test; returns a before/after census report ({ before, after, equal, lossless }).import_cluster_asset { clusters_url }— import a pre-baked cluster-LOD asset as a view-only mesh, rendered through the bounded cluster pipeline (the same path the player uses). Use this instead ofimport_model_from_urlfor heavy static meshes: it renders multi-million-triangle geometry without the dense visibility-geometry explode that would otherwise crash the editor.clusters_urlpoints at a<id>.clusters.binproduced offline by theawsm-renderer-lod-bakeCLI (awsm-renderer-lod-bake model.glb --out ./assets). The node is not editable (no geometry stack — it IS the LOD); move/scale it and assign a material like any node. Per-mesh LOD for editable meshes is the separateset_mesh_lod { node, kind }(none/cluster/discrete, consumed by the export bake).
Materials
add_builtin_material { shading }(pbr/unlit),add_custom_material— return the new id.register_material,delete_custom_material,copy_material_instance { from, to },update_builtin_material(replace a built-in's variantMaterialDefwholesale).- Material variants — a mesh renders only entries of its own palette
(
mesh.material_variants; each entry = a library material + THIS mesh's independent overrides + a stable id + a display name). There is NOassign_material:add_material_variant { node, material, name? }(returns the new variant id; never changes what renders), thenselect_material_variant { node, variant? }(omit variant → unassigned magenta).remove_material_variant { node, variant },rename_material_variant { node, variant, name }. Every material tool (set_builtin_param,set_node_texture,set_node_material_uniform, …) edits the SELECTED variant, and each variant's tuning persists across switches — add the same library material twice for two independent looks. set_material_wgsl { material, wgsl }— replace source + synchronous recompile; answers truthfully (errors carry the compiler diagnostics, no silentok). The WGSL is validated against the contract for the material's CURRENT alpha mode (Blend → transparentTransparentShadingOutput; Opaque/Mask → opaqueOpaqueShadingOutput), so setset_material_alpha_mode blendBEFORE pushing a transparent body — otherwise that one call reports a transient contract error (the final state self-corrects once both are set, in either order, since each re-validates). Tool calls in one message aren't ordered — sequence them.- Authoring:
set_material_alpha_mode,set_material_double_sided,set_material_debug_color,set_material_layout { uniforms, textures, buffers },set_material_includes { keys },set_material_fragment_inputs { keys },set_material_uniform { material, name, value },set_material_texture { node, slot, texture? },set_builtin_param { node, param, value }(base_coloraccepts 3 floats RGB or 4 = RGBA with the 4th = alpha),set_builtin_alpha_mode { node, mode: opaque|mask|blend, cutoff? }— typed alpha mode for a built-in/inline material (glass =blend+ base_color alpha<1), no whole-MaterialDefresend.
Lighting / environment
set_light_color,set_light_intensity,set_light_range,set_light_angles,set_light_volumetric_intensity(presence in volumetric media, independent of surface intensity; needsset_post_process { atmosphere_mode: "fog" | "volumetric" }to be visible — with no medium there's nothing to scatter in).set_environment { skybox?, specular?, irradiance?, zenith?, nadir?, probe?, skybox_rotation?, specular_rotation?, irradiance_rotation? }— THREE independent slots (skybox background / specular IBL / irradiance IBL), each"builtin", a KTX cubemap (asset id or.ktx2URL), OR an agent-authored sky gradient: passzenith+nadir([r,g,b]linear) and it sets all three slots to that two-color gradient (author dusk/overcast/night/studio from your own colors, no hosted.ktx2). PARTIAL update: an omitted slot KEEPS its current binding (pass"builtin"to explicitly reset one). Read the slots back viaget_snapshot→project.environment.skybox_rotation/specular_rotation/irradiance_rotation([x,y,z]Euler degrees, applied X→Y→Z) turn each cubemap independently. Use them to aim the interesting quadrant of a bake at the camera, or swing a distracting one (an LED wall, a sun) out of shot, without re-baking the cubemap;yspins the room horizontally and is the usual knob. Set all three the same to turn the whole room — or set them differently on purpose: keying reflections from one direction while the visible backdrop faces another is a legitimate move, which is exactly why these are per-slot. Notespecular_rotationalso drives the SSR miss fallback (that path stands in for the IBL specular term, so the two must agree). Applies identically in the editor viewport, the exported bundle, and the player. Each omitted rotation is preserved;[0,0,0]clears one. Also available via Rotation… in the Environment ribbon, which opens a per-slot modal.set_shadows { … }/get_shadows— patch/read the renderer-wide shadow config (scene.shadows, persisted + carried in the player bundle): thesscs_*contact-shadow block,atlas_size,evsm_atlas_size/evsm_exponent/evsm_blur_radius,max_point_shadows/point_shadow_resolution,debug_cascade_colors. Patch semantics — only the fields you pass change.set_sscsis the legacy SSCS-only subset (now routed through the same patch).
Textures
add_texture_asset { proc }(checker/gradient/noise) andimport_texture_from_url { url }(PNG/JPEG/WebP, fetched + uploaded to the GPU) — both return the new id; bind withset_material_texture, orset_node_texture { node, slot, texture? }for a mesh node's built-in (inline PBR) slot (base_color | metallic_roughness | normal | occlusion | emissive).set_node_texture_transform { node, slot, offset?, scale?, rotation?, flow?, wrap_u?, wrap_v?, mag_filter?, min_filter?, mipmap_filter?, uv_set? }— patch the UV transform / flow / sampler of a built-in slot that already has a texture bound (patch-style: only the fields you pass change).scale>1tiles;flow=[u,v]auto-scrolls the texture in UV-units/sec (conveyors/water/lava;[0,0]stops it);wrap_*= repeat|clamp_to_edge|mirrored_repeat;*_filter= nearest|linear. Applying to an empty slot is rejected, not silently ignored. For a directional/keyframed scroll, use atexture_transformanimation track instead.
No inline base64. There is no create_texture tool — the agent authors a
texture by generating + hosting it, then import_texture_from_url { url }. Same
for environments (bake a .ktx2 cubemap offline, set_environment by URL) and
heightmaps (displace_from_texture { node, url, strength }). See the
awsm://docs/asset-workflows resource for the full pipelines.
set_node_material_uniform { node, name, value }— set a PER-MESH uniform override on a node with a custom material (value= typed{kind, value}), distinct fromset_material_uniform(shared asset default).purge_unused— delete every asset not referenced by the live scene (one undoable step; never removes an in-use asset).
View / camera / time
switch_mode { mode },snap_camera_to_axis { axis },reset_camera.set_camera_orbit { yaw, pitch, radius, look_at },set_camera_projection { perspective, fov_y? },frame_node { node, padding? }(padding 0 = tight; fits the node's bounds to fill the view),set_camera_clip { manual?, near?, far? }(pin or restore AUTO clip planes). Depth is reverse-Z by default (near=1, far=0 — huge far/near ratios don't z-fight;?noreversezon the editor URL rolls back to forward-Z).set_view_options { grid?, gizmos?, light_gizmos?, skeleton_viz?, msaa?, smaa?, … }/get_view_options— viewport toggles (turn grid/gizmos off for clean verification screenshots).set_post_process { … }/get_post_process— tonemapping / bloom / dof / exposure + the flatssr_*andatmosphere_*blocks (patch semantics).atmosphere_modeis three-way —off/fog(analytic) /volumetric(froxel, light shafts) — not an enable plus a style flag: volumetric REPLACES fog, since both describe the same air.reset_pose { node }— restore a node + all descendants to their scene base transforms; reverts a clip's last-previewed pose left baked after clearing the current clip (pass a rig root to reset a skeleton). Transient, not undoable.set_frame_time { seconds }/clear_frame_time— pinframe_globals.timefor deterministic temporal-material screenshots. Also pins texture UV flow scroll (set_node_texture_transform flow=) to that absolute time (offset = base + velocity*t), so a scrolling texture screenshots the same phase every call.
Animation
add_clip(returns the new id),delete_clip,duplicate_clip,rename_clip,set_clip_duration,set_clip_speed,set_clip_loop,set_current_clip,set_playhead { t },set_playing { on }.- Typed tracks/keys:
add_track { clip, target },add_keyframe,set_keyframe,delete_keyframe,delete_track { clip, index }.target.kind: transform | morph | uniform | builtin_param | light | camera | texture_transform (node+slot[base_color|metallic_roughness|normal|occlusion|emissive] +prop[offset(vec2) | scale(vec2) | rotation(scalar)] — keyframe a built-in texture's UV transform, e.g. a directional/reversible conveyor scroll per clip). - Track flags + transport:
set_track_mute,set_track_solo(any solo ⇒ only soloed tracks pose),set_track_sampler { sampler: step|linear|cubic },step_playhead { to: home|prev|next|end }.
Mesh editing (procedural stacks + raw vertices)
- Every procedural node is an editable
Meshbacked by aModifierStack(MeshDef).get_mesh_modifiers { mesh }reads the recipe{ base, modifiers }(null if none yet);set_mesh_modifiers { mesh, stack }replaces it;add_modifier/set_modifier { index }/remove_modifier { index }edit it incrementally (mesh refs are asset UUIDs, not node ids). collapse_mesh_stack { mesh }bakes the stack to frozen-topology raw triangles (undoable);bake_alldoes it project-wide (finalize).get_mesh_layersshows live-vs-locked layers;get_mesh_stats/get_node_bounds/get_mesh_cross_sectionmeasure resolved geometry.- Raw-vertex editing (after collapse, or on captured meshes):
get_vertex_data { node, indices },select_vertices_where { node, predicate }→ indices,set_vertex_positions,set_vertex_normals,paint_vertex_colors,soft_transform_vertices { falloff }(radial falloff),set_vertex_selection(viewport highlight). - Fused select-and-act (scales to full-res meshes — the index array stays
server-side, never round-trips):
paint_where { node, predicate, color }(= select_vertices_where + paint_vertex_colors in one call) andtransform_where { node, predicate, translation, falloff }(= select + soft_transform). Prefer these over the select→indices→act pattern when a predicate matches thousands of verts (a real terrain's height band), which overflows the tool-result token cap if returned. Painted colors still only DISPLAY under a vertex-color-reading material (built-in PBR withvertex_colors_enabled). - Reusable selection HANDLE (§10 — when one selection drives many ops):
select_vertices_where { …, store: true }keeps the indices server-side and returns{ id, count }. Thenpaint_vertex_colors,soft_transform_vertices,set_vertex_positions,set_vertex_normalsandget_vertex_dataall acceptselection: <id>instead ofindices— so one full-res selection can be painted, sculpted, then read back without the index array ever crossing the wire.count_only: truereturns just the count;offset/limitpage the rawindices(andget_vertex_data's output) for a large selection.- ⚠️ Splat-weight footgun: unpainted vertex color is
(1,1,1,1)white, not 0 —mix(base, snow, vColor.r)reads full weight everywhere until you paint. Clear-to-0 first:paint_where { node, predicate: {"kind":"within_aabb","min":[-1e9,-1e9,-1e9],"max":[1e9,1e9,1e9]}, color:[0,0,0,1] }zeroes every vertex in one call, then paint the band.
- ⚠️ Splat-weight footgun: unpainted vertex color is
Rig / skin
get_skin_data(joints as node ids — see Discover),get_skin_weights { node }/set_skin_weights { node, entries }(per-vertex joints+weights, live re-deform),solve_ik { end_node, target, root_node? }(analytic two-bone IK;root_nodepins the chain root when the auto end→parent→grandparent walk picks wrong bones),drop_skinning { node }(bake a skinned mesh to a static editable Mesh).
Bake / export / bundle
export_scene_glb/export_node_glb— bake to binary glTF; PBR→glTF PBR, Unlit→KHR_materials_unlit, custom/Toon→AWSM_materials_none. The bytes ride the/glb/<id>side-channel — the tool returns a temp-file path + byte length, never inline base64.export_player_bundle— bake the project to a runtime bundle dir (scene.tomlassets/); files ride the/bundleside-channel, the tool returns{ bundle_dir, files, total_bytes, url_base }. Per-texture bundle encoding is authored with the dedicatedset_texture_exporttool (lossless WebP default; lossy + quality per texture).load_player_bundle— round-trip self-test: bundle the current project in-memory, reset, reload throughpopulate_awsm_scene.
Batch + generic escape hatches — full coverage
dispatch_batch { commands }— a list of rawEditorCommands applied atomically as one undo step (one round-trip).dispatch_command { command }— a single rawEditorCommand(tagged by"cmd").run_query { query }— a rawEditorQuery(tagged by"query").patch_kind { node, patch }— edit a node's kind with an RFC 7386 JSON merge-patch instead of resending the wholeNodeKindviaSetKind. Only the fields inpatchchange;nullremoves a key; nested objects merge; arrays replace. The result must still be a validNodeKind(rejected loudly). The ergonomic pattern for escape-hatch edits without a typed tool:get_node_detailsto see the exact shape + field names, then send just the delta.
Resources (read-only docs): awsm://docs/mcp, awsm://docs/agent-guide,
awsm://docs/asset-workflows, awsm://docs/material-recipes,
awsm://docs/animation, awsm://docs/mesh-tools,
awsm://docs/material-contract-opaque, awsm://docs/material-contract-transparent,
awsm://docs/material-contract-vertex.
Prompts (workflow templates): author_lit_material, setup_rotation_clip,
import_and_frame_model.
Push channel — the editor relays toasts (warning/error) and selection changes
to the agent as MCP notifications/message logging notifications, so an agent can
react to compile errors or a human clicking a node.
The escape hatches reach every EditorCommand / EditorQuery variant. The
authoritative inventory is the enums themselves:
controller/command.rs
and controller/query.rs
(which re-export from awsm-renderer-editor-protocol).
Wire protocol
The link is one ordered WebSocket. The server tags each Request with an id;
the editor replies with a Response carrying the same id (ids correlate
request↔response). Frames are the WsServerMsg / WsClientMsg envelopes,
serialized as JSON text. A single writer on each side owns the socket so
concurrent replies/events never interleave a half-written frame.
// awsm-renderer-editor-protocol
pub enum Request {
Dispatch(EditorCommand), // mutate
DispatchBatch(Vec<EditorCommand>), // atomic multi-command (one undo entry)
Query(EditorQuery), // structured read (snapshot / timeseries / pixels / stats / wgsl)
Undo, Redo, // controller methods, not EditorCommands
ScenePng, MaterialPng, TexturePng(AssetId), // rendered PNGs (returned as a handle)
Mode, // current workspace mode
}
pub enum Response {
Ok,
Query(Box<QueryResult>),
Png(PngHandle), // { id, byte_len, width, height } — bytes are at /png/<id>
Mode(EditorMode),
Err(String),
}
pub enum WsServerMsg { Request { id, req }, PairingRequired, Detached }
pub enum WsClientMsg { Pair { code }, Response { id, resp }, Event(EditorEvent) }
Why JSON. EditorCommand / EditorQuery are internally tagged
(#[serde(tag = "cmd")] / "query") and QueryResult is untagged, which require
a self-describing format (deserialize_any). JSON handles all of them and is
debuggable in the browser devtools. Since PNG bytes ride the /png side-channel
(not the link), the control frames stay small and human-readable.
The /png/<id> side-channel. A screenshot_* request renders the PNG, the
editor POSTs the raw bytes to POST /png/<id> (a separate HTTP connection, off
the control link), and returns only a PngHandle. The rmcp tool reads the bytes
back from the temp file the upload landed in and returns them to the agent as an
MCP image block. Retained files are LRU-capped on disk.
No certificates. The link is a plain ws:// (loopback). For a TLS-terminated
remote server, tick "Use TLS" in the connect modal (or set it via the modal) for
wss://. There is no cert-pinning / /control handshake anymore.
POST /debug. The server exposes a raw-request seam: POST a JSON Request
and it's relayed to the editor, returning the Response as JSON (a PNG request
returns the handle; fetch the bytes at /png/<id>). Handy for curl-driving the
pipeline without an MCP client. Example:
curl -s -X POST http://127.0.0.1:9086/debug -H 'content-type: application/json' \
-d '{"Dispatch":{"cmd":"insert","spec":{"primitive":{"box":{"dims":[1,1,1]}}},"parent":null}}'
Toolchain notes
- rmcp 1.x (
rmcp = "1", featuresserver, macros, schemars, transport-streamable-http-server). It's edition-2024, so the workspacerust-versionis 1.85 — with that floor a plainrmcp = "1"resolves cleanly (no pinning).ServerInfois#[non_exhaustive]in 1.x; build it fromDefault+ field assignment.StreamableHttpServiceis a tower service mounted on the axum router vianest_service("/mcp", …)— rmcp ships no HTTP listener of its own, hence axum. Loopback agents sit idle between tool calls, so the rmcp sessionkeep_aliveis set to a day (the 5-minute default would reap a live-but-idle session). - The link is
axum's built-in WebSocket (thewsfeature) on the server andgloo-net's WebSocket on the wasm editor — no extra transport stack, no certs.
Troubleshooting
no editor attached— no editor tab is connected. Openhttp://localhost:9085/?mcp=http://127.0.0.1:9086and wait foreditor attachedin the server log. The editor auto-reconnects with backoff, so a server restart re-attaches on its own (no tab reload needed).No editor is paired with this MCP session— more than one tab/agent is connected, so the server can't auto-bind. Callpairing_statusto get this session's code, then enter it in the editor's MCP modal (or open the editor with&pair=<code>).- The editor's socket dropped (tab reloaded/closed/frozen) — the server forgets
the connection, so
GET /healthreportseditor_attached:false(andlast_boot_errorif the page failed to init). The tab re-attaches when it's back. - Black
screenshot_scene—requestAnimationFrame(and thus the WebGPU draw loop) pauses for hidden/background tabs, and a WebGPUtoDataURLread can come back black if it lands between presents. Make sure the editor tab is the visible, foreground tab; see DEBUGGING-PREVIEW.md. - Verify two ways —
screenshot_scenelets the agent see its own effect through MCP; a human (or the Claude-in-Chrome extension) can watch the same live tab to confirm visually.
Typed-tool coverage (escape hatches serve the long tail, not core features)
A capability you can only reach by reverse-engineering a contract doc or hand-rolling
a raw dispatch_command / run_query isn't really discoverable. The rule: every
core capability gets a typed tool + a recipe + a point-of-use link, and typed
coverage is kept complete and consistent across a family so absence is never a
surprise. In particular, every per-vertex attribute now has a typed authoring verb
— set_vertex_positions / paint_vertex_colors / set_vertex_normals /
set_vertex_uvs (UVs were the missing one), with the matching read
get_vertex_data { include_source } and topology read get_mesh_data.
dispatch_command / run_query remain for the genuine long tail (uncommitted
commands, NLA mixer ops, one-off queries) — not as the only path to a real feature.
Known limitations / future
- Per-tab isolation. Each editor tab (one
/editorsocket) and each MCP agent get their own identity, bound to each other. Binding is automatic when exactly one unbound tab and one unbound agent exist; otherwise the agent surfaces a 4-char pairing code the tab presents (via the modal or?pair=). Requests, responses, and events never cross between sessions (link::EditorLink). - Editor→agent push is implemented for toasts (warning/error) and selection
changes: the editor sends a
WsClientMsg::Event(remote::notify_event), the server tags it with the originating connection id and fans it out, and each agent's forwarder keeps only its bound tab's events, relaying them asnotifications/messagelogging notifications (on_initialized). Other event kinds (and an MCP resource-subscription model) are future work.
Source anchors
- Protocol crate:
packages/mcp/editor-protocol/src(command.rs,query.rs,node_spec.rs,anim_ui.rs,transport.rs). - Server:
packages/mcp/src—mcp.rs(tools),ws.rs(/editorWebSocket, single-writer),link.rs(connections / agents / pairing),http.rs(/editor,/png,/debug,/health,/mcpmount). - Editor remote:
packages/frontend/editor/src/remote.rs;?mcp=parsing inmain.rs. - Controller surface:
controller/command.rs,controller/query.rs,controller/state.rs(dispatch/query/snapshot),engine/query.rs(PNG / canvas readback).