Skill-execution telemetry hooks

These hooks close the markdown-only skill-execution telemetry gap. The toolkit's existing instrumentation only fires when a skill calls one of its generated scripts (scripts/search_docs.mjs, scripts/validate.mjs) or when the bundled MCP server (shopify-dev-mcp) is invoked. Skills that are pure SKILL.md prose — or skill loads where the agent reads the SKILL.md and follows the instructions without invoking a script — emit nothing.

The track-telemetry hook runs on every PostToolUse event in supported agents — and, on Claude Code, on UserPromptSubmit (see "What is reported") — and reports a skill_invocation event whenever the agent does one of:

1. Calls the host agent's Skill / skill tool with a Shopify AI Toolkit skill name (e.g. shopify-admin, shopify-storefront-graphql, shopify-liquid).
2. Reads a SKILL.md file from a recognized Shopify AI Toolkit install path.

Both branches end up in the same place: a POST to https://shopify.dev/mcp/usage whose shape matches the payload the existing instrumentation modules already send. Server-side, the same handler routes everything into monorail.

Coverage

The hook script (scripts/track-telemetry.sh, .ps1) is wired up in two places so it fires for both plugin installs and standalone skill installs:

Surface	Manifest / Config	Honored by
Plugin manifest	hooks.json	Claude Code
Plugin manifest	cursor-hooks.json	Cursor
Plugin manifest	copilot-hooks.json	GitHub Copilot CLI, VS Code Copilot
Skill frontmatter hooks: block	injected by generate-agent-skills.ts into every generated SKILL.md	Claude Code (only agent that supports skill-frontmatter hooks today)

The plugin manifests are auto-loaded when the user installs the plugin (e.g. gemini extensions install, /plugin install ...). The skill-frontmatter block is what fires when a user installs skills directly — e.g. npx skills add Shopify/shopify-ai-toolkit — on an agent that supports frontmatter hooks. Without the frontmatter block, those standalone installs emit no skill telemetry at all.

For Codex and Gemini, skill telemetry stays limited to whatever the generated scripts and MCP server self-report — they expose neither a plugin-manifest hook API nor frontmatter hook support.

Source labeling and downstream dedup

When the plugin is installed on Claude Code, both the plugin-manifest hook and the skill-frontmatter hook fire for the same Skill("shopify-admin") invocation. Each event is labeled so consumers can collapse duplicates after the fact:

• Plugin manifests prefix the hook command with SHOPIFY_AI_TOOLKIT_HOOK_SOURCE=plugin. The skill-frontmatter invocation leaves the variable unset and defaults to skill. Both values flow out as the hookSource field inside the body's parameters object.
• Each event also carries the agent's sessionId and toolUseId inside parameters, so consumers can dedup on (sessionId, toolUseId) and keep whichever source they prefer.

All three labels ride inside the body's parameters object — never as HTTP headers. The /mcp/usage handler at shopify.dev reads only a short allow-list of headers (X-Shopify-Surface, X-Shopify-Client-Name /-Version/-Model) into first-class monorail columns; the entire parameters object is JSON-stringified into a single monorail column, so anything inside it survives without a schema change. Headers outside the allow-list are silently dropped.

This is intentionally simpler than client-side dedup, which had to guess at install paths and was fragile across OSes and stale plugin caches. Downstream consumers see the truth (one tool call, N events with the same sessionId + toolUseId) and decide from there.

What is reported

POST https://shopify.dev/mcp/usage
X-Shopify-Surface: skills-hook
X-Shopify-Client-Name: claude-code | cursor | copilot-cli | vscode | vscode-insiders

{
  "tool": "skill_invocation",
  "parameters": {
    "skill": "shopify-admin",
    "skillVersion": "1.2.2",       // null when not recoverable from path
    "trigger": "skill-tool",        // or "skill-md-read"
    "client": "claude-code",
    "hookSource": "plugin",         // or "skill" (skill-frontmatter origin)
    "sessionId": "abc-123",         // null when client doesn't supply one
    "toolUseId": "toolu_abc123",    // null when client doesn't supply one
    "user_prompt": "fix the cart…"  // Claude Code only, ≤2000 chars; omitted otherwise
  },
  "result": "ok"
}

The hook does not report tool inputs, file contents, generated code, or other tool arguments. It does capture user_prompt on Claude Code, out-of-band: a UserPromptSubmit hook stashes the verbatim prompt to a per-session file under a per-user (per-uid) temp directory (${TMPDIR:-/tmp}/shopify-ai-toolkit-telemetry-$(id -u)/), written 0600 so the prompt stays owner-only even on a shared /tmp fallback — local only, never sent on its own. The PostToolUse path attaches it as user_prompt only when a Shopify skill actually activates. Prompts from sessions that never touch a Shopify skill are never transmitted, and OPT_OUT_INSTRUMENTATION=true disables the capture entirely. On other hosts (Cursor, Copilot) the hook carries no prompt — user_prompt there comes from the per-skill script surfaces (scripts/validate.mjs for skills with validation, scripts/log_skill_use.mjs for skills without). The hook always also supplies the dedup keys (sessionId + toolUseId).

What is not reported

• Tool calls against the shopify-dev-mcp MCP server (already self- reported by packages/dev-mcp/src/utils/instrumentation.ts).
• Invocations of scripts/search_docs.mjs or scripts/validate.mjs inside a skill folder (already self-reported by packages/shopify-dev-tools/src/agent-skills/scripts/instrumentation.ts).
• Any tool call whose target is not a Shopify AI Toolkit skill or SKILL.md — the hook fires for every PostToolUse but only emits when it can identify a Shopify-owned skill.

Opt-out

The hook honors the shared toolkit opt-out. Any one of these disables every telemetry surface at once — skill scripts (reportValidation()), MCP server tool calls (recordUsage()), and this hook:

• A user-level opt-out file: $XDG_CONFIG_HOME/shopify-ai-toolkit/opt-out, ~/.config/shopify-ai-toolkit/opt-out, ~/Library/Application Support/shopify-ai-toolkit/opt-out (macOS), or %APPDATA%\shopify-ai-toolkit\opt-out (Windows). An empty file is enough; false/0/no/off inside it means "not an opt-out". SHOPIFY_AI_TOOLKIT_OPT_OUT_FILE overrides the location.
• OPT_OUT_INSTRUMENTATION=true.
• DO_NOT_TRACK=1.

Hooks are the surface most exposed to the env-var gap: the host spawns them as short-lived subshells, and several hosts do not pass the user's exported environment through (Shopify-AI-Toolkit#32). The file is read from disk, so it is the signal that always applies — prefer it.

Resolution is monotone: any signal saying "opted out" wins, and nothing re-enables telemetry. A wrapper exporting OPT_OUT_INSTRUMENTATION=false cannot override the file.

Opting out also disables capture, not just transmission — no prompt is stashed to disk.

Canonical implementation and the contract these scripts mirror: packages/shopify-dev-tools/src/telemetry/opt-out.ts.

Failure semantics

The hook is required to never break the host tool call. The script:

• Exits 0 even on JSON parse errors, missing curl, or unreachable endpoint.
• Emits {"continue":true} on stdout on every code path.
• Sends the HTTP request in the background with a 5-second timeout so agent tool loops are never delayed by network latency.

Local testing

# Skill tool call (Claude Code) — plugin source
SHOPIFY_AI_TOOLKIT_HOOK_SOURCE=plugin \
echo '{"hook_event_name":"PostToolUse","tool_name":"Skill","tool_input":{"skill":"shopify-plugin:shopify-admin"},"session_id":"local","tool_use_id":"toolu_local"}' \
  | bash scripts/track-telemetry.sh

# Same call, skill-frontmatter source (default when the env var is unset).
echo '{"hook_event_name":"PostToolUse","tool_name":"Skill","tool_input":{"skill":"shopify-admin"},"session_id":"local","tool_use_id":"toolu_local"}' \
  | bash scripts/track-telemetry.sh

# SKILL.md read (VS Code)
echo '{"hook_event_name":"PostToolUse","tool_name":"read_file","tool_use_id":"x__vscode","tool_input":{"path":"/Users/me/.vscode/agent-plugins/github.com/Shopify/shopify-ai-toolkit/.github/plugins/shopify-ai-toolkit/skills/shopify-liquid/SKILL.md"}}' \
  | bash scripts/track-telemetry.sh

# Opt-out via env var (no network call)
echo '{"tool_name":"Skill","tool_input":{"skill":"shopify-admin"}}' \
  | OPT_OUT_INSTRUMENTATION=true bash scripts/track-telemetry.sh

# Opt-out via the on-disk file, with the environment scrubbed — this is the
# shape that hosts like Hermes and Codex exec actually produce.
sandbox=$(mktemp -d)
mkdir -p "$sandbox/.config/shopify-ai-toolkit" && touch "$sandbox/.config/shopify-ai-toolkit/opt-out"
echo '{"tool_name":"Skill","tool_input":{"skill":"shopify-admin"}}' \
  | env -i PATH="$PATH" HOME="$sandbox" SKILL_TELEMETRY_TEST_MODE=1 bash scripts/track-telemetry.sh
# Expect: only {"continue":true} — no [TEST_TELEMETRY_BODY] marker on stderr.

Override the endpoint for staging or local tests via SHOPIFY_MCP_USAGE_ENDPOINT (hook-only) or SHOPIFY_DEV_INSTRUMENTATION_URL (shared with packages/shopify-dev-tools/src/http/index.ts, used by the evals harness to black-hole telemetry — set this to redirect both hook and TS-side telemetry to the same target).

Automated tests

Run the bash test suite:

bash packages/plugins/hooks/test/track-telemetry-test.sh

Bash plus jq (the suite guards for jq up front and fails with a clear message if it's missing) — no bats. Runs on every PR (ubuntu-latest + macos-latest) via .github/workflows/test.yml, so sed-portability regressions are caught on both GNU sed (Linux) and BSD sed (macOS).

Both scripts support a hidden SKILL_TELEMETRY_TEST_MODE=1 env var that skips the network call and writes the would-be request to stderr as [TEST_TELEMETRY_ENDPOINT], [TEST_TELEMETRY_HEADER], and [TEST_TELEMETRY_BODY] markers — that's the surface both test suites assert on.

PowerShell

pwsh -NoProfile -File packages/plugins/hooks/test/track-telemetry-test.ps1

# Windows PowerShell 5.1 — the version that ships with Windows 10/11.
powershell -NoProfile -File packages\plugins\hooks\test\track-telemetry-test.ps1

This suite is deliberately self-contained: it needs only PowerShell and the hook script — no pnpm, Node, or monorepo install — so it runs on a bare Windows VM.

It does not run in CI. The test matrix has no Windows runner (shop/setup-javascript-action lacks Windows support), and the .ps1 is a separate implementation from the .sh, so the bash suite is not a substitute. Run this manually on Windows when changing either hook — especially the opt-out resolution (mirrored across both) and the detached sender (Test 14 is the only automated coverage of the real send path; it caught both the ThreadJob-dies-with-parent and the unquoted-spaced-temp-path bugs).

Mirror layout

This directory lives at packages/plugins/hooks/ in ai-toolkit-source and rsyncs to /hooks/ at the root of the public Shopify/Shopify-AI-Toolkit mirror — matching the layout each agent's plugin loader expects (${CLAUDE_PLUGIN_ROOT}/hooks/..., ${CURSOR_PLUGIN_ROOT}/hooks/..., ${PLUGIN_ROOT}/hooks/...).

The two script files (track-telemetry.sh, .ps1) are also copied into each generated skill's scripts/ directory by packages/shopify-dev-tools/scripts/generate-agent-skills.ts. The skill-frontmatter hook invokes $CLAUDE_PLUGIN_ROOT/scripts/track-telemetry.sh: Claude Code runs frontmatter-hook commands in the session cwd (not the skill dir) and exposes no skill-dir variable, but it sets $CLAUDE_PLUGIN_ROOT to the skill's own directory for a frontmatter hook, so the reference resolves on standalone installs regardless of cwd. The command is guarded with if [ -f ] so it is a silent no-op in plugin installs (where $CLAUDE_PLUGIN_ROOT is the plugin root and the plugin-manifest hook already reports telemetry).