ClickUp AI Benchmark

A head-to-head benchmark of two ways to give an LLM agent the same ClickUp capabilities:

• axi - clickup-axi, an agent-ergonomic CLI the model discovers on demand via --help / inline hints (its surface is delivered as one short skill doc, loaded only when needed).
• MCP - the official ClickUp MCP server, whose ~52 tool schemas are injected into context every turn.

The tool-delivery surface is the only independent variable. Model, prompt, backend (a real ClickUp workspace), task set, and grader are identical across both arms. The question is not "can the model use tools" but "which way of delivering the tools is leaner, cheaper, and more reliable."

Full write-up: the complete study - methodology, results, cost analysis, and honest limitations - is the paper docs/BENCHMARK.md. This README is the short version.

Results

Completed sweep: 38 grading-fair tasks x 5 repetitions on two models of different capability (Claude Haiku 4.5, Claude Sonnet 5), plus a single-repetition probe on Claude Opus 4.8 (directional; see the paper §4.6). One task (O4, reparent) is a capability MCP lacks entirely and is reported separately rather than scored (paper §6).

	Haiku 4.5	Sonnet 5	Opus 4.8 (1 rep)
Success - axi vs MCP	89.5% vs 81.1%	83.2% vs 91.6%	89.5% vs 92.1%
Significant?	no (tie)	no (tie)	tie (within n=1 noise)
Tool surface (axi/MCP)	0.33x	0.32x	0.26x
Cost / task, cold (axi/MCP)	0.43x (2.3x cheaper)	0.33x (3.0x cheaper)	0.30x (3.3x cheaper)
Wrong changes (axi vs MCP)	11 vs 26	15 vs 16	2 vs 2

Takeaways.

• Task success is a statistical tie on both 5-rep models. axi leads nominally on the weak model, MCP on the strong one, but every confidence interval includes zero. The Opus probe is consistent (a one-task difference on 38).
• The durable, significant wins are structural: axi carries a ~3x smaller tool surface and costs 2.3-3.3x less per task, and the advantage widens as an agent's ClickUp use becomes sporadic - up to ~13x per request when only 5% of calls touch ClickUp, because MCP's surface stays resident on every request while axi's is disclosed on demand (paper §5.3).
• A weak-model safety edge did not survive. On Haiku, MCP made ~2.4x more wrong changes; by Sonnet and Opus the wrong-change counts are even. Reported as such.
• axi's honest gaps (search recall/default scope, content-generation reticence on the strong model) are named precisely in paper §4.4 and are the v1.1 backlog.

How it works

The two arms

Both arms share one base system prompt and one turn budget; they differ only in how the ClickUp tools reach the model. Isolation is the core validity control - each arm sees exactly one tool surface (verified via the SDK, not the model's self-report):

	axi arm	MCP arm
Surface	Bash + the clickup-axi skill doc in the system prompt	the official ClickUp MCP, tool schemas auto-injected
Ops	agent shells out: clickup-axi ...	agent calls mcp__clickup__*
SDK options	setting_sources=[], tools=["Bash"], strict_mcp_config=True	setting_sources=[], tools=[], mcp_servers={clickup}

setting_sources=[] and an explicit tools=[...] are both required: allowed_tools is only a permission allowlist and does not remove tools from the model's view, and the default settings would load ambient skills / MCP servers.

Grading (two tiers)

• Deterministic state assertions are authoritative and can never be overridden: read live ClickUp state after each run (status equals, comment added / contains, my-open count, and - for ambiguity tasks - no wrong mutation). Retry-storm detection caps tool calls where relevant.
• A blind LLM judge (one fixed strong model, never told which arm produced the answer) grades answer quality 0-10 against a per-task rubric on read/ambiguity tasks, where the correct wording is open and substring matching is brittle. Success = state passes and judge score >= threshold.
• Outcome taxonomy. A failed action is labeled wrong_result (a wrong mutation happened) or deferred (the agent asked/declined, changing nothing) - so caution is not conflated with harm (paper §4.3).

Tasks (39, across 9 categories)

Prompts reference targets by topic, never by id, so the agent must find them.

• read (5) - status lookup, my-open count, comment summary, list-by-status, deep description read.
• single mutation (4) - set status, add comment, mark done, multi-line comment.
• multi-step (4) - find-read-move, read-then-reply, move + comment, triage-then-act.
• recovery (2) - misspelled status (recover from echoed options); non-existent task (report absence, no retry storm).
• ambiguity / judgment (6) - two matches (ask, don't guess); already done (no-op); not your task (decline); phantom duplicate (report none); "most security-critical?" (judgment); conditional if-open-then-act.
• create (3) - plain create; create with priority + due + assignee + description in one shot; create a subtask under a topic-referenced parent.
• fields (6) - priority, absolute due date, relative due date ("three days from today"), tag add + remove, append-to-description without clobbering, hand-off to a teammate.
• organize (5) - move to another list (status preserved), move into a list that lacks the task's status (recovery), guarded close, re-parent, subtree read.
• discovery (4) - list spaces, lists in a space, my tasks with due dates + priorities, a teammate's open tasks in a space.

The rationale for the split, and every deliberate quirk, is written up in the paper (§3.2).

Metrics & statistics

Per run: success, input/output/cache tokens, a cache-independent tool-surface size, tool_calls, assistant_turns, wall-clock, and USD cost re-priced from a fixed sheet under the cold / warm cache bounds (cache state affects only cost, never behavior). Reported per (model, arm) with a two-stage cluster bootstrap 95% CI, pass^k reliability, between-query SE, and a paired bootstrap for the axi-vs-MCP delta. The always-on tax model (cost on requests unrelated to ClickUp) is in the paper §5.3.

Setup

Requires Python >= 3.10 and a ClickUp workspace you can seed into. A free ClickUp account is enough - the sandbox was built and verified on the free plan. The only free-plan effects are the 5-space cap (the sandbox scatters its noise across lists instead) and tighter MCP rate limits (the runner is resume-safe and spreads the sweep across days to stay within them).

pip install -e .
cp .env.example .env      # then fill it in (see below)

Fill .env:

• ANTHROPIC_API_KEY - for the agent runs and the LLM judge.
• CLICKUP_API_KEY - a ClickUp personal token (pk_...) for the REST seed/reset/grade lifecycle. (This is not the MCP auth.)
• CLICKUP_TEAM_ID, CLICKUP_SANDBOX_SPACE_ID, CLICKUP_SANDBOX_LIST_ID - read these ids from your own workspace/space/list URLs.

One-time prerequisites:

1. Primary space statuses. Custom statuses cannot be created via the REST API, so configure the primary space in the ClickUp UI with: to do -> in progress -> review -> blocked -> complete. clickup-bench-seed asserts they exist and fails with instructions otherwise.
2. Official MCP OAuth (MCP arm). Authorize once; the token caches under ~/.mcp-auth and is reused headlessly thereafter:

   claude mcp add --transport http clickup https://mcp.clickup.com/mcp
   # then in a Claude Code session: /mcp -> authorize `clickup` in the browser
   

3. clickup-axi (axi arm). Install clickup-axi (>= v1.0.0, for move/close and the session-hook context payload). Its SKILL.md is injected into the axi arm; override the path with CLICKUP_AXI_SKILL_PATH if it lives outside ~/.agents/skills/clickup-axi/SKILL.md, and the binary with CLICKUP_AXI_BIN.

Usage

clickup-bench-seed          # build the sandbox + write the manifest (once)
clickup-bench-run           # run the sweep (resume-safe; stop/resume any time)
clickup-bench-report        # text report: success, CIs, pass^k, cost bounds
clickup-bench-present       # standalone HTML report with charts
clickup-bench-seed --teardown   # delete everything the manifest created

Each also runs without installing, e.g. python -m clickup_ai_benchmark.runtime.runner. Useful flags:

clickup-bench-run --arms axi --models claude-haiku-4-5-20251001 --tasks R1,M1
clickup-bench-run --limit 5 --dry-run     # preview the next pending cells
clickup-bench-run --no-judge              # deterministic grading only
clickup-bench-report --json-out results/report.json --csv-out results/runs.csv

The runner is resume-safe: each completed cell is appended to results/runs.jsonl with a key, and a re-run skips cells already present. Runs are strictly sequential (every cell mutates shared ClickUp state and is isolated by a reset), which also spreads the sweep across days to live within MCP rate limits. To reproduce this repo's figures from the published data:

python -m clickup_ai_benchmark.analysis.report                    # the numbers
python -m clickup_ai_benchmark.analysis.charts --out docs/assets  # the 8 SVGs

Cost to run

Anthropic API spend only - ClickUp REST is free and the MCP calls are free (just rate-limited on the free plan). Measured from this repo's actual sweep:

Scope	Runs	~Cost (API)
Haiku only, 5 reps	390	~$4
Default: Haiku + Sonnet, 5 reps	780	~$12
+ Opus 4.8 probe, 1 rep	+78	~$4

The default sweep is Haiku + Sonnet 5 (config.MODELS) - weak + strong-modern anchors. Opus 4.8 is opt-in (run it with --models claude-opus-4-8 --arms axi,mcp --reps 1 for a directional third point). The fixed Sonnet judge adds ~$2 across a full sweep. Calibrate with a single cell first: clickup-bench-run --limit 1.

Layout

clickup_ai_benchmark/
  config.py            # env-derived secrets, sandbox ids, sweep params, RESULTS_DIR
  clickup.py           # thin ClickUp REST v2 client
  sandbox/             # spec, manifest, build/teardown (seed), per-run reset
  grading/             # task catalog, deterministic graders, blind LLM judge
  runtime/             # the two arms, run metrics, resume-safe sweep runner
  analysis/            # bootstrap/pass^k report, SVG charts, cost model, HTML present
docs/
  BENCHMARK.md        # the full paper (read this)
  SPEC.md              # design spec (as-built)
  assets/              # the 8 figures, regenerated from results/runs.jsonl
results/
  runs.jsonl           # raw per-run records (published for reproducibility)

License

MIT - see LICENSE.