Harbor Task Format

May 20, 2026 · View on GitHub

This document describes the Harbor task format used by LiveClawBench and the hybrid evaluation rubric design.

Task Directory Structure

Each task follows this standardised directory layout:

<task_name>/
├── task.toml               # Task metadata & resource config
├── instruction.md          # Agent-facing task description
├── environment/
│   └── Dockerfile          # Build environment
├── solution/
│   └── solve.sh            # Reference solution
└── tests/
    └── test.sh             # Verification script

File responsibilities:

task.toml — declares task metadata (difficulty, domain, complexity factors) and resource config (CPU, memory, timeouts)
instruction.md — the task description shown to the agent, simulating a user's natural language request
Dockerfile — builds the task runtime environment on top of liveclawbench-{task}-base:latest (the per-task image layer), including app dependency installation and database init. Do not add a task-local ENTRYPOINT or copy startup.sh into writable paths — services are started by the per-task image's /opt/mock/entrypoint.sh from the read-only /opt/mock/startup.d/{task}.sh
solve.sh — reference solution script used to verify task solvability (not exposed to the agent)
test.sh — verification entry point; scoring files vary by task (see Evaluation Patterns below)

task.toml Template

version = "1.0"

[metadata]
difficulty = "medium"          # easy | medium | hard
category = "open-world"
tags = ["e-commerce_daily_svcs", "communication_email"]

domain = "E-commerce & Daily Svcs"
domains_multi = ["E-commerce & Daily Svcs", "Communication & Email"]

# Triple-Axis Complexity Factors (0 = absent, 1 = present)
factor_a1 = 1   # A1: Cross-Service Dependency
factor_a2 = 0   # A2: Contaminated Initial State
factor_b1 = 0   # B1: Implicit Goal Resolution
factor_b2 = 0   # B2: Knowledge System Maintenance

case_id = 99    # Unique integer across all tasks (check ../../metadata/cases_registry.csv)

[verifier]
timeout_sec = 900.0

[agent]
timeout_sec = 1800.0

[environment]
build_timeout_sec = 600.0
cpus = 2
memory_mb = 4096
storage_mb = 10240
allow_internet = true   # required if the agent needs LLM API access

Key fields:

Field	Description
`case_id`	Unique integer identifier; check `../../metadata/cases_registry.csv` before assigning
`domain`	Primary task domain (see Available Domains below)
`domains_multi`	All domains the task touches, including primary
`factor_a1` .. `factor_b2`	Complexity factor flags per the Triple-Axis Framework
`verifier.timeout_sec`	Maximum execution time for the verification script
`agent.timeout_sec`	Maximum time the agent has to complete the task
`environment.build_timeout_sec`	Docker environment build timeout
`allow_internet`	Set to `true` if the agent must call external LLM APIs

Available Domains

LiveClawBench tasks are classified into the following primary domains:

Domain	Description	Example Tasks
Documents & Knowledge	Skill/knowledge repository management	skill-creation, skill-conflict-resolution
Communication & Email	Email composition and management	email-writing, email-reply
E-commerce & Daily Svcs	Shopping, orders, daily services	watch-shop, flight-booking
Calendar & Task Mgmt	Scheduling and task coordination	schedule-change-request, flight-info-change-notice
Coding & Software Dev	Full-stack development tasks	blog-site-from-scratch, blog-site-completion-from-starter
DevOps & Env Repair	Build fixes and environment repair	vue-build-fix-single, vue-build-fix-chain
Deep Research & Report	Research synthesis and reporting	noise-filtering, live-web-research-sqlite-fts5
Health & Fitness	Diet and health tracking	mint-diet-snack-log
Social Media	Social platform interactions	social-media-posting, social-unlike-post
Finance & Data Analytics	Expense and financial data management	expense-draft-delete
Health & Wellness	Health records and wellness tracking	health-daily-record

Complexity factor fields (set to 1 when the factor applies, 0 when absent):

Field	Factor
`factor_a1`	A1 — Cross-Service Dependency
`factor_a2`	A2 — Contaminated Initial State
`factor_b1`	B1 — Implicit Goal Resolution
`factor_b2`	B2 — Knowledge System Maintenance

Hybrid Evaluation Rubric

LiveClawBench uses an outcome-driven hybrid evaluation strategy that balances determinism with flexibility.

Rule-based verification (test.sh):

The verification script checks environment state after task completion:

Database state — query SQLite databases to verify expected records exist (orders, emails, schedules, etc.)
File contents — check generated files for expected content (skill files, code, reports, etc.)
API responses — call mock service APIs to verify service state matches expectations

Outcome-driven principle:

The verifier checks final state, not the agent's action sequence
Agents can achieve the goal via any strategy: direct API calls, web UI interaction, scripting, etc.
This ensures evaluation reflects genuine agent capability rather than path memorisation

Partial credit:

For multi-step tasks, the verifier decomposes the task into independent checkpoints
Each checkpoint is scored independently; completing partial steps yields partial credit
Example: the flight-info-change-notice task has three checkpoints — "identify change email", "find affected schedule", "send notification"
This provides fine-grained capability measurement and avoids all-or-nothing scoring

Evaluation Patterns

LiveClawBench tasks use one of three evaluation patterns, each suited to different verification needs:

Pattern	Files in `tests/`	Score Source	Used By
verify.py	`test.sh` + `verify.py`	`Score: X.X/1.0`	E-commerce, email, flight, calendar, blog, vue tasks (18)
evaluate.py	`test.sh` + `evaluate.py` + `run_benchmark.sh` [+ `reference/`]	`TOTAL SCORE: X / 100` → normalized to 0.0–1.0	skill-* tasks (5)
LLM judge	`test.sh` + `deterministic_checks.py` + `llm_judge.py` + `answer_key.json` + `rubric.json`	Structured JSON → `reward.txt`	Research/complex tasks (6)

All patterns ultimately write a scalar score to /logs/verifier/reward.txt. The verify.py pattern is recommended for new tasks (see Adding Tasks for the full contract).

LLM-judge implementation contract

Tasks using the LLM judge pattern must follow these conventions.

Credential variables — passed via --ee at run time (not --ae; see Running Tasks for the distinction):

Variable	Required	Default
`JUDGE_BASE_URL`	Yes	—
`JUDGE_API_KEY`	Yes	—
`JUDGE_MODEL_ID`	No	`deepseek-v3.2`

llm_judge.py must raise RuntimeError immediately if JUDGE_BASE_URL or JUDGE_API_KEY is unset. No hardcoded fallback URLs or model names are permitted — silent fallbacks mask misconfiguration during evaluation.

Variable naming — use the JUDGE_* prefix, uppercase, with no provider-specific prefix (not OPENCLAW_ARK_API_KEY or similar). This keeps the verifier interface provider-agnostic.

Output path — llm_judge.py must write reward files directly to /logs/verifier/:

import json, pathlib

verifier_dir = pathlib.Path("/logs/verifier")
verifier_dir.mkdir(parents=True, exist_ok=True)
(verifier_dir / "reward.json").write_text(json.dumps(reward_json, indent=2))
(verifier_dir / "reward.txt").write_text(str(final_score))

Do not write to ~/.openclaw/reward.* and rely on test.sh to copy — that intermediate step is unnecessary and creates two inconsistent copies. /logs/verifier/ is created by Harbor before test.sh runs (Stage 5), so llm_judge.py called from test.sh (Stage 6) can always write there directly.

Note: existing PKB tasks (noise-filtering, conflict-repair-acb, mixed-tool-memory, incremental-update-ctp, live-web-research-sqlite-fts5) still use the ~/.openclaw/reward.* → cp pattern. They will be refactored in a future cleanup; new tasks must follow this spec.

HTTP client — current implementations use urllib.request (zero external dependencies). A future improvement is to replace this with LiteLLM, which provides a unified interface across OpenAI-compatible, Anthropic, Gemini, and other provider APIs. Priority: OpenAI-compatible endpoints first; other formats can be validated incrementally.

`reward.json` Structure

Tasks that produce sub-dimension scores write /logs/verifier/reward.json alongside reward.txt. Two rules apply universally; everything else is task-type specific:

Rule	Description
`reward` is mandatory	The canonical aggregate score — `float ∈ [0.0, 1.0]`, normalized weighted sum of all sub-dimensions. Harbor uses this key for dataset-level metrics.
`_meta_` prefix for non-float fields	Any string or nested-object field (rationales, model names, mode flags) must carry the `_meta_` prefix. Harbor tracks all `float \| int` keys in `reward_stats`; un-prefixed string values corrupt dataset-level aggregation.

All other float | int keys are unrestricted and task-type specific (e.g. answer_accuracy, contract_valid, db_integrity). Harbor tracks every numeric key independently in reward_stats; aggregate reward via weights declared in rubric.json.

Minimal example:

{
  "contract_valid":   1.0,
  "answer_accuracy":  0.75,
  "_meta_rationale":  "The agent correctly identified ...",
  "_meta_judge_model": "kimi-k2.5",
  "reward":           0.80
}

reward.txt must contain exactly the value of reward:

0.8