QA / VQA Benchmarks

This directory documents the lightweight ResearchHarness contract for question-answering benchmarks, including plain-text QA and multimodal VQA-style tasks.

Recommended Server Command

For ordinary QA/VQA benchmark runs, start the OpenAI-compatible synchronous API server with the QA benchmark role overlay and no wrappers:

python3 run_server.py \
  --api-runs-dir ./api_runs \
  --host 127.0.0.1 \
  --port 8686 \
  --role-prompt-file ./benchmarks/QA/role_prompt.md \
  --no-input-wrapper \
  --no-output-wrapper

For large benchmark batches, raise --max-concurrent-runs when local resources and backend API quota allow more simultaneous agent runs.

For strict-format QA benchmarks, wrapper passes are optional and should be enabled only when they match the benchmark contract:

python3 run_server.py \
  --api-runs-dir ./api_runs \
  --host 127.0.0.1 \
  --port 8686 \
  --role-prompt-file ./benchmarks/QA/role_prompt.md \
  --input-wrapper \
  --output-wrapper

In practice, --output-wrapper is often more useful than --input-wrapper because it can format the final answer without rewriting the original question. Use --input-wrapper only when input normalization is known to be safe for the benchmark.

By default, each request creates a fresh run directory:

./api_runs/
└── run_YYYYMMDD_HHMMSS_<random>/
    ├── agent_workspace/          # visible to the agent
    │   └── inputs/
    │       └── images/           # user-provided images, when present
    └── agent_trace/              # server-side trace and session state
        ├── api_trace.jsonl
        ├── trace_*.jsonl
        └── session_state_*.json

OpenAI Test Example

The example below is directly runnable after the server is started. It creates a local workspace and sends a complete QA prompt through the OpenAI SDK.

from pathlib import Path
from openai import OpenAI


workspace = Path("./workspace/qa_example").resolve()
workspace.mkdir(parents=True, exist_ok=True)

client = OpenAI(api_key="unused", base_url="http://127.0.0.1:8686/v1")

response = client.chat.completions.create(
    model="RH",
    messages=[
        {
            "role": "user",
            "content": (
                "Who introduced the Transformer architecture, and in what year "
                "was the paper 'Attention Is All You Need' published? "
                "Answer in one sentence."
            ),
        }
    ],
    extra_body={"workspace-root": str(workspace)},
)

print(response.choices[0].message.content)

If workspace-root is provided and points to an existing directory, that directory is the workspace for this request; RH does not create any run_.../ subdirectory inside it. If workspace-root is absent, relative, or not an existing directory, RH falls back to the default per-request agent_workspace/. The agent_trace/ directory is always created under --api-runs-dir/run_.../ for auditability. For custom workspaces, uploaded images are saved under inputs/images/ directly inside that workspace. Use exactly workspace-root; synonymous request fields such as workspace_root are rejected.

The input and output LLM wrappers are disabled by default in normal deployment mode:

• --input-wrapper / --no-input-wrapper controls the input normalization pass.
• --output-wrapper / --no-output-wrapper controls the final answer formatting pass.

To return the agent's direct final text, use the default QA deployment command without wrapper flags. Advanced deployments can manually combine role prompts and wrapper flags as needed.

Multimodal Input

For image benchmarks, send OpenAI-style content parts. The first API version supports one or more data:image/...;base64,... URLs in the same request.

import base64
from io import BytesIO
from pathlib import Path
from PIL import Image, ImageDraw
from openai import OpenAI


image = Image.new("RGB", (320, 120), "white")
draw = ImageDraw.Draw(image)
draw.text((40, 45), "7 + 5 = ?", fill="black")
buffer = BytesIO()
image.save(buffer, format="PNG")
data_url = "data:image/png;base64," + base64.b64encode(buffer.getvalue()).decode("ascii")

workspace = Path("./workspace/qa_vqa_example").resolve()
workspace.mkdir(parents=True, exist_ok=True)

client = OpenAI(api_key="unused", base_url="http://127.0.0.1:8686/v1")

response = client.chat.completions.create(
    model="RH",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": (
                        "The image contains a simple arithmetic expression. "
                        "Return JSON with exactly two keys: expression and answer."
                    ),
                },
                {"type": "image_url", "image_url": {"url": data_url}},
            ],
        }
    ],
    extra_body={"workspace-root": str(workspace)},
)

print(response.choices[0].message.content)

Use RH or omit model for the server's default MODEL_NAME. Use RH--<llm-model-name> with exactly two hyphens for a per-request backend model override. Direct model names such as gpt-5.5 are rejected so benchmark runners do not accidentally confuse the ResearchHarness endpoint label with the backend LLM selection.

The API saves each submitted image inside the selected workspace, passes the image content to the first ResearchHarness model call when the backend model supports image parts, and includes each saved path in the agent-visible text. With the default workspace this is agent_workspace/inputs/images/; with a custom workspace-root, this is inputs/images/ inside that workspace.

The returned answer should be self-contained for a remote evaluator. Workspace files may support the run, but the response should not only say to consult answer.md, report.md, an image file, or another local artifact.

Scope

• The endpoint is synchronous and returns one final text answer.
• Each request gets a separate workspace subdirectory.
• QA benchmark mode can use the ResearchHarness agent directly, or optionally add input/output wrappers when the benchmark contract benefits from them.
• Streaming, async run status, artifact download, and remote image fetching are intentionally out of scope for this minimal QA contract.