Benchmark research workspace

August 16, 2026 ยท View on GitHub

This is LoopX's active workspace for benchmark practice and narrow runner examples. Its authority is the Long-Horizon Harness Benchmark and Research Program v0.

The workspace follows three rules:

  1. benchmark-native task, runner, verifier, and score semantics remain authoritative;
  2. current real-run practice outranks legacy LoopX benchmark abstractions;
  3. only generalized code and public-safe conclusions enter this directory.

Reusable product policy stays in benchmark-toolkit. The toolkit owns provider-neutral permission, artifact, and integrity boundaries. This directory may contain thin examples and practice notes, but it is not installed as a second LoopX Python package and does not grant execution or publication authority.

Current work

  • deepswe/README.md records the current public-safe DeepSWE method: frozen selection, matched-arm authority, native Goal proof, independent verification, invalid-run replacement, and compact evidence.
  • native_codex_goal.py is a compatibility import for the benchmark toolkit's installed native Goal runtime. The runnable deepswe/run_native_codex_goal.py example connects that runtime to a real codex app-server. Benchmark-family adapters should import the installed runtime and its formal isolated profile helper, then retain only their isolation, environment bridge, verifier, and scoring concerns. The profile helper renders the real Goal prompt with its installed CLI and proves that the prompt, discovered skills, and release-snapshot CLI belong to one pinned product path.

Legacy runners and dated packets are archived under deprecate/benchmark-legacy/. They are candidate evidence only, not the architecture for new work.