omo-jevlike-router

September 18, 2026 · View on GitHub

A Jev-style one-pass skill router for OmO.

On every agent start, a small local model (frozen Qwen2.5-0.5B + a tiny option-attention head, jevlike) scores your entire skill catalog against the user's prompt in a single forward pass. An OmO extension then shrinks the <available_skills> block of the system prompt to the top-K skills before your expensive main model sees it.

Smaller prompts, fewer wasted skill reads, sub-100 ms routing on a warm cache — and fail-open: if the router is unreachable, OmO behaves exactly as before.

Inspired by TypeSafe's Jev "System One Model" (context + N options → one probability per option, no text generation).

user prompt ──▶ router (:8318, CPU Docker, one forward pass over N skills)
                    │ top-K (default 12)
                    ▼
        OmO extension rewrites <available_skills> in the system prompt
                    ▼
           main model (unchanged, e.g. via your CPA proxy)

What you get

From the deployment this package was extracted from (1,414 labeled turns, 141 skills, held-out test of 132 turns):

metricvaluebaseline
top-127.3%~29% majority class; 19.7% shuffled-context control
top-347.0%
recall@24 (top-K filter keeps the needed skill)84.1%recall@12 74.2%, recall@32 90.2%
ECE (calibration)~0.10
warm routing latency~50–70 msfirst full-catalog call ~9 s (option cache warm-up)

Measuring honestly: an earlier version of this README quoted 46.2% top-1 / 95.5% recall@12. That was an evaluation bug — the trainer's eval function iterated range(len(rows)) instead of the held-out row indices, i.e. it measured the first 132 training rows. The numbers above are from the fixed evaluator (analyze/eval_report.py independently agrees via the live HTTP path). v1 routing is weak on short prompts — which is exactly why the extension is lossless by design: skills cut from the top-K keep their names in an other_skill_names index, and flat-confidence turns skip filtering entirely, so no skill is ever hidden from the model.

Setup guide

0. Requirements

  • OmO (omo CLI) installed
  • Docker on any machine the OmO host can reach (same box is fine)
  • Python 3.10+ on the OmO host (only for the one-time dataset extraction)

1. Extract a training dataset from your own sessions

git clone https://github.com/islee23520/omo-jevlike-router
cd omo-jevlike-router

python3 extract/extract_dataset.py \
  --sessions ~/.omo/sessions \
  --skills   ~/.omo/skills \
  --out      server/data

This is read-only over your session logs. It produces server/data/{train,validation,test}.jsonl + catalog.json, split by session so related turns never leak across splits. Your data never leaves this directory.

2. Build and start the router (Docker)

cd server
docker compose up --build -d
docker compose logs -f        # first boot: downloads Qwen2.5-0.5B, trains, evals, serves

The container publishes 127.0.0.1:8318. To reach it from another machine (e.g. OmO on your Mac, router on a LAN box), set the bind address first:

ROUTER_BIND=0.0.0.0 docker compose up -d     # or a specific IP

Smoke test:

../scripts/smoke.sh                          # expects aside-browser-style top-1 on a browser task

3. Install the OmO extension

cp extension/jevlike-router.js ~/.omo/extensions/
# if the router is not on 127.0.0.1, tell the extension where it is:
#   add e.g. JEVLIKE_ROUTER_URL=http://192.168.1.10:8318 to the omo process env

Restart OmO. From the next prompt on, the skill catalog in your system prompt is filtered to the top-K (plus any skill you invoke explicitly with $skill:name / /skill:name / skill:name).

4. Tune or disable

env vardefaultmeaning
JEVLIKE_ROUTER_URLhttp://127.0.0.1:8318router base URL
JEVLIKE_ROUTER_TOPK24skills kept in the prompt
JEVLIKE_ROUTER_TIMEOUT_MS2500give up (fail-open) after this
JEVLIKE_ROUTER_OFF–set to 1 to disable entirely

5. Retrain later

The checkpoint lives in the jevlike-lab-data Docker volume. To retrain on a fresher dataset:

docker exec jevlike-lab-router python /app/train_router.py \
  --train /dataset/train.jsonl --validation /dataset/validation.jsonl \
  --test /dataset/test.jsonl --output /data/runs/router.pt
docker restart jevlike-lab-router

How it works

  • Extractor pairs each user message with the skill(s) actually read right after it in your session logs. Options are name: description strings from your live ~/.omo/skills catalog.
  • Trainer freezes a pretrained encoder and trains only a small option-attention head (jevlike architecture). Catalog option embeddings are computed once and cached, so training on ~1.4k rows takes ~2 minutes on CPU and serving a warm request takes tens of milliseconds.
  • Server exposes POST /score {context, options?} — omit options to score the trained catalog itself; pass options to score arbitrary lists.
  • Extension rewrites only the <available_skills>…</available_skills> block, keeps explicit invocations, and returns the prompt unchanged on any error.

Limitations (honest ones)

  • Routing quality is bounded by your data: ~1.4k labeled turns got top-1 46% / recall@12 95.5%. More history → better routing.
  • Label bias transfers: our most-read skill (aside-browser, 29% of labels) dominates short prompts. The top-K filter absorbs most of it.
  • Skills added after training are invisible to the router (they simply never rank; explicit invocation still works). Retrain after catalog changes.
  • The first full-catalog call after server boot embeds every option (~9 s on CPU); the extension times out fail-open on that turn and is warm from the next one.
  • This is an independent experiment, not a Jev reproduction (see the jevlike README for what that means).

Privacy

Your session logs and dataset stay local. This repository contains no data — server/data/ is gitignored by design. Don't commit it.

Credits

License

MIT — see LICENSE.