JEV + DSPy Typed Control Plane

A benchmark-first experiment for one concrete question:

Can a prefill-only or otherwise constrained classifier act as a typed control plane around a probabilistic DSPy agent?

The project routes free-form banking-support cases into a closed ontology, applies deterministic confidence and safety rules, permits only allow-listed tools, and lets DSPy draft the customer-facing response after the route and action are fixed.

This is an engineering experiment, not a production banking system. The tools are deterministic mocks; no real accounts, payments, cards, or customer data are accessed.

What it compares

Pipeline	Classifier	Control plane	Response layer
jev	OpenJEV System One, OpenJEV Hugging Face NLI, or offline heuristic	Typed deterministic rules	Template or optional DSPy
dspy	DSPy typed signature	Same typed rules	DSPy when configured
json_schema	OpenAI-compatible strict JSON Schema	Same typed rules	Template or optional DSPy

The same ontology, dataset, state machine, tool allow-list, and metrics are used for all three paths. That makes the benchmark a comparison of classifier/control behavior rather than three unrelated demos.

Architecture

Customer text
     │
     ▼
┌──────────────────────────────────────────────────────────────────┐
│ Classifier adapter                                                │
│ OpenJEV System One | OpenJEV NLI | DSPy | JSON Schema | heuristic│
└───────────────────────────────┬──────────────────────────────────┘
                                │ Classification
                                ▼
┌──────────────────────────────────────────────────────────────────┐
│ Typed control plane                                               │
│ ontology validation → security override → confidence thresholds   │
│ → state transition validation → tool allow-list                   │
└───────────────────────────────┬──────────────────────────────────┘
                                │ Fixed route + fixed action
                  ┌─────────────┴─────────────┐
                  ▼                           ▼
          Guarded mock tools          DSPy/template response
                  └─────────────┬─────────────┘
                                ▼
                     Typed pipeline result

The key boundary is intentional: an LLM may explain a decision, but it cannot add a route, change the selected action, or invoke an unapproved tool.

Banking ontology

The first release contains ten top-level intents with 39 sub-intents:

• UPI transfer, card transaction, ATM cash, account access, KYC update
• Bank transfer, cash deposit, loan EMI, term deposit, fraud/security

Unknown combinations cannot silently pass validation. Low-confidence outputs abstain or request clarification, while fraud/security signals force a human-security path even if the classifier predicts a routine intent.

Quick start: zero-download smoke benchmark

Python 3.11 or newer is required.

python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'

jev-control generate-data --output data/eval.jsonl
jev-control benchmark \
  --dataset data/eval.jsonl \
  --pipeline all \
  --output artifacts/offline-smoke.json

This works without model weights or API keys. The jev path uses the deterministic heuristic backend, while unconfigured dspy and json_schema paths use explicitly labelled contract stubs. They prove that all three pipeline envelopes, metrics, and report generation work; their quality numbers are not a model comparison. The synthetic generator and heuristic intentionally share route vocabulary, so a high offline score is only a regression signal for plumbing and guardrails—not evidence of model generalization.

Classify one case:

jev-control classify \
  'UPI was debited but the receiver did not get it' \
  --backend heuristic

Machine-readable output:

jev-control classify \
  'I see a transaction that is not mine' \
  --backend heuristic \
  --json

Run with real backends

Copy the environment template first:

cp .env.example .env
set -a; source .env; set +a

1. Local OpenJEV System One server

The System One adapter posts the customer state and the complete route map to /v1/systemone as one choice question.

export JEV_CP_OPENJEV_URL=http://127.0.0.1:8000
export JEV_CP_OPENJEV_API_KEY=

jev-control benchmark \
  --dataset data/eval.jsonl \
  --pipeline jev \
  --jev-backend systemone \
  --output artifacts/openjev-systemone.json

This is the most practical Apple-silicon path when using an MLX-based OpenJEV server: the model stays loaded once, while this project remains a thin benchmark/control client.

2. Hugging Face OpenJEV NLI model

python -m pip install -e '.[hf]'
export JEV_CP_HF_MODEL_ID=AlexWortega/openjev
export JEV_CP_HF_SUBFOLDER=qwen3.5-4b-nli
export JEV_CP_HF_BATCH_SIZE=8

jev-control benchmark \
  --dataset data/eval.jsonl \
  --pipeline jev \
  --jev-backend hf \
  --output artifacts/openjev-hf.json

The adapter scores every allowed route using the entailment logit and normalizes those scores across the closed route set. It performs no response decoding. Route pairs are processed in configurable batches to avoid loading all 39 sequences into one forward pass. On Apple Silicon it chooses MPS when available; CUDA is preferred on compatible Linux systems, otherwise CPU is used.

The model is large enough that the first download and load can be substantial. Start with --limit 10, then run the complete dataset after confirming memory and latency.

3. DSPy classifier and response drafter

python -m pip install -e '.[dspy]'
export JEV_CP_DSPY_MODEL='openai/your-model'
export JEV_CP_LM_API_KEY='your-key'
# Optional for an OpenAI-compatible provider:
export JEV_CP_LM_BASE_URL='https://your-provider.example/v1'

jev-control benchmark \
  --dataset data/eval.jsonl \
  --pipeline dspy \
  --output artifacts/dspy.json

Use a JEV classifier with DSPy only for the final wording:

jev-control benchmark \
  --dataset data/eval.jsonl \
  --pipeline jev \
  --jev-backend systemone \
  --dspy-agent \
  --output artifacts/jev-plus-dspy.json

DSPy is deliberately kept downstream of the decision boundary in this hybrid mode. The DSPy program receives the fixed route, fixed action, policy reason, and tool outcome; the returned text cannot mutate those fields.

4. Direct strict JSON-schema baseline

The baseline calls an OpenAI-compatible /chat/completions endpoint that supports response_format.type=json_schema.

export JEV_CP_JSON_MODEL='your-model'
export JEV_CP_LM_BASE_URL='https://your-provider.example/v1'
export JEV_CP_LM_API_KEY='your-key'

jev-control benchmark \
  --dataset data/eval.jsonl \
  --pipeline json_schema \
  --output artifacts/json-schema.json

Provider support for strict JSON Schema varies. Unsupported or malformed output is converted to an abstention rather than coerced into the nearest route.

Interactive demo

python -m pip install -e '.[demo]'
streamlit run app.py

The UI shows the classifier output, deterministic control decision, state trace, tool result, and customer response separately so that model behavior and policy behavior remain inspectable.

Evaluation dataset

jev-control generate-data creates a deterministic synthetic JSONL suite with more than 150 labelled cases:

• clean phrasing
• typo-heavy variants
• Hindi/English and Tamil/English code-mixed variants
• prompt-injection text appended to valid requests
• deliberately ambiguous multi-intent cases that should abstain

Each row contains the expected intent/sub-intent, category, language, and abstention expectation. Replace or augment it with de-identified, human-labelled examples before drawing conclusions about production performance.

Metrics

Each report includes the ontology version, dataset SHA-256 fingerprint, stress-category counts, backend/agent identities, policy thresholds, and:

• intent and sub-intent accuracy
• invalid-output and policy-violation rates
• abstention, coverage, and selective accuracy
• Brier score and expected calibration error
• repeated-run consistency
• p50 and p95 end-to-end latency
• per-category breakdowns for clean, typo, code-mixed, adversarial, and ambiguous cases
• classifier-reported token cost when a backend exposes usage; response-agent token cost is not yet included

Accuracy alone is not enough for a control plane. A useful result should also show that the classifier abstains on unsupported cases, remains calibrated, produces stable decisions, and never escapes the tool allow-list.

Configuration

Variable	Purpose	Default
JEV_CP_OPENJEV_URL	System One server base URL	http://127.0.0.1:8000
JEV_CP_OPENJEV_API_KEY	Optional server bearer token	empty
JEV_CP_HF_MODEL_ID	Hugging Face NLI repository	AlexWortega/openjev
JEV_CP_HF_SUBFOLDER	Published 4B NLI checkpoint directory	qwen3.5-4b-nli
JEV_CP_HF_BATCH_SIZE	Route-pair batch size for memory control	8
JEV_CP_DSPY_MODEL	DSPy LM identifier	empty
JEV_CP_JSON_MODEL	JSON-schema baseline model	empty
JEV_CP_LM_BASE_URL	OpenAI-compatible provider base URL	empty
JEV_CP_LM_API_KEY	LM provider API key	empty
JEV_CP_AUTO_ROUTE_THRESHOLD	Automatic tool-routing threshold	0.75
JEV_CP_CLARIFY_THRESHOLD	Clarify/escalate boundary	0.55
JEV_CP_INPUT_COST_PER_MILLION	Input-token price for estimates	0
JEV_CP_OUTPUT_COST_PER_MILLION	Output-token price for estimates	0

Thresholds are configuration, not universal constants. Tune them on held-out data using the cost of false routing, false escalation, and customer friction.

Development

make setup
make check
make data
make benchmark

Or run the checks directly:

ruff check .
mypy src
pytest

Project structure:

src/jev_dspy_control_plane/
├── ontology.py             closed intent/sub-intent contract
├── control_plane.py        thresholds, safety override, state graph
├── pipelines.py            end-to-end orchestration
├── tools.py                guarded deterministic mock tools
├── agents.py               deterministic response baseline
├── dspy_agent.py           constrained DSPy response drafter
├── classifiers/            heuristic, System One, HF NLI, DSPy, JSON
└── evals/                  dataset, records, metrics, runner

Interpreting the experiment

A positive result is not merely “JEV has higher accuracy.” The stronger hypothesis is that a constrained scorer provides a useful systems boundary:

1. the route space is explicit and carries an ontology version;
2. every candidate is scored against the same customer state;
3. confidence can drive deterministic abstention;
4. only typed states may reach tools;
5. DSPy is free to optimize explanations without owning authorization.

A negative result is also valuable. If an NLI/option scorer is poorly calibrated on banking language, misses multilingual cases, or becomes expensive as the route set grows, the report should expose that. The next step would then be labelled-data calibration, a hierarchical route scorer, or a trained head—not hiding the failure behind a more persuasive chat response.

Research snapshot

The repository includes docs/research-notes.md, with the public implementations reviewed, their GitHub activity snapshot, and the design ideas incorporated here. The user's private TweetSmash/X bookmarks were not available through the connected tools, so the notes do not claim to have reviewed them.

Push the local repository to GitHub

This artifact is already a committed Git repository on branch feat/initial-scaffold. After choosing the target account and visibility:

gh repo create jev-dspy-control-plane --source . --private --push

Change --private to --public when appropriate.

License

MIT. See LICENSE.