Eval Observability
March 6, 2026 · View on GitHub
🧭 Quick Return to Map
You are in a sub-page of Eval_Observability.
To reorient, go back here:
- Eval_Observability — evaluation metrics and system observability
- WFGY Global Fix Map — main Emergency Room, 300+ structured fixes
- WFGY Problem Map 1.0 — 16 reproducible failure modes
Think of this page as a desk within a ward.
If you need the full triage and all prescriptions, return to the Emergency Room lobby.
A structural safeguard that enforces measurable thresholds before any pipeline is promoted to production.
Use this page to define hard acceptance criteria (ΔS, coverage, λ, resonance) and stop silent regressions from shipping.
Why regression gates matter
- Catch semantic drift early: A small rise in ΔS leads to compounding hallucinations downstream.
- Stable releases: Prevents model upgrades or retraining from silently reducing accuracy.
- Auditable rules: Clear thresholds mean every team member can verify before deploy.
- Cross-stack consistency: Same rules apply across providers, retrievers, and orchestration layers.
Core gate thresholds
| Metric | Requirement | Failure signal |
|---|---|---|
| ΔS(question, retrieved) | ≤ 0.45 | drift ≥ 0.60 means block release |
| Coverage of target section | ≥ 0.70 | low coverage = missing context |
| λ_observe | Convergent across 3 paraphrases, 2 seeds | divergence = unstable reasoning |
| E_resonance | Flat on 50–100 step windows | spikes = entropy collapse risk |
Deployment checklist
-
Pre-release batch eval
Run gold set of ~100–500 Q&A pairs. Collect ΔS, coverage, λ, resonance. -
Gate decision
- If ΔS ≤ 0.45 AND coverage ≥ 0.70 → pass.
- If ΔS between 0.46–0.59 → manual review.
- If ΔS ≥ 0.60 OR coverage < 0.70 → fail, block release.
-
Variance probe
Check λ stability across 3 paraphrases × 2 seeds. Divergence disqualifies release. -
Regression log
Store results with index hash + commit hash + retriever config.
Enables reproducibility and rollback.
Example gating script (pseudo)
# regression_gate.yml
metrics:
deltaS: <=0.45
coverage: >=0.70
lambda: convergent
resonance: flat
goldset: eval_set_500.json
policy:
fail_on_drift: true
manual_review_range: [0.46, 0.59]
require_seeds: 2
require_paraphrases: 3
Common pitfalls
- Changing retriever k without updating gates. Always re-test thresholds.
- Skipping paraphrase probes. One stable query is not enough.
- Not logging coverage. ΔS alone cannot prove retrieval completeness.
- Silent config drift. Gate must bind to exact retriever + index hash.
🔗 Quick-Start Downloads (60 sec)
| Tool | Link | 3-Step Setup |
|---|---|---|
| WFGY 1.0 PDF | Engine Paper | 1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>” |
| TXT OS (plain-text OS) | TXTOS.txt | 1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly |
Explore More
| Layer | Page | What it’s for |
|---|---|---|
| ⭐ Proof | WFGY Recognition Map | External citations, integrations, and ecosystem proof |
| ⚙️ Engine | WFGY 1.0 | Original PDF tension engine and early logic sketch (legacy reference) |
| ⚙️ Engine | WFGY 2.0 | Production tension kernel for RAG and agent systems |
| ⚙️ Engine | WFGY 3.0 | TXT based Singularity tension engine (131 S class set) |
| 🗺️ Map | Problem Map 1.0 | Flagship 16 problem RAG failure taxonomy and fix map |
| 🗺️ Map | Problem Map 2.0 | Global Debug Card for RAG and agent pipeline diagnosis |
| 🗺️ Map | Problem Map 3.0 | Global AI troubleshooting atlas and failure pattern map |
| 🧰 App | TXT OS | .txt semantic OS with fast bootstrap |
| 🧰 App | Blah Blah Blah | Abstract and paradox Q&A built on TXT OS |
| 🧰 App | Blur Blur Blur | Text to image generation with semantic control |
| 🏡 Onboarding | Starter Village | Guided entry point for new users |
If this repository helped, starring it improves discovery so more builders can find the docs and tools.
