ExLLaMA: Guardrails and Fix Patterns

🧭 Quick Return to Map

You are in a sub-page of LocalDeploy_Inference.
To reorient, go back here:

• LocalDeploy_Inference — on-prem deployment and model inference
• WFGY Global Fix Map — main Emergency Room, 300+ structured fixes
• WFGY Problem Map 1.0 — 16 reproducible failure modes

Think of this page as a desk within a ward.
If you need the full triage and all prescriptions, return to the Emergency Room lobby.

ExLLaMA (and its fork ExLLaMA2/ExLLaMA-HF) is a highly optimized CUDA inference backend used under TextGen WebUI and custom pipelines. It can run very large models (65B+) on limited VRAM, but often shows instability when sharded, quantized, or paired with retrieval layers. This guide stabilizes ExLLaMA with structural guardrails.

---

Open these first

• Visual recovery map: RAG Architecture & Recovery
• Retrieval and eval knobs: Retrieval Playbook
• Boot and ordering: bootstrap-ordering.md, deployment-deadlock.md, predeploy-collapse.md
• Snippet and trace schema: retrieval-traceability.md, data-contracts.md

---

Core acceptance

• ΔS(question, retrieved) ≤ 0.45
• Coverage ≥ 0.70 against anchor snippet
• λ convergent across 3 paraphrases × 2 seeds
• E_resonance flat across quantization modes (int4, int8)

---

Common ExLLaMA breakpoints

Symptom	Cause	Fix
First run slower or unstable than warm cache	Lazy CUDA graph compile, missing warm-up fence	bootstrap-ordering.md
ΔS spikes when using quantized weights	Tokenizer drift vs chunked embeddings	embedding-vs-semantic.md, chunking-checklist.md
Memory corruption after long runs	Fragmented KV cache, no eviction strategy	context-drift.md, entropy-collapse.md
API or WebUI tool schema breaks	JSON schema not enforced at inference layer	prompt-injection.md, logic-collapse.md
Multi-shard mismatch on large models	Rank-order desync across GPUs	deployment-deadlock.md

---

Fix in 60 seconds

1. Always warm-up: run a 10-token dummy batch before production queries.
2. Schema lock: enforce snippet_id, section_id, tokens in every trace.
3. λ probe: measure stability under 2 quant modes (int4 vs int8).
4. Cache rotation: reset KV cache every N tokens (e.g., 8192) to prevent drift.
5. Verify: coverage ≥ 0.70, ΔS ≤ 0.45 across three paraphrase probes.

---

Diagnostic prompt (copy-paste)

I am running ExLLaMA backend with quant={mode}, shards={n}, extensions={list}.  
Question: "{user_question}"  

Please output:
- ΔS vs retrieved snippet
- λ over 3 paraphrases × 2 seeds
- Quantization impact (int4 vs int8)
- Cache stability (tokens until drift)
- Minimal WFGY fix page if ΔS ≥ 0.60

---

🔗 Quick-Start Downloads (60 sec)

Tool	Link	3-Step Setup
WFGY 1.0 PDF	Engine Paper	1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS)	TXTOS.txt	1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

---

Explore More

Layer	Page	What it’s for
⭐ Proof	WFGY Recognition Map	External citations, integrations, and ecosystem proof
⚙️ Engine	WFGY 1.0	Original PDF tension engine and early logic sketch (legacy reference)
⚙️ Engine	WFGY 2.0	Production tension kernel for RAG and agent systems
⚙️ Engine	WFGY 3.0	TXT based Singularity tension engine (131 S class set)
🗺️ Map	Problem Map 1.0	Flagship 16 problem RAG failure taxonomy and fix map
🗺️ Map	Problem Map 2.0	Global Debug Card for RAG and agent pipeline diagnosis
🗺️ Map	Problem Map 3.0	Global AI troubleshooting atlas and failure pattern map
🧰 App	TXT OS	.txt semantic OS with fast bootstrap
🧰 App	Blah Blah Blah	Abstract and paradox Q&A built on TXT OS
🧰 App	Blur Blur Blur	Text to image generation with semantic control
🏡 Onboarding	Starter Village	Guided entry point for new users

If this repository helped, starring it improves discovery so more builders can find the docs and tools.