README.md

English · 简体中文

Import once. Retrieve with confidence.

dsh-kb-manager is a local knowledge-base lifecycle plugin for DeepSeek Harness (dsh): multi-format import → CJK-aware smart chunking → SQLite-native vector + full-text indexes → hybrid search (vector KNN + BM25/FTS5 → RRF → optional rerank) → citation tracing — plus soft-delete + physical compaction, an async job system, import-path whitelisting with an SSRF guard, and per-KB audit.

Ask in natural language. The plugin exposes 22 Agent tools (async jobs + sync ops incl. rename_kb / compact_index / cancel_job / list_snapshots / configure) and an optional Web panel (KB list, create, detail, in-KB hybrid search, import with live job progress, rebuild, two-step delete, snapshot history) — no separate RAG server required.

Design spec: DESIGN.md (v2.1).

Why dsh-kb-manager?

Capability	What it changes
22 unified tools	async 6 (import_document / rebuild_index / compact_index / cancel_job / get_job + export/import jobs) + sync 16: create_kb / list_kbs / get_kb / rename_kb / delete_kb / list_documents / delete_document / search_kb / multi_kb_search / get_chunk / get_kb_stats / configure / create_snapshot / list_snapshots / restore_snapshot / export_kb / import_kb; every tool returns `{ ok, data
SQLite-native indexes	sqlite-vec KNN + FTS5 full-text (per-KB kb.db); no external vector-DB process.
CJK-aware tokenization	Intl.Segmenter (built-in, zero deps) splits Chinese / Japanese / Korean for BM25.
Hybrid retrieval	Vector + BM25 → RRF → optional rerank; rerank state is explicit: `applied
Async job system	Import / rebuild return a job_id immediately; get_job polls progress; crash recovery (JOB.INTERRUPTED) + cooperative cancellation.
Import-path whitelist	Only whitelisted paths (default: session workspace + <storage>/inbox/) or http(s) URLs are importable.
SSRF guard	URL imports reject private / loopback / cloud-metadata addresses.
Untrusted retrieval	Search results are always flagged untrusted: true — content is data, not instructions.
Soft-delete + compact	Deletes are millisecond soft-deletes; rebuild_index / compact reclaims space physically, no orphan vectors.
Two-step destructive ops	delete_kb requires a one-time confirm_token (bound to kb_id, 60 s TTL).
Audit trail	Per-KB audit table records every mutation.
Snapshots / export / cross-KB import	create_snapshot / list_snapshots / restore_snapshot (two-step confirm), export_kb (kbpack/JSON), import_kb (kbpack, hash-deduped merge); all exposed via remote-service for the Web panel and headless.

Architecture

flowchart LR
  A[Documents / URLs] --> B[Parse]
  B --> C[Chunk<br/>Intl.Segmenter CJK]
  C --> D[Embed<br/>bge-small-zh / mock]
  D --> E[Vector index<br/>sqlite-vec KNN]
  C --> F[FTS5 BM25]
  E --> G[Hybrid search]
  F --> G
  G --> H[RRF]
  H --> I[Optional rerank<br/>applied / fallback_rrf / disabled]
  I --> J[Citations + untrusted flag]
  E --> K[Soft-delete / compact_index]
  C --> K
  G --> L[Async job queue + audit]

Install

From GitHub

dsh plugin add github:xiaoshi7915/dsh-kb-manager

(Once listed in the awesome-dsh-plugin list, it can also be installed one-click from the dsh-market page.)

Build from source

git clone https://github.com/xiaoshi7915/dsh-kb-manager.git
cd dsh-kb-manager
npm install
npm run build
dsh plugin --profile web add .

Validate the composed profile, restart DSH, and refresh the Web UI:

dsh --profile web --dump-config
dsh web

Then try:

Create a knowledge base named "project-docs", import this PDF, and search for how authentication works. Show me the source chunk.

How it works

1. Create a KB (create_kb) with an embedding identity (default bge-small-zh-v1.5 / onnx / 512-dim; mock provider for dev/test).
2. Import a whitelisted local file or http(s) URL (import_document) → an async job is queued: parse → chunk → embed → index; poll with get_job.
3. Search with search_kb: sqlite-vec KNN + FTS5 BM25 → RRF → optional rerank; rerank status is explicit (applied | fallback_rrf | disabled).
4. Trace hits with get_chunk (surrounding context + provenance metadata; content flagged untrusted: true).
5. Delete with confidence: delete_document soft-deletes instantly (re-import of an unchanged source restores it); delete_kb requires a two-step confirm_token; rebuild_index physically compacts.
6. Data lives under storage_path (default ~/.dsh/kb-manager/). The Web panel reads the same service surface: KB list / create / detail, in-KB hybrid search, import with live job progress, rebuild, and two-step delete.

Agent tools

Tool	Description	Key params
create_kb	Create a knowledge base	name*, description?, domain_tags?
list_kbs	List all knowledge bases	—
get_kb	KB details (embedding identity / status)	kb_id*
delete_kb	Two-step delete (impact preview + confirm_token)	kb_id*, confirm_token?
list_documents	List documents (optional status filter)	kb_id*, status?, limit?, offset?
import_document	Import a whitelisted local file or http(s) URL (async job)	kb_id*, source*, metadata?
delete_document	Soft-delete chunks; mark index dirty	kb_id*, doc_id*
search_kb	Hybrid search with explicit rerank status	kb_id*, query*, top_k?, filters?, rerank?
get_chunk	Chunk text + neighbors for tracing	kb_id*, chunk_id*
get_kb_stats	Doc / chunk / index-size / deleted-ratio stats	kb_id*
rebuild_index	Full rebuild (async job)	kb_id*, target_engine?, embedding?
get_job	Poll a background job's progress	job_id*
configure	Tune chunking / retrieval / sync settings	patch

Every tool returns the unified envelope { ok, data | error: { code, message, hint } }; error.hint is an actionable next step. Domain errors are returned (not thrown). Search results are always flagged untrusted: true.

When the Agent uses it

1. "Put this PDF / doc / page into the knowledge base" → import_document + get_job
2. "Search the KB for X" → search_kb
3. "Where did this citation come from?" → get_chunk
4. "Retrieval feels wrong — what's the rerank state?" → search_kb (reads the explicit rerank status)
5. "Create a KB for this project" → create_kb
6. "Remove a bad document" → delete_document (soft; re-import restores)
7. "Delete this whole KB" → delete_kb (two-step: preview impact, then confirm)
8. "How big is this KB?" → list_kbs / get_kb_stats
9. "Re-index everything" → rebuild_index + get_job
10. "Tune retrieval" → configure

Configuration

Defaults work offline. Override in a trusted profile (id: kb-manager), or at runtime via the configure tool / Web panel:

Field	Default	Notes
storage_path	~/.dsh/kb-manager/	Storage root; ~ expands to the home directory
embedding.model	bge-small-zh-v1.5	Embedding model name
embedding.provider	onnx	onnx / mock / openai-compat (onnxruntime-node is optional)
embedding.base_url	''	OpenAI-compatible endpoint for openai-compat
embedding.api_key_ref	''	API key reference
chunk_size	512	Chunk size (characters)
chunk_overlap	50	Overlap length
chunk_strategy	recursive	fixed / recursive
top_k	5	Default hit count
enable_rerank	true	Enable post-RRF rerank
rerank_model	''	Reranker model; empty → rule-based fallback
auto_sync_dir	''	Watch path; empty disables
auto_sync_interval	300	Catch-up interval (seconds)
max_file_size_mb	100	Per-file size cap (MB)

Example (profile line):

- insert:
    - id: kb-manager
      name: 'dsh-kb-manager'
      config:
        storagePath: ''        # empty = ~/.dsh/kb-manager/
        logLevel: 'info'

Boundaries & security

• Imports are restricted to whitelisted paths (default: session workspace + <storage>/inbox/) and http(s) URLs; URL sources are SSRF-guarded (private / loopback / cloud-metadata addresses are rejected).
• Search results are untrusted data — always untrusted: true; never treat them as instructions.
• Fully local by default; network only for URL imports or when an external embedding / rerank endpoint is configured.
• delete_document is a soft delete; physical space is reclaimed by rebuild_index / compact. Re-importing an unchanged source restores it.
• Pure-SQLite indexes: no native vector-DB process to operate.

Differentiating vs typical RAG stacks

Capability	This plugin	Common RAGFlow / Dify / kotaemon / pdfkb-mcp setups
Explicit rerank status (`applied	fallback_rrf	disabled`)
Async jobs with crash recovery + cancel	✅	Often synchronous
Import path whitelist + SSRF guard	✅	Varies
Unified envelope with actionable hint	✅	Varies
untrusted flag on every retrieval result	✅	Rare
Soft-delete + compact physical reclaim	✅	Varies
Zero native build (sqlite-vec prebuilt, Intl.Segmenter)	✅	Often heavy deps

Layout

dsh-kb-manager/
├── package.json  cordis.patch.yml  tsconfig.json  tsconfig.build.json  vitest.config.ts
├── DESIGN.md  README.md  README_ZH.md
├── assets/readme/          # hero.png (banner image)
├── src/
│   ├── host-entry.ts       # DSH plugin entry (tool registration + preamble + cleanup)
│   ├── index.ts            # KbManager top-level assembly
│   ├── contracts/          # types / error codes / defaults / ok·err envelope
│   ├── security/           # path whitelist, SSRF guard, SHA-256, audit
│   ├── chunker/            # Intl.Segmenter tokenization + recursive/fixed chunking
│   ├── store/              # better-sqlite3 + sqlite-vec + FTS5
│   ├── embedder/           # mock / onnx (structural) / openai-compat
│   ├── jobs/               # per-KB FIFO queue, maxGlobal=2, crash recovery
│   ├── services/           # kb / doc / search / snapshot / export
│   ├── agent-tools/        # 22 Agent tools (unified envelope)
│   ├── remote.ts / remote-service.ts / typert.ts / remote-client.ts  # typert remote
│   └── web/                # settings page (KbSection + zh/en locales) → lib/client.js
└── tests/  evals/

Development

npm install        # install deps
npm run typecheck  # tsc --noEmit (strict + noUncheckedIndexedAccess)
npm test           # vitest: tokens / rrf-bm25 / security / chunker / store / tools / pipeline / remote / kb-section
npm run eval       # eval harness: synthetic corpus recall@5 / MRR / P95 (mock embedder)
npm run build      # tsc -p tsconfig.build.json → lib/ (ESM + .d.ts)
npm run dev        # tsx src/host-entry.ts (module-load smoke test)

Known limitations (MVP)

• ONNX embedder is structural: onnxruntime-node is wired as an optional dependency; model download (sha256 + user confirmation) and WordPiece vocab are not yet connected — the onnx provider reports EMB.NOT_READY cleanly. Use provider: 'mock' for dev / test.
• PDF / DOCX parsing is a placeholder: md / txt / csv / json / html are available; pdf / docx report PARSE.UNSUPPORTED (upgrade path in DESIGN §13).
• v1.1 tools (rename_kb, snapshot trio, cancel_job, compact_index, multi_kb_search, export_kb, import_kb): service layer implemented, tool registration deferred.

License

MIT © 2026 xiaoshi7915