Knowledge Base

June 25, 2026 · View on GitHub

The Knowledge module turns unstructured documents (Markdown, PDF, DOCX, …) into a searchable topic library. Upload a file, and EverOS extracts a structured topic tree via LLM, indexes it for keyword + vector search, and keeps the original file for reference.

Quick start

The examples below assume EverOS is running on the default port 8000. See README or QUICKSTART to start the server.

# Upload a document
curl -s -X POST http://localhost:8000/api/v1/knowledge/documents \
  -F "file=@my-report.pdf" \
  -F "title=Q1 Engineering Report" \
  | jq .data
# → { "doc_id": "d_a1b2c3d4e5f6", "category_id": "Technology", "topic_count": 8, ... }

# Search
curl -s -X POST http://localhost:8000/api/v1/knowledge/search \
  -H "Content-Type: application/json" \
  -d '{"query": "performance bottleneck", "method": "hybrid"}' \
  | jq '.data.hits[:3] | .[] | {topic_name, score}'

Three-tier hierarchy

Knowledge is organized into three levels, from broadest to most granular:

L0  Category      ← taxonomy bucket (e.g., "Technology", "Finance")
L1  Document      ← one uploaded file = one document
L2  Topic         ← LLM-extracted section with content

Each level corresponds to a different granularity of API:

Level	Endpoint	Returns
L0	`GET /categories`	`category_id`, `description`, `document_count`
L1	`GET /documents`	`doc_id`, `title`, `category_id`, `topic_count`, `created_at`
L1 (detail)	`GET /documents/{id}`	Full detail: summary, source info, original file path, topic list
L2	`GET /topics/{id}`	Full topic: content, labels, tree position

Storage layout

Every document is a self-contained directory. Markdown files are the single source of truth; SQLite and LanceDB are derived indexes built automatically by the cascade daemon.

~/.everos/<app>/<project>/knowledge/
├── .taxonomy.md                              ← category definitions (YAML)
├── Technology/
│   └── Q1_Engineering_Report_d_a1b2c3d4e5f6/
│       ├── index.md                          ← document metadata + summary
│       ├── 1_Performance_Analysis.md         ← topic with full content
│       ├── 2_Infrastructure_Costs.md
│       ├── 3_Team_Velocity.md
│       └── _original/                        ← original uploaded file
│           └── my-report.pdf
└── Finance/
    └── Budget_Review_d_f6e5d4c3b2a1/
        ├── index.md
        ├── 1_Revenue.md
        └── _original/
            └── budget.xlsx

Storage roles

Markdown (source of truth)  +  SQLite (structured state)  +  LanceDB (vector + BM25 index)

Store	What it holds	Role
Markdown	Document metadata, summaries, topic content, original files	Single source of truth; human-readable and editable
SQLite	Document rows, topic rows (with content), change queue	Structured queries, paginated lists, count aggregation
LanceDB	Topic vectors, BM25 tokens, scalar fields	Search index (fully rebuildable from Markdown)

Even if SQLite and LanceDB data is corrupted, as long as the Markdown files are intact, the indexes can be fully rebuilt via the cascade daemon.

Markdown format

index.md (document root):

---
type: knowledge_document
id: d_a1b2c3d4e5f6
doc_id: d_a1b2c3d4e5f6
category_id: Technology
title: Q1 Engineering Report
schema_version: 1
source_name: my-report.pdf
source_type: file
---
This report covers Q1 engineering outcomes including performance
analysis, infrastructure costs, and team velocity metrics.

The body is an LLM-generated summary of the entire document.

Topic files (e.g., 1_Performance_Analysis.md):

---
type: knowledge_topic
id: d_a1b2c3d4e5f6_1
node_id: d_a1b2c3d4e5f6_1
doc_id: d_a1b2c3d4e5f6
category_id: Technology
topic_index: 1
topic_name: Performance Analysis
topic_path: Q1 Engineering Report > Performance Analysis
summary: Analysis of API latency, database query times, and caching hit rates.
depth: 1
parent_node_id: d_a1b2c3d4e5f6_0
children_node_ids: []
content_labels: ["performance", "latency", "caching"]
schema_version: 1
---
The P99 API latency dropped from 450ms to 120ms after the Redis
caching layer was deployed in week 6. Database query times improved
by 40% following the index optimization sprint...

The body is the full extracted content for this topic.

The taxonomy file uses kind (not type) in its frontmatter to distinguish it from document and topic files, which use type.

Original file preservation

The _original/ subdirectory stores the uploaded binary file unchanged. Users can locate the original via the original_file_path field returned by GET /documents/{doc_id}.

The underscore prefix follows the Jekyll/Eleventy convention for non-content directories that the cascade daemon should skip.

Lifecycle:

POST (create) — writes _original/<filename>
PUT (replace) — clears the old directory and writes the new file
DELETE — removes the entire document directory including _original/
PATCH (category change) — moves the whole directory; _original/ follows

Taxonomy

Categories are defined in .taxonomy.md at the knowledge root. EverOS ships with 20 default categories:

Category	Description
Technology	CS, software, AI/ML, cloud, cybersecurity
Science	Physics, chemistry, biology, astronomy
Medical	Clinical medicine, drugs, public health
Finance	Securities, banking, accounting, fintech
Legal	Laws, contracts, compliance, IP
Education	Teaching, curriculum, e-learning
Business	Strategy, marketing, operations, HR
Engineering	Mechanical, civil, electrical engineering
Arts	Visual arts, music, literature, film
Sports	Athletics, fitness, sports science
Travel	Tourism, hospitality, transportation
Food	Culinary, nutrition, food safety
Environment	Climate, ecology, sustainability
Politics	Government, international relations, policy
History	Historical events, civilizations, historiography
Psychology	Cognitive science, behavioral psychology, mental health
Agriculture	Farming, crop science, agribusiness
RealEstate	Property development, urban planning, housing
Media	Journalism, social media, PR
Others	Fallback for unclassified documents

Customization

Edit .taxonomy.md directly to add, remove, or rename categories:

---
kind: knowledge_taxonomy
categories:
  - id: Technology
    description: Computer science, software engineering, AI/ML.
  - id: InternalOps
    description: Company-specific operational procedures and runbooks.
  - id: CustomerSuccess
    description: Customer onboarding, support playbooks, case studies.
---

Taxonomy changes are hot-reloaded — no server restart needed. The system reads .taxonomy.md from disk on every upload and category list request, so edits take effect immediately.

When a document is uploaded, the LLM selects the best-matching category from this list. If no category matches, the document falls back to Others. You can also specify category_id explicitly in the upload request to bypass LLM classification.

API reference

All endpoints are under /api/v1/knowledge. Responses use the envelope format {"request_id": "...", "data": {...}}. The request_id field is omitted from examples below for brevity.

Upload a document

POST /documents
Content-Type: multipart/form-data

Field	Type	Required	Description
`file`	file	yes	The document to upload
`title`	string	yes	Human-readable title
`source_type`	string	no	Provenance type (`"file"`, `"url"`, …)
`category_id`	string	no	Skip LLM classification; use this category
`app_id`	string	no	Tenant app (default: `"default"`)
`project_id`	string	no	Tenant project (default: `"default"`)

Response (201):

{
  "data": {
    "doc_id": "d_a1b2c3d4e5f6",
    "category_id": "Technology",
    "topic_count": 8,
    "source_name": "my-report.pdf",
    "md_path": "/home/user/.everos/default_app/default_project/knowledge/Technology/Q1_Report_d_a1b2c3d4e5f6",
    "original_file_path": "/home/user/.everos/.../Q1_Report_d_a1b2c3d4e5f6/_original/my-report.pdf"
  }
}

original_file_path is the absolute path to the preserved upload, or null when no binary was stored (e.g. an empty filename).

Example — Python:

from pathlib import Path

import httpx


async def upload_document(file_path: str, title: str) -> dict:
    async with httpx.AsyncClient(base_url="http://localhost:8000") as client:
        with open(file_path, "rb") as f:
            resp = await client.post(
                "/api/v1/knowledge/documents",
                files={"file": (Path(file_path).name, f)},
                data={"title": title},
            )
        resp.raise_for_status()
        return resp.json()["data"]

Example — curl:

curl -X POST http://localhost:8000/api/v1/knowledge/documents \
  -F "file=@report.pdf" \
  -F "title=Quarterly Report" \
  -F "category_id=Finance"

Replace a document

PUT /documents/{doc_id}
Content-Type: multipart/form-data

Same fields as POST. Returns 404 if doc_id does not exist; on success returns 200 (not 201). Atomic operation: if extraction fails, the old document is restored from backup.

Update metadata

PATCH /documents/{doc_id}
Content-Type: application/json

{
  "title": "Updated Title",
  "category_id": "Finance"
}

Returns doc_id, updated_at, and updated_fields: ["title", "category_id"]. Changing category_id moves the document directory to the new category folder.

Delete a document

DELETE /documents/{doc_id}

Returns 204 when no topics were removed (document absent or present with zero topics); 200 with doc_id + deleted_topics otherwise.

List documents

GET /documents?page=1&page_size=20&sort_by=created_at&sort_order=desc

Optional filter: category_id=Technology. sort_by accepts created_at (default), updated_at, or title; sort_order is asc or desc.

{
  "data": {
    "documents": [
      {
        "doc_id": "d_a1b2c3d4e5f6",
        "category_id": "Technology",
        "title": "Q1 Engineering Report",
        "topic_count": 8,
        "created_at": "2026-06-24T10:00:00Z"
      }
    ],
    "total": 42,
    "page": 1,
    "page_size": 20
  }
}

Get document detail

GET /documents/{doc_id}

Returns full metadata, summary, original file path, and topic overview list.

{
  "data": {
    "doc_id": "d_a1b2c3d4e5f6",
    "category_id": "Technology",
    "title": "Q1 Engineering Report",
    "summary": "This report covers Q1 engineering outcomes...",
    "source_name": "my-report.pdf",
    "source_type": "file",
    "original_file_path": "/home/user/.everos/.../Q1_Report_d_a1b2c3d4e5f6/_original/my-report.pdf",
    "topics": [
      {
        "topic_id": "d_a1b2c3d4e5f6_1",
        "topic_name": "Performance Analysis",
        "topic_path": "Q1 Engineering Report > Performance Analysis",
        "depth": 1,
        "summary": "Analysis of API latency..."
      }
    ],
    "created_at": "2026-06-24T10:00:00Z",
    "updated_at": "2026-06-24T10:00:00Z"
  }
}

original_file_path is null for documents created before the original file preservation feature, or when no file was attached.

Get topic detail

GET /topics/{topic_id}

Returns the full topic content, tree structure, and labels.

{
  "data": {
    "topic_id": "d_a1b2c3d4e5f6_1",
    "doc_id": "d_a1b2c3d4e5f6",
    "category_id": "Technology",
    "topic_name": "Performance Analysis",
    "topic_path": "Q1 Engineering Report > Performance Analysis",
    "depth": 1,
    "summary": "Analysis of API latency, database query times...",
    "content": "The P99 API latency dropped from 450ms to 120ms...",
    "content_labels": ["performance", "latency", "caching"],
    "parent_topic_id": "d_a1b2c3d4e5f6_0",
    "children_topic_ids": [],
    "created_at": "2026-06-24T10:00:00Z",
    "updated_at": "2026-06-24T10:00:00Z"
  }
}

Search

POST /search
Content-Type: application/json

{
  "query": "performance bottleneck",
  "method": "hybrid",
  "top_k": 10,
  "include_content": true,
  "score_threshold": 0.5
}

Field	Type	Default	Description
`query`	string	—	Search query (required, 1–2000 chars)
`method`	string	`"hybrid"`	`"keyword"`, `"vector"`, or `"hybrid"`
`top_k`	int	10	Max results (1–100)
`include_content`	bool	false	Include full topic content in results
`score_threshold`	float	null	Drop results below this score

Search methods:

keyword — BM25 sparse retrieval over tokenized summary + content
vector — Dense ANN over embedded summary vectors (requires embedding provider)
hybrid — Parallel keyword + vector, fused with Reciprocal Rank Fusion (RRF), then cross-encoder reranking

All three methods embed the query and apply cross-encoder reranking, so knowledge search requires both an embedding and a rerank provider — there is no provider-free fallback (this is by design: no silent degradation). The two failure modes map to distinct status codes:

Provider not configured → 500 CONFIGURATION_ERROR (a required setting is missing; retrying will not help — set EVEROS_EMBEDDING__* / EVEROS_RERANK__*).
Provider configured but failing/timing out at call time → 503 EXTERNAL_SERVICE_UNAVAILABLE (transient; retryable).

Response:

{
  "data": {
    "hits": [
      {
        "topic_id": "d_a1b2c3d4e5f6_1",
        "category_id": "Technology",
        "topic_name": "Performance Analysis",
        "topic_path": "Q1 Engineering Report > Performance Analysis",
        "depth": 1,
        "summary": "Analysis of API latency...",
        "content": "The P99 API latency dropped...",
        "score": 0.92,
        "retrieval_method": "hybrid",
        "source": null,
        "document": {
          "doc_id": "d_a1b2c3d4e5f6",
          "title": "Q1 Engineering Report",
          "summary": "This report covers..."
        }
      }
    ],
    "total": 3,
    "took_ms": 245.6
  }
}

GET /categories

{
  "data": {
    "categories": [
      {"category_id": "Technology", "description": "Computer science...", "document_count": 12},
      {"category_id": "Finance", "description": "Securities...", "document_count": 5},
      {"category_id": "Others", "description": "Fallback...", "document_count": 0}
    ]
  }
}

Search pipeline

query ─→ embed ─→ keyword (BM25) ─┐
                   vector  (ANN) ──┤─→ RRF fusion ─→ rerank ─→ top_k

Embed — the query is embedded using the configured embedding provider
Recall — dual-channel retrieval from LanceDB:
- BM25 channel: keyword matching on summary_tokens + content_tokens
- ANN channel: nearest-neighbor search on the vector column
- In hybrid mode, both channels run in parallel
Fuse — Reciprocal Rank Fusion merges the two candidate lists
Rerank — cross-encoder reranker rescores the top candidates
Filter — drop results below score_threshold and limit to top_k

Configuration

Search tuning parameters in src/everos/config/default.toml:

[knowledge.search]
recall_n = 200       # initial recall pool size per channel
rerank_n = 50        # candidates sent to reranker
mass_top_m = 50      # category-aware retrieve pool
lambda = 0.1         # category boost weight
top_k_cap = 100      # hard cap on returned results

Override via environment variables:

export EVEROS_KNOWLEDGE__SEARCH__RECALL_N=500
export EVEROS_KNOWLEDGE__SEARCH__RERANK_N=100

Cascade sync

The cascade daemon watches the knowledge Markdown directory for file changes and keeps SQLite + LanceDB in sync.

md file written
  → FSEvents / watchdog detects change
  → worker picks up from queue (≤1s poll interval)
  → handler dispatched by file type:
      index.md    → KnowledgeDocumentHandler → SQLite upsert (metadata)
      N_topic.md  → KnowledgeTopicHandler    → tokenize + embed + SQLite + LanceDB upsert

The topic handler uses a SHA-256 content digest to skip unchanged files — re-embedding only happens when the content actually changes.

Typical latency from file write to search availability: 1–3 seconds.

Supported file formats

EverOS accepts text-based files natively. Binary formats require the everos[multimodal] extra (depends on LibreOffice for document conversion).

Category	Formats	Requires `[multimodal]`
Text	`.txt`, `.md`, `.csv`, `.tsv`, `.vtt`	No
Documents	`.pdf`, `.docx`, `.doc`, `.rtf`, `.odt`, `.pages`	Yes
Spreadsheets	`.xlsx`, `.xls`, `.ods`, `.numbers`	Yes
Presentations	`.pptx`, `.ppt`, `.odp`, `.key`	Yes
Web	`.html`, `.htm`, `.eml`	Yes
Images (OCR)	`.png`, `.jpg`, `.webp`, `.tiff`, `.bmp`, `.svg`	Yes
Audio (transcription)	`.mp3`, `.wav`, `.m4a`, `.amr`, `.aiff`, `.aac`, `.ogg`, `.flac`	Yes

pip install everos[multimodal]

Error handling

HTTP	Error code	Scenario
404	`NOT_FOUND`	Document or topic does not exist
409	`CONFLICT`	`doc_id` already exists (use PUT to replace)
415	`UNSUPPORTED_FORMAT`	File format not parseable
422	`INVALID_INPUT`	Empty/oversized query, empty title, invalid ID format
500	`CONFIGURATION_ERROR`	Embedding or rerank provider not configured
503	`EXTERNAL_SERVICE_UNAVAILABLE`	Configured embedding/rerank provider failing at call time
422	`EXTRACTION_EMPTY`	Document parsed but extractor produced no topics
503	`CAPABILITY_UNAVAILABLE`	`everos[multimodal]` not installed

All error responses use the standard error envelope — see api.md → Errors.

Multi-tenancy

All endpoints accept app_id and project_id parameters (default: "default"). Data is fully isolated per tenant pair:

# Tenant A uploads
curl -X POST .../documents -F "file=@a.pdf" -F "title=A" \
  -F "app_id=tenant_a" -F "project_id=proj_1"

# Tenant B cannot see Tenant A's data
curl .../documents?app_id=tenant_b&project_id=proj_1
# → { "documents": [], "total": 0 }

Storage paths, SQLite rows, and LanceDB indexes are all scoped by app_id + project_id.

End-to-end walkthrough

A complete workflow from upload to search:

BASE=http://localhost:8000/api/v1/knowledge

# 1. List available categories
curl -s "$BASE/categories" | jq '[.data.categories[] | .category_id]'
# → ["Technology", "Science", "Medical", ..., "Others"]

# 2. Upload a document
DOC_ID=$(curl -s -X POST "$BASE/documents" \
  -F "file=@architecture-guide.md" \
  -F "title=System Architecture Guide" \
  | jq -r .data.doc_id)
echo "Created: $DOC_ID"

# 3. View document detail (with topic list)
curl -s "$BASE/documents/$DOC_ID" | jq '{
  title: .data.title,
  category: .data.category_id,
  topics: [.data.topics[] | .topic_name],
  original: .data.original_file_path
}'

# 4. Read a topic — pick the first from the detail response
TOPIC_ID=$(curl -s "$BASE/documents/$DOC_ID" \
  | jq -r '.data.topics[0].topic_id')
curl -s "$BASE/topics/$TOPIC_ID" | jq '{
  name: .data.topic_name,
  path: .data.topic_path,
  content: .data.content[:200],
  labels: .data.content_labels
}'

# 5. Search (index is typically ready within 1–3 seconds)
sleep 3
curl -s -X POST "$BASE/search" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "how to handle service failures",
    "method": "hybrid",
    "top_k": 5,
    "include_content": true
  }' | jq '.data | {
    total,
    took_ms,
    top_hit: .hits[0] | {topic_name, score, content: .content[:100]}
  }'

# 6. Move document to a different category
curl -s -X PATCH "$BASE/documents/$DOC_ID" \
  -H "Content-Type: application/json" \
  -d '{"category_id": "Engineering"}' \
  | jq .data.updated_fields
# → ["category_id"]

# 7. Clean up
curl -s -X DELETE "$BASE/documents/$DOC_ID" | jq .