RAG System

Table of Contents

• RAG System — Retrieval-Augmented Generation with Hybrid Search
  • Table of Contents
  • Overview
  • Architecture
  • Quick Start
    • Upload documents
    • Try the agent
    • GPU mode
  • Ingestion Pipeline
  • Search Pipeline
  • API Reference
    • RAG API (:8003)
    • Embedder (:8001)
    • Agent (:8011)
  • MCP Tools
  • Project Structure
  • Development
    • Prerequisites
    • Setup
    • Testing
    • Linting and type checking
  • Key Design Decisions
  • License

Overview

A production-grade RAG (Retrieval-Augmented Generation) system that ingests documents, indexes them for hybrid search, and exposes an AI agent with a chat interface. The agent answers user questions by retrieving relevant document chunks and citing its sources.

It uses BGE-M3 for dense and sparse embeddings, Qdrant for vector storage with Reciprocal Rank Fusion (RRF) hybrid search, bge-reranker-v2-m3 for cross-encoder rescoring, and an OpenAI-compatible LLM (DeepSeek by default) behind a LangGraph agent. The agent connects to the RAG backend through the Model Context Protocol (MCP), giving it access to search, document listing, and markdown retrieval tools.

This project demonstrates intermediate-level patterns: content-addressable document storage, async background ingestion, GPU/CPU profile switching, in-process test infrastructure, and transport-agnostic service design.

Architecture

graph TB
    frontend["Frontend<br/>React + CopilotKit<br/>Port 8012"]
    agent["Agent<br/>LangGraph + LLM<br/>Port 8011"]
    ragapi["RAG API<br/>FastAPI<br/>Port 8003"]
    embedder["Embedder<br/>BGE-M3 via FlagEmbedding<br/>Port 8001"]
    reranker["Reranker<br/>bge-reranker-v2-m3 via TEI<br/>Port 8002"]
    qdrant["Qdrant<br/>Vector Database<br/>Port 6333"]
    minio["MinIO<br/>Object Storage<br/>Port 9000"]

    frontend -->|"AG-UI"| agent
    agent -->|"MCP over HTTP"| ragapi
    ragapi -->|"HTTP"| embedder
    ragapi -->|"HTTP"| reranker
    ragapi -->|"REST"| qdrant
    ragapi -->|"S3 API"| minio

Service	Port	Stack	Role
Frontend	8012	React 19, Vite 6, @copilotkit/react-core, @copilotkit/react-ui, AG-UI protocol	Chat interface with multi-thread support
Agent	8011	LangGraph, ChatOpenAI, langchain-mcp-adapters	AI agent that uses MCP tools to answer questions
RAG API	8003	FastAPI, pymupdf4llm, LlamaIndex, FastMCP	Document ingestion, hybrid search, MCP server
Embedder	8001	BGE-M3, FlagEmbedding, FastAPI	Dense (1024-dim) + sparse vector embeddings
Reranker	8002	bge-reranker-v2-m3, HuggingFace TEI	Cross-encoder relevance rescoring
Qdrant	6333	Qdrant (REST)	Vector database with dense + sparse named vectors
MinIO	9000	MinIO, boto3	S3-compatible document storage

Request flow. A user sends a question through the chat UI. The frontend streams it to the agent via the AG-UI protocol. The agent calls MCP tools on the RAG API — typically search_documents — which embeds the query, runs a hybrid search in Qdrant, optionally reranks results, and returns them with source citations. The agent composes an answer and streams it back to the frontend.

Quick Start

Prerequisites: Docker and Docker Compose. An OpenAI-compatible API key is required for the agent (DeepSeek, OpenAI, vLLM, or any compatible provider).

# 1. Clone and configure
git clone <repo-url> && cd rag
cp .env.example .env
# Edit .env: set OPENAI_API_KEY=sk-...

# 2. Start all services (CPU mode, default)
docker compose up -d --build

The first build takes a few minutes — the embedder downloads BGE-M3 (~2 GB) and the reranker pulls its ONNX model. Subsequent starts are fast. Wait for rag-api to become healthy (check with docker compose ps).

Upload documents

Documents are uploaded as a ZIP archive through the RAG API. Processing is async — the endpoint returns a batch_id immediately, and documents are parsed, chunked, embedded, and indexed in the background.

Supported formats: PDF, EPUB, XPS, MOBI, FB2, images (PNG, JPG, TIFF, BMP, WebP), plain text (TXT, MD, HTML, CSV, JSON, XML).

# 3. Create a ZIP archive with your documents
zip -r documents.zip /path/to/your/docs/

# 4. Upload — returns {"batch_id": "...", "total": N}
curl -X POST http://localhost:8003/documents/upload-batch/zip \
  -F "file=@documents.zip"

# 5. Check batch progress (optional)
curl "http://localhost:8003/documents/upload-batch/zip?batch_id=<batch_id>"

# 6. List indexed documents
curl http://localhost:8003/documents | python -m json.tool

Each document gets a SHA-256-based ID. Uploading the same file twice is safe — the second upload returns "existed": true without re-processing.

Try the agent

# 7. Open the chat interface
# http://localhost:8012

Ask questions about your documents. The agent searches the knowledge base, cites document names and chunk indices in its answers.

Example queries:

• "What are the key findings in the report?"
• "Summarize the main arguments from document X"
• "Compare the approaches described in the uploaded papers"

GPU mode

The embedder can run on GPU for faster embedding. Activate with the GPU overlay:

docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d --build

This rebuilds the embedder with CUDA 12.4 support and reserves one NVIDIA GPU. The CPU build produces a ~2.7 GB image; the GPU build is larger but embeds faster.

Ingestion Pipeline

flowchart LR
    A["Document<br/>Upload"] --> B["SHA-256<br/>Hash"]
    B --> C["Dedup<br/>Check"]
    C --> D["pymupdf4llm<br/>Parse"]
    D --> E["MinIO<br/>Store"]
    E --> F["Sentence<br/>Chunking"]
    F --> G["BGE-M3<br/>Embed"]
    G --> H["Qdrant<br/>Upsert"]

1. Upload — documents arrive as a ZIP archive or from a server-side directory.
2. SHA-256 hashing — the file content is hashed to produce a content-based document_id. Identical files produce the same ID regardless of filename.
3. Deduplication — if the document ID already exists in MinIO, ingestion returns immediately with existed=True.
4. Parsing — pymupdf4llm converts PDFs, EPUBs, images, and other formats to Markdown. Plain-text formats (TXT, HTML, CSV, JSON, XML) are read directly.
5. Storage — three objects are written to MinIO per document: the original file ({id}/original.{ext}), the Markdown conversion ({id}/content.md), and metadata ({id}/meta.json).
6. Sentence-based chunking — LlamaIndex SentenceSplitter splits the Markdown into fixed-size chunks (512 chars, 50-char overlap) with sentence-boundary awareness.
7. Embedding — each chunk is sent to the embedder, which returns a dense vector (1024 floats) and a sparse vector (lexical token weights in Qdrant format).
8. Upsert — chunks are written to Qdrant with both vectors and payload fields (document_id, document_name, chunk_index) for provenance tracking.

Ingestion is async: the batch endpoint returns a batch_id immediately and processes documents in the background. The global concurrency semaphore (default 2) prevents CPU oversubscription during parsing.

Search Pipeline

flowchart LR
    A["Query<br/>String"] --> B["BGE-M3<br/>Embed"]
    B --> C["Qdrant<br/>Hybrid RRF"]
    C --> D{"Rerank?"}
    D -- Yes --> E["Cross-encoder<br/>Reranker"]
    D -- No --> F["Results"]
    E --> F

1. Embedding — the query is sent to the embedder, producing dense and sparse vectors.
2. Hybrid search — Qdrant runs both vector searches in parallel and fuses results with Reciprocal Rank Fusion (RRF). Dense search captures semantic similarity; sparse search handles lexical matches (keywords, technical terms, identifiers). The fusion balances both signals.
3. Optional reranking — if rerank=True, the top results are rescored by the bge-reranker-v2-m3 cross-encoder. Reranking considers the full query-document pair, producing more precise relevance scores than embedding similarity alone.
4. Results — returned with text, score, document provenance (document_id, document_name, chunk_index), and optional rerank_score.

The search endpoint also supports filtering by document_id to scope searches to a specific document.

API Reference

RAG API (:8003)

Method	Path	Description
GET	/health	Liveness check
POST	/documents/upload-batch/zip	Upload a ZIP archive of documents (returns batch_id, async processing)
GET	/documents/upload-batch/zip?batch_id=...	Poll batch ingestion progress
GET	/documents	List all documents with metadata
GET	/documents/{document_id}	Document metadata with a presigned download URL
GET	/documents/{document_id}/markdown	Full Markdown content of a document
POST	/search	Hybrid search with optional rerank
*	/mcp	MCP server endpoint (see MCP Tools)

Search request example:

{
  "query": "What is the system architecture?",
  "collection": "default",
  "top_k": 10,
  "rerank": true,
  "document_id": null
}

Embedder (:8001)

Method	Path	Description
GET	/health	Liveness check
POST	/v1/embeddings	OpenAI-compatible embeddings (dense + sparse)

The /v1/embeddings endpoint accepts {"input": ["text"], "model": "BAAI/bge-m3"} and returns standard dense embeddings plus Qdrant-formatted sparse vectors.

Agent (:8011)

Method	Path	Description
POST	/agent	AG-UI streaming chat endpoint
GET	/agent/health	Agent health check

MCP Tools

The RAG API exposes three tools through the Model Context Protocol at /mcp. The agent consumes these tools via streamable_http transport.

Tool	Parameters	Description
search_documents	query, top_k (5), rerank (false), document_id	Hybrid search with optional rerank
list_documents	—	List all indexed documents with metadata
get_document_markdown	document_id	Retrieve full document markdown (truncated at 30K chars)

These tools run in the same process as the RAG API — they share the same RAGClient and DocumentStorage instances, with no additional network calls beyond what the search pipeline already does.

Project Structure

rag/
├── docker-compose.yml            # Service definitions (CPU, default)
├── docker-compose.gpu.yml        # GPU overlay for embedder
├── docker-compose.test.yml       # Test isolation overlay
├── .env.example                  # Environment variable template
│
├── services/
│   ├── rag/                      # RAG API (FastAPI)
│   │   ├── rag/
│   │   │   ├── main.py           # App factory, routes, lifespan
│   │   │   ├── services.py       # RAGService (ingestion), RAGClient (search)
│   │   │   ├── parser.py         # pymupdf4llm parsing, SHA-256
│   │   │   ├── chunker.py        # LlamaIndex sentence splitting
│   │   │   ├── storage.py        # MinIO boto3 document storage
│   │   │   ├── qdrant.py         # Qdrant hybrid search + upsert
│   │   │   ├── clients.py        # Async HTTP to embedder + reranker
│   │   │   ├── config.py         # Environment variable config
│   │   │   ├── schemas.py        # Pydantic models
│   │   │   ├── mcp_server.py     # FastMCP tool definitions
│   │   │   ├── broker.py         # Taskiq broker (in-memory or Redis)
│   │   │   ├── tasks.py          # Background ingestion tasks
│   │   │   ├── batch_service.py  # Batch ingestion orchestrator
│   │   │   └── batch.py          # In-memory batch tracker
│   │   ├── tests/                # pytest suite
│   │   └── Dockerfile
│   │
│   ├── agent/                    # LangGraph agent (LLM + MCP)
│   │   ├── agent/
│   │   │   ├── main.py           # FastAPI + AG-UI protocol
│   │   │   ├── graph.py          # LangGraph agent definition
│   │   │   └── config.py         # LLM + MCP config
│   │   └── Dockerfile
│   │
│   ├── embedder/                 # BGE-M3 embedding service
│   │   ├── embedder/
│   │   │   ├── main.py           # FastAPI, /v1/embeddings
│   │   │   ├── model.py          # BGEM3FlagModel wrapper
│   │   │   └── schemas.py        # Pydantic models
│   │   ├── Dockerfile            # CPU variant
│   │   └── Dockerfile.gpu        # GPU variant (CUDA 12.4)
│   │
│   └── frontend/                 # React chat UI
│       ├── src/
│       │   ├── App.tsx           # Main app with chat + suggestions
│       │   ├── MyRuntimeProvider.tsx  # AG-UI multi-thread runtime
│       │   └── components/
│       ├── Dockerfile            # Multi-stage: Node build + nginx serve
│       └── nginx.conf            # Reverse proxy to agent
│
└── docs/
    └── screenshots/              # UI screenshots for documentation

Development

Prerequisites

• uv — Python dependency management
• Node.js 22+ — frontend build
• Docker and Docker Compose

Setup

# Python services
cd services/rag && uv sync
cd services/agent && uv sync
cd services/embedder && uv sync

# Frontend
cd services/frontend && npm install

Testing

Tests live in services/rag/tests/. The default mode uses in-process backends — no Docker required, ~15 seconds for the full suite.

Service-level tests (no Docker):

cd services/rag && uv run pytest -v

# Parallel execution
cd services/rag && uv run pytest -n auto -v

The _in_process_backends autouse fixture replaces all external dependencies:

• Qdrant — real Qdrant engine in :memory: mode
• MinIO/S3 — moto library intercepts boto3 calls
• Embedder — deterministic fake returning [0.1] * 1024 dense + fixed sparse vectors
• Reranker — fake returning 1.0 / (i + 1) scores
• Chunker — returns each input text as a single chunk
• Parser — returns a hardcoded ParsedDocument

Each test gets a uniquely-named Qdrant collection and MinIO bucket — fully isolated, fully parallel-safe.

Integration tests (requires Docker):

docker compose -f docker-compose.yml -f docker-compose.test.yml up -d
cd services/rag && uv run pytest -m integration -v

Integration tests hit real services and are excluded from the default test run via pytest markers.

Linting and type checking

# Each Python service
cd services/rag && uv run ruff check --fix . && uv run ruff check . && uv run mypy rag/
cd services/agent && uv run ruff check . && uv run mypy agent/
cd services/embedder && uv run ruff check . && uv run mypy embedder/

# Frontend
cd services/frontend && npm run lint:check && npm run format:check

Configuration: Ruff with py312 target, 100-char lines, rules E/F/I/N/W/UP/B/SIM/C4. Mypy with check_untyped_defs, strict_equality, and no_implicit_optional.

Key Design Decisions

• Content-addressable documents. SHA-256 of file content serves as document_id. Re-uploading the same file returns existed=True without re-processing. This makes document identity independent of filename and enables safe idempotent uploads.

• Hybrid search with RRF fusion. Dense embeddings capture semantic meaning; sparse (lexical) embeddings excel at keywords, identifiers, and technical terms. Reciprocal Rank Fusion balances both signals without requiring score calibration.

• MCP protocol for agent integration. The agent accesses RAG tools through the Model Context Protocol, decoupling the agent service from the RAG backend. The same RAGClient that powers the REST API also serves MCP tool calls — no code duplication.

• Stateless RAGClient. The search client holds no state — it receives a SearchRequest and returns a SearchResponse. This lets it be reused across transports (FastAPI dependency injection, MCP server) without modification.

• CPU-optimized by default. The embedder uses PyTorch CPU packages (~2.7 GB image vs ~9 GB with GPU). A docker-compose.gpu.yml overlay swaps in CUDA 12.4 support when GPU acceleration is available.

• In-process test infrastructure. Tests run against real Qdrant (in-memory mode) and moto S3 with deterministic fakes for external HTTP services. This gives integration-level confidence with unit-level speed and zero Docker overhead.

• Thin controllers. FastAPI endpoints validate input, call the service layer, and return a response. All business logic lives in RAGService and RAGClient, making the system testable without HTTP and easy to extend with new transports.

• Async ingestion with backpressure. Batch document processing runs as background asyncio tasks. A global semaphore (configurable via INGEST_CONCURRENCY) limits concurrent CPU-heavy parse operations, preventing resource exhaustion.

License

MIT