retriv

Hybrid search for TypeScript/JavaScript projects. AST-aware chunking, camelCase tokenization, local-first with optional cloud backends.

Why?

Building search for TS/JS codebases is harder than it looks:

• Keyword search misses semantic matches — "authentication" won't find verifyCredentials()
• Vector search misses exact identifiers — getUserName returns fuzzy matches instead of the function
• Generic chunkers break code mid-function — you need AST-aware splitting that understands TS/JS syntax

retriv is purpose-built for the JS ecosystem: TypeScript compiler API for AST parsing (zero native deps), automatic camelCase/snake_case tokenization, hybrid BM25+vector search with RRF fusion. Start with SQLite, scale to Turso/Upstash/Cloudflare when needed.

Features

• 🎯 Hybrid search — BM25 keywords + vector semantic, merged via RRF
• 🌳 TS/JS AST chunking — TypeScript compiler API splits on function/class boundaries, extracts signatures and scope (zero native deps)
• 🔤 Code-aware tokenization — getUserName → get User Name getUserName for better BM25 recall
• 📦 Local to cloud — start with SQLite, scale to Turso/Upstash/Cloudflare
• 🔍 Metadata filtering — narrow by file type, path prefix, or custom fields

Installation

pnpm add retriv

npx skilld add retriv

Quick Start - Local Hybrid Search

1. Install extra dependencies:

# typescript for AST chunking, sqlite-vec for vector storage, transformers for local embeddings
pnpm add typescript sqlite-vec @huggingface/transformers

2. Create your retriv search instance:

import { createRetriv } from 'retriv'
import { autoChunker } from 'retriv/chunkers/auto'
import sqlite from 'retriv/db/sqlite'
import { transformersJs } from 'retriv/embeddings/transformers-js'

const search = await createRetriv({
  driver: sqlite({
    path: './search.db',
    embeddings: transformersJs(),
  }),
  chunking: autoChunker(), // TS/JS AST + markdown splitting
})

3. Index documents:

await search.index([
  { id: 'src/auth.ts', content: authFileContents, metadata: { type: 'code', lang: 'typescript' } },
  { id: 'docs/guide.md', content: guideContents, metadata: { type: 'docs' } },
])

4. Search!

// hybrid search finds both code and docs
const results = await search.search('password hashing', { returnContent: true })
// [
//   {
//     id: 'src/auth.ts#chunk-2', score: 0.82,
//     content: 'async function hashPassword(raw: string) {\n  ...',
//     _chunk: {
//       parentId: 'src/auth.ts', index: 2,
//       range: [140, 312], lineRange: [12, 28],
//       entities: [{ name: 'hashPassword', type: 'function' }],
//       scope: [{ name: 'AuthService', type: 'class' }],
//     },
//   },
//   {
//     id: 'docs/guide.md#chunk-0', score: 0.71,
//     content: '## Password Hashing\n\nUse bcrypt with...',
//     _chunk: { parentId: 'docs/guide.md', index: 0, range: [0, 487] },
//   },
// ]

// filter to just code files
await search.search('getUserName', { filter: { type: 'code' } })
// [
//   { id: 'src/auth.ts#chunk-0', score: 0.91, _chunk: { parentId: 'src/auth.ts', index: 0, range: [0, 139] } },
// ]

Split-Category Search

When one category of documents (e.g. prose docs) outnumbers another (e.g. code definitions), the majority can drown out minority results. Split-category search fixes this by running parallel filtered searches per category and fusing with RRF — each category gets equal representation regardless of volume.

const search = await createRetriv({
  driver: sqlite({ path: './search.db', embeddings: transformersJs() }),
  categories: doc => doc.metadata?.type || 'other',
})

await search.index([
  { id: 'src/auth.ts', content: authCode, metadata: { type: 'code' } },
  { id: 'docs/guide.md', content: guide, metadata: { type: 'docs' } },
  { id: 'docs/api.md', content: apiRef, metadata: { type: 'docs' } },
])

// Code results won't be buried by docs, even when outnumbered
await search.search('authentication')

The categories function receives each document at index time and returns a category string. The category is stored in metadata.category automatically. At search time, one query runs per seen category and results are fused with RRF.

Categories can be derived from any document property:

// By file extension
categories: doc => /\.(?:ts|js)$/.test(doc.id) ? 'code' : 'docs'

// By explicit metadata
categories: doc => doc.metadata?.category

Cloud Embeddings

pnpm add @ai-sdk/openai ai sqlite-vec

import { openai } from 'retriv/embeddings/openai'

const search = await createRetriv({
  driver: sqlite({
    path: './search.db',
    embeddings: openai(), // uses OPENAI_API_KEY env
  }),
})

Cloud Vector DB

For serverless or edge deployments, compose separate vector and keyword drivers:

import { createRetriv } from 'retriv'
import libsql from 'retriv/db/libsql'
import sqliteFts from 'retriv/db/sqlite-fts'
import { openai } from 'retriv/embeddings/openai'

const search = await createRetriv({
  driver: {
    vector: libsql({
      url: 'libsql://your-db.turso.io',
      authToken: process.env.TURSO_AUTH_TOKEN,
      embeddings: openai(),
    }),
    keyword: sqliteFts({ path: './search.db' }),
  },
})

How It Works

Chunking

Chunking is opt-in via retriv/chunkers/*:

import { autoChunker } from 'retriv/chunkers/auto'
import { markdownChunker } from 'retriv/chunkers/markdown'
import { codeChunker } from 'retriv/chunkers/typescript'

chunking: autoChunker() // Routes by file extension
chunking: markdownChunker() // Heading-aware splitting
chunking: codeChunker() // AST-aware (TS/JS only)

The autoChunker routes by file extension — .ts, .tsx, .js, .jsx, .mjs, .mts, .cjs, .cts use AST splitting, everything else uses heading-aware markdown splitting.

Query Tokenization

Search queries are automatically expanded for code identifier matching:

This improves BM25 recall on code identifiers while being transparent for natural language queries.

Available standalone:

import { tokenizeCodeQuery } from 'retriv/utils/code-tokenize'

Filtering

Narrow search results by metadata using a MongoDB-style filter DSL. Filters are applied at the SQL level (not post-search), so you get exact result counts without over-fetching.

// Attach metadata when indexing
await search.index([
  { id: 'src/auth.ts', content: authCode, metadata: { type: 'code', lang: 'typescript' } },
  { id: 'src/api.ts', content: apiCode, metadata: { type: 'code', lang: 'typescript' } },
  { id: 'docs/guide.md', content: guide, metadata: { type: 'docs', category: 'guide' } },
  { id: 'docs/api-ref.md', content: apiRef, metadata: { type: 'docs', category: 'reference' } },
])

// Search only code files
await search.search('authentication', {
  filter: { type: 'code' },
})

// Search only docs under a path prefix
await search.search('authentication', {
  filter: { type: 'docs', category: { $prefix: 'guide' } },
})

// Combine multiple conditions (AND)
await search.search('handler', {
  filter: { type: 'code', lang: { $in: ['typescript', 'javascript'] } },
})

When chunking is enabled, chunks inherit their parent document's metadata — so filtering works on chunks too.

Operators

Multiple keys in a filter are ANDed together.

Per-driver implementation

Drivers

Hybrid (BM25 + Vector)

Vector-Only

Keyword-Only (BM25)

Embedding Providers

All vector drivers accept an embeddings config:

// Cloud (require API keys)
openai({ model: 'text-embedding-3-small' })
google({ model: 'text-embedding-004' })
mistral({ model: 'mistral-embed' })
cohere({ model: 'embed-english-v3.0' })

// Local (no API key)
ollama({ model: 'nomic-embed-text' })
transformersJs({ model: 'Xenova/all-MiniLM-L6-v2' })

API

SearchProvider

All drivers implement the same interface:

interface SearchProvider {
  index: (docs: Document[]) => Promise<{ count: number }>
  search: (query: string, options?: SearchOptions) => Promise<SearchResult[]>
  remove?: (ids: string[]) => Promise<{ count: number }>
  clear?: () => Promise<void>
  close?: () => Promise<void>
}

SearchOptions

interface SearchOptions {
  limit?: number // Max results (default varies by driver)
  returnContent?: boolean // Include original content in results
  returnMetadata?: boolean // Include metadata in results
  returnMeta?: boolean // Include driver-specific _meta
  filter?: SearchFilter // Filter by metadata fields
}

SearchResult

interface SearchResult {
  id: string // Document ID (or chunk ID like "src/auth.ts#chunk-0")
  score: number // 0-1, higher is better
  content?: string // If returnContent: true
  metadata?: Record<string, any> // If returnMetadata: true
  _chunk?: ChunkInfo // When chunking enabled (see below)
  _meta?: SearchMeta // If returnMeta: true (driver-specific extras)
}

ChunkInfo

When chunking is enabled, each result includes _chunk with source mapping and AST metadata:

interface ChunkInfo {
  parentId: string // Original document ID
  index: number // Chunk position (0-based)
  range?: [number, number] // Character range in original content
  lineRange?: [number, number] // Line range in original content
  entities?: ChunkEntity[] // Functions, classes, methods defined in this chunk
  scope?: ChunkEntity[] // Containing scope chain (e.g. class this method is inside)
}

interface ChunkEntity {
  name: string // e.g. "hashPassword"
  type: string // e.g. "function", "class", "method"
  signature?: string // e.g. "async hashPassword(raw: string): Promise<string>"
  isPartial?: boolean // true if entity was split across chunks
}

Code chunks also produce imports and siblings on the ChunkerChunk level (available when writing custom chunkers):

interface ChunkerChunk {
  text: string
  lineRange?: [number, number]
  context?: string // Contextualized prefix for embeddings
  entities?: ChunkEntity[]
  scope?: ChunkEntity[]
  imports?: ChunkImport[] // { name, source, isDefault?, isNamespace? }
  siblings?: ChunkSibling[] // { name, type, position: 'before'|'after', distance }
}

Related

• skilld — Generate agent skills from npm package docs, uses retriv for search

Sponsors

License

Licensed under the MIT license.