Writing a search adapter

A search adapter is the only thing standing between deepdive and a new search backend. Each one is a small class that turns a query string into a ranked list of candidate URLs. Adding one is ~30 lines plus a test. This doc is the contract every adapter must satisfy and a copy-paste scaffold to start from.

deepdive ships these: duckduckgo (default, no key), searxng, brave, tavily, exa, auto, wikipedia, arxiv, github.

The interface

From src/search.ts:

export interface SearchResult {
  url: string;     // absolute http(s) URL of the page to fetch
  title: string;   // display title ("" if unknown)
  snippet: string; // short context excerpt ("" if unknown)
  rank: number;    // 1-based position in this query's result list
}

export interface SearchAdapter {
  readonly name: string;
  search(query: string, limit: number, signal?: AbortSignal): Promise<SearchResult[]>;
}

The contract

1. Return absolute http:/https: URLs. The agent fetches each url verbatim through the headless browser (or the PDF path). Relative URLs, javascript:, and tracking-redirect wrappers must be resolved/unwrapped before you return them.
2. Honor limit. Return at most limit results. Asking the backend for more and slicing is fine; returning 200 when asked for 5 is not.
3. Rank from 1. rank is 1-based and dense. The agent uses it for ordering only; the cross-query dedupe (dedupeByUrl in src/search.ts) happens upstream, so you don't need to dedupe across calls — but don't emit the same URL twice within one call.
4. Throw on failure, don't return [] to hide an error. A non-2xx response or malformed payload should throw new Error("<name> <status> ..."). Empty results for a genuinely empty backend response are fine; swallowing a 500 as [] is not — it hides outages and breaks the auto adapter's fallback logic, which keys on thrown errors.
5. Respect the signal + apply a timeout. Wrap the caller's signal with searchTimeoutSignal(signal) (from src/search.ts) so a hung endpoint can't block the whole run. It composes the caller's abort signal with a hard per-request timeout (DEEPDIVE_SEARCH_TIMEOUT_MS, default 15s).
6. No new runtime dependencies. Use fetch (global in Node 20+) and hand-rolled parsing. If the backend returns HTML/XML, parse it with a regex and a // fix the parser if it breaks comment — see duckduckgo.ts / arxiv.ts. Do not add cheerio, xml2js, etc.
7. Keep the parsing pure and exported. Put the response→SearchResult[] transform in an exported function (mapXResults / parseXHtml) that takes the already-parsed payload and returns the array. That function is what your test exercises — no network needed.
8. Never log or persist credentials. API keys arrive via the constructor (resolved from env in resolveSearchAdapter). They must not appear in any thrown error message, event, or returned field.

Steps

1. Create src/search/<name>.ts implementing SearchAdapter (scaffold below). Keep the network call in search() and the transform in an exported pure function.
2. Register it in src/search.ts — add a case "<name>": to resolveSearchAdapter. Read any key from env there and throw a clear "<name> adapter requires DEEPDIVE_<NAME>_KEY" if it's required and missing. Keyless adapters just construct and return.
3. Add a test test/<name>-adapter.test.mjs importing from ../dist/<name>.js: cover the pure mapper (mapping, ranking, limit, missing fields), the resolveSearchAdapter wiring (key required/optional), and a globalThis.fetch-stubbed search() call asserting the request shape + that non-2xx throws.
4. Document it — add a row to the README "Search adapters" table and list any DEEPDIVE_* env var in the CLI --help Environment block (src/cli.ts) and --search adapter list.
5. Run npm run build && npm test — tsc --strict must pass and your new tests must be green.

Scaffold

// src/search/example.ts
// Example search adapter. <one line on the backend + whether it needs a key.>

import { searchTimeoutSignal, type SearchAdapter, type SearchResult } from "../search.js";

interface ExampleItem {
  link?: string;
  name?: string;
  blurb?: string;
}

export class ExampleSearch implements SearchAdapter {
  readonly name = "example";
  // Drop the constructor arg if the backend needs no key.
  constructor(private readonly key?: string) {}

  async search(query: string, limit: number, signal?: AbortSignal): Promise<SearchResult[]> {
    const url = new URL("https://api.example.com/search");
    url.searchParams.set("q", query);
    url.searchParams.set("count", String(Math.min(limit, 50)));
    const headers: Record<string, string> = {
      accept: "application/json",
      "user-agent": "deepdive (+https://github.com/askalf/deepdive)",
    };
    if (this.key) headers.authorization = `Bearer ${this.key}`;
    const res = await fetch(url, { headers, signal: searchTimeoutSignal(signal) });
    if (!res.ok) throw new Error(`example ${res.status} ${res.statusText}`);
    const json = (await res.json()) as { items?: ExampleItem[] };
    return mapExampleResults(json.items ?? [], limit);
  }
}

// Pure, exported for unit tests — no I/O.
export function mapExampleResults(items: ExampleItem[], limit: number): SearchResult[] {
  return items
    .filter((r) => typeof r.link === "string" && r.link.length > 0)
    .slice(0, limit)
    .map((r, i) => ({
      url: r.link as string,
      title: r.name ?? "",
      snippet: (r.blurb ?? "").trim(),
      rank: i + 1,
    }));
}

// in src/search.ts → resolveSearchAdapter(name, env)
case "example": {
  const { ExampleSearch } = await import("./search/example.js");
  return new ExampleSearch(env.DEEPDIVE_EXAMPLE_KEY); // omit arg if keyless
}

That's the whole surface. PRs adding adapters are welcome — keep them small, keyless-by-default where the backend allows it, and tested.