Embedding Options

July 2, 2026 · View on GitHub

cocoindex-code supports a variety of embedding models and providers. This guide helps you choose the right path for your hardware, privacy requirements, and codebase size.

Which Path Should I Choose?
Understanding Speed, Context, and Performance
The ccc init Wizard
Local Sentence-Transformers
LiteLLM Remote (Cloud Providers)
LiteLLM Local
Choosing Based on Your Content
Pacing & Rate Limits

Which Path Should I Choose?

Path	Best For...	Key Advantage	Trade-off
Local Sentence-Transformers	Most users, laptops, quick setup.	Fastest (in-process), private, offline.	Larger initial pip install (`[full]`).
Cloud LiteLLM Remote	Large codebases, weak local hardware.	Top performance, zero local resource usage.	Per-token costs, data leaves machine.
Local LiteLLM	Power users, shared GPU resources.	Flexibility, unified model management.	Requires managing a separate server.

Understanding Speed, Context, and Performance

Speed & Latency

Local Sentence-Transformers: Typically the fastest option for small-to-medium encoder models. Because it runs directly inside the cocoindex-code process, it avoids the network latency of Cloud APIs and the communication overhead of Local Servers (Ollama). Decoder-based models (e.g. harrier-oss-v1) are also in-process, but process tokens sequentially — expect 3–10× slower indexing per chunk on CPU unless you have a GPU.
Local Servers (Ollama): Ideal for running heavy models (like mxbai-embed-large) on a GPU. While it has slight overhead compared to in-process execution, it is much faster than running large models on a CPU.
Cloud APIs: Slower per-request due to network latency, but highly parallel. Best for the initial indexing of massive repositories.

Does Context Size Matter?

Most local models have a 512-token context window, while cloud models (OpenAI, Voyage) support 8k to 32k.

In cocoindex-code, this matters less than you might expect due to our Language-Aware Chunking strategy:

Logical Boundaries: The tool uses Tree-Sitter to understand code structure. It tries to split files at logical boundaries like functions, classes, or methods.
Target Size: While respecting boundaries, it targets a chunk size of ~1,000 characters (~300 tokens).
Compatibility: This hybrid approach ensures code snippets are contextually coherent while remaining small enough to fit perfectly within even the smallest 512-token context windows.

CPU vs. GPU and Architecture

Indexing speed depends on both parameter count and model architecture:

Encoder models (BERT/ModernBERT-based, e.g. Snowflake, LateOn-Code, CodeSearch-ModernBERT): Process all tokens in parallel — fast on any modern CPU. The default xs and all models marked Fast or Very Fast in the table below are encoder-based.
Decoder models (LLM-based, e.g. harrier-oss-v1): Process tokens sequentially, making them 3–10× slower than an encoder of comparable parameter count on CPU. A GPU is strongly recommended to achieve acceptable indexing throughput with these models.

For encoder models at medium or large sizes, a GPU will still accelerate indexing. Add device: cuda (or mps on Mac) to global_settings.yml.

The easiest way to configure embeddings is by running ccc init. On first run, it will guide you through an interactive wizard. To reconfigure later, delete ~/.cocoindex_code/global_settings.yml and re-run.

Provider Selection: Choose between sentence-transformers (local, free) or litellm (cloud/local server).
Model Selection: Enter a HuggingFace ID or a LiteLLM model string (e.g., voyage/voyage-code-3).
Automatic Tuning: ccc init will automatically apply curated defaults (like input_type or prompt_name) and test the connection.

Sentence-Transformers (Local)

This option runs embedding models directly on your machine using the library.

Recommended Models

These are based on MTEB datasets as of 15-Jun-2026. All listed models have been verified to work with the sentence-transformers provider in cocoindex-code.

Tier	Model	Params	Dims	Code Score	Arch	CPU Speed	Best For
Default	`Snowflake/arctic-embed-xs`	22M	384	0.67	Enc	Very Fast	Smallest, most compatible default
Micro	`lightonai/LateOn-Code-edge`	17M	256	0.82	Enc	Crazy Fast (231 cps)	Efficiency King. Incredible code performance for its size.
Small	`lightonai/LateOn-Code`	149M	768	0.85	Enc	Fast (7 cps)	Great balance of speed and accuracy on modern laptops.
Medium	`Shuu12121/CodeSearch-ModernBERT-Crow-Plus`	152M		0.89	Enc	Fast	High accuracy with encoder speed; best for CPU-only indexing at this tier.
Larger	`microsoft/harrier-oss-v1-270m`	270M	640	0.90	Dec	Slow	Highest local accuracy; GPU strongly recommended for acceptable speed.
Multi-Lingual	`ibm-granite/granite-embedding-97m-multilingual-r2`	97M	384	0.80	Enc	Fast	Multilingual codebases (e.g. Code + Docs in different languages).

Other Model Options

The default of Snowflake/arctic-embed-xs is a good choice in most situations, but if you want other options...

Discovery Script: The easiest way is to run our included script to find the current best models for your hardware: uv run scripts/find_best_models.py.
MTEB v3 Leaderboard: For manual discovery, visit the MTEB v3 Leaderboard:
1. Go to the Benchmarks tab
2. Select Code Information Retrieval (CoIR).
3. Filter Model Type to Dense.
4. Enable the Sentence-Transformers Compatible toggle.
5. Adjust Model Size to fit your hardware (e.g., < 500M for CPUs).
Compatibility: Look for Bi-encoders with a fixed dimension size (e.g., 384, 768, 1024). Avoid "Late Interaction" (ColBERT) or "Cross-Encoders".
Architecture: Encoder models (BERT/ModernBERT-based) are much faster on CPUs than Decoder models (LLM-based). A decoder model of the same parameter count can be 3–10× slower per chunk due to sequential token processing. Prefer encoders for CPU-only workloads.

Installation & Configuration

Install with the full extra: pip install "cocoindex-code[full]".

Example global_settings.yml:

embedding:
  provider: sentence-transformers
  model: lightonai/LateOn-Code-edge
  device: cpu # Use 'cuda' or 'mps' if you have a GPU

For more information, see the Sentence-Transformers Documentation.

LiteLLM Remote (Cloud Providers)

Use external API providers for high-quality embeddings via the LiteLLM bridge.

Recommendations

Well ranked in the MTEB v3 benchmarks

Voyage AI (voyage-4-large): Current #1 for code (Score: 0.97).
Gemini (text-embedding-004): Top-tier performance (Score: 0.97) with a generous free tier.
OpenAI (text-embedding-3-small): Reliable and very cost-effective for large codebases.

Configuration

Example for Voyage AI:

embedding:
  provider: litellm
  model: voyage/voyage-4-large
envs:
  VOYAGE_API_KEY: your-api-key-here

For more information, see the LiteLLM Providers Documentation.

Local LiteLLM Providers

Connect to a local embedding server (Ollama, llama.cpp or compatible) for privacy and flexibility.

Ollama

Ensure Ollama is running and you have pulled the model (ollama pull jina/jina-embeddings-v5).

Suggested Models:

Low-end: ollama/all-minilm
Mid-range: ollama/jina-embeddings-v5
High-end: ollama/mxbai-embed-large

Configuration:

embedding:
  provider: litellm
  model: ollama/jina-embeddings-v5

See the Ollama Model Library for more options.

llama.cpp

If you prefer running llama.cpp directly (e.g., using llama-server), you can connect via the OpenAI-compatible interface.

Configuration:

Start your server: llama-server --embedding -m your_model.gguf
Configure global_settings.yml:

embedding:
  provider: litellm
  model: openai/your-model-name
envs:
  OPENAI_API_BASE: http://localhost:8080/v1
  OPENAI_API_KEY: "not-needed"

Choosing Based on Your Content

Heavy Source Code, CPU-only: Use LateOn-Code (Micro/Small) or CodeSearch-ModernBERT-Crow-Plus (Medium Encoder). Both are encoder-based and score >0.85 on code search benchmarks with good CPU throughput.
Heavy Source Code, GPU available: Harrier 270m (Medium Decoder, 0.90) gives the highest local accuracy and is much faster with a GPU.
Large Documentation / Files: Models with large context windows like Voyage 4 Large (Cloud) or OpenAI v3 Large (8k).
Multilingual Projects: Granite 97m (Small Local) or Cohere Multilingual v3 (Cloud).

Fine-Tuning with `indexing_params` and `query_params`

The ccc init wizard will automatically apply recommended defaults for known models.

Example for LiteLLM (Voyage, Gemini):

embedding:
  provider: litellm
  model: voyage/voyage-4-large
  indexing_params:
    input_type: document
  query_params:
    input_type: query

Example for Sentence-Transformers (symmetric models — Harrier, CodeSearch-ModernBERT, etc.):

Most local models don't require asymmetric prompts. If ccc init doesn't auto-detect defaults for your model, an explicit null disables any inherited prompt:

embedding:
  provider: sentence-transformers
  model: microsoft/harrier-oss-v1-270m  # or Shuu12121/CodeSearch-ModernBERT-Crow-Plus
  indexing_params:
    prompt_name: null
  query_params:
    prompt_name: null

Pacing & Rate Limits

When using cloud providers, you often encounter rate limits (number of requests per minute). cocoindex-code provides several mechanisms to manage this:

min_interval_ms (Pacing): Introduces a mandatory delay between requests (e.g., 500 for 2 req/sec).
Automatic Retries: The daemon automatically retries rate-limited requests (429 errors) with exponential backoff (up to 6 times).
Batching: cocoindex-code automatically batches up to 64 text chunks into a single API request to maximize throughput.