vtcode_indexer.md

vtcode-indexer provides a lightweight, pluggable workspace indexer suitable for command-line tooling and autonomous agents that need fast filesystem scans without external services.

Core concepts

• SimpleIndexer walks a workspace, caches metadata, and offers helpers for search, file lookup, and content retrieval.
• SimpleIndexerConfig lets callers toggle hidden directory handling, specify custom index directories, and refine include/exclude lists without hardcoding VT Code's .vtcode layout.
• IndexStorage is a trait abstraction for persisting FileIndex entries. The crate ships with a Markdown implementation and accepts custom backends through SimpleIndexer::with_storage.
• TraversalFilter centralizes directory descent and file-level inclusion decisions, giving downstream users a single place to implement glob rules or binary detection before indexing.

Customizing persistence

The default MarkdownIndexStorage writes summaries as Markdown files under the configured index directory. Downstream projects can provide their own implementation to target alternative formats or external services:

use std::path::Path;
use std::sync::Arc;
use anyhow::Result;
use vtcode_indexer::{FileIndex, IndexStorage, SimpleIndexer};

#[derive(Clone, Default)]
struct MemoryStorage;

impl IndexStorage for MemoryStorage {
    fn init(&self, _index_dir: &Path) -> Result<()> {
        Ok(())
    }

    fn persist(&self, _index_dir: &Path, entry: &FileIndex) -> Result<()> {
        println!("indexed {} ({} bytes)", entry.path, entry.size);
        Ok(())
    }
}

let mut indexer = SimpleIndexer::new(workspace_root.clone())
    .with_storage(Arc::new(MemoryStorage::default()));
indexer.init()?;
indexer.index_directory(workspace_root.as_path())?;

Any IndexStorage implementation is free to establish connections, emit telemetry, or fan out writes during persist as long as it surfaces errors through anyhow::Result.

Tailoring traversal

TraversalFilter implementors can short-circuit directory descent or file-level indexing. The default ConfigTraversalFilter follows SimpleIndexerConfig settings. Custom filters can extend those decisions with domain-specific logic:

use std::path::Path;
use std::sync::Arc;
use vtcode_indexer::{ConfigTraversalFilter, SimpleIndexer, SimpleIndexerConfig, TraversalFilter};

#[derive(Default)]
struct SkipGeneratedFilter {
    inner: ConfigTraversalFilter,
}

impl TraversalFilter for SkipGeneratedFilter {
    fn should_descend(&self, path: &Path, config: &SimpleIndexerConfig) -> bool {
        if path.ends_with("generated") {
            return false;
        }
        self.inner.should_descend(path, config)
    }

    fn should_index_file(&self, path: &Path, config: &SimpleIndexerConfig) -> bool {
        if path
            .extension()
            .and_then(|ext| ext.to_str())
            .is_some_and(|ext| ext.eq_ignore_ascii_case("lock"))
        {
            return false;
        }
        self.inner.should_index_file(path, config)
    }
}

let config = SimpleIndexerConfig::new(workspace_root.clone());
let mut indexer = SimpleIndexer::with_config(config)
    .with_filter(Arc::new(SkipGeneratedFilter::default()));
indexer.init()?;
indexer.index_directory(workspace_root.as_path())?;

Filters can coordinate with configuration to honor allowlists while still blocking noisy directories or file types.

End-to-end example

The examples/custom_storage.rs program demonstrates indexing a temporary workspace using in-memory storage and a filter that skips Rust sources. Run it with cargo run -p vtcode-indexer --example custom_storage to see the indexed paths printed to stdout.

Next steps

• Publish release notes and crate-level documentation updates before the first crates.io release.
• Gather feedback from early adopters to validate trait ergonomics and identify additional feature flags worth exposing.