Anchor Engine - Project Plan & Roadmap

June 17, 2026 · View on GitHub

Project Age: 9 months (July 2025 - May 2026) | Status: Production Ready + Security Hardening + v5.2.0

Current Status: v5.2.0 (May 2026)

Version Source: user_settings.json.template → generates $HOME/.anchor/user_settings.json on pnpm install + pnpm start

Recent Major Additions

Streaming Search (/v1/memory/search/stream) - SSE-based, 20 results/batch, 60% lower peak memory
Streaming Ingest (/v1/ingest/streaming) - Large file processing in 1MB chunks with progress tracking
Zod Validation Framework - Centralized schemas in engine/src/config/index.ts (645 lines)
Performance Monitoring Service - Memory, CPU, engine status tracking (engine/src/utils/performance-monitor.ts)
Security Hardening Complete - Path traversal, SQL injection, auth bypass, API key strength
Frictionless Experience - Version banner, watchdog auto-enable, MCP settings integration
Runtime Data Consolidation - All runtime data routes to ~/.anchor/ via engine/src/config/paths.ts
Test Suite Stabilization - 100% pass rate, vitest migration complete

Upcoming: v5.3.0 (June 2026)

Integration test suite (search pipeline, distillation, MCP, memory pressure)
Failure tracking + circuit breaker pattern
Tag sanitization at write time (not just render time)
WASM health check with JS fallbacks
Prometheus metrics export

9-Month Timeline: July 2025 - May 2026

Month 1-8: July 2025 - February 2026 — Foundation to Production

Theme: Build production-ready knowledge engine

✅ See completed phases in specs/tasks.md

Month 9: March 2026 — Documentation & Reproducibility

Theme: Make benchmarks reproducible, lower integration barrier

Month 10: April 2026 — Security Hardening (COMPLETED)

Theme: Address critical security vulnerabilities

Month 11: April-May 2026 — Frictionless Experience (COMPLETED)

Theme: Zero-conf installation, automatic discovery, transparent operations

Project consolidation - Removed redundant anchor-engine-node version
README updates with consolidated documentation references
Standards alignment - Unified standard numbering (001-029)
Spec updates in specs/plan.md and specs/spec.md

Month 12: May 2026 — Streaming & Observability (v5.2.0) (CURRENT)

Theme: Memory-efficient streaming, centralized validation, observability

Completed

Streaming Search - SSE-based endpoint /v1/memory/search/stream with progressive results
Streaming Ingest - Large file processing in configurable chunks (default 1MB)
Zod Validation Framework - Centralized schemas in engine/src/config/index.ts (645 lines)
Performance Monitoring Service - Memory, CPU, engine status, DB health tracking (engine/src/utils/performance-monitor.ts)
UI Stats Dashboard - Real-time system metrics display
DB Clearing & Distill Output - Clean state management for distillation
Runtime Data Consolidation - All runtime data routes to ~/.anchor/ via engine/src/config/paths.ts
Test Suite Stabilization - 100% pass rate, vitest migration complete

In Progress (v5.3.0)

Integration test suite
Failure tracking + circuit breaker
Tag sanitization at write time
WASM health checks with fallbacks
Prometheus metrics export

The database schema is undergoing a major refactor to remove the redundant compounds table. This migration simplifies the schema by moving all unique metadata from compounds into the atoms and molecules tables, aligning with the pointer-only architecture principle (Standard 051).

Current Status: Phase 2 of 5 — Schema Migration In Progress

Phase 1: Schema Analysis ✅ Complete

Completed (March 2026):

Audit all compounds table fields and identify unique vs. redundant data
Map compound metadata to atoms/molecules tables
Document query dependencies on compounds table
Assess breaking changes for ingestion pipeline, radial distiller, search service

Deliverables:

MIGRATION_ANALYSIS.md - Technical analysis of schema
Data mapping document in MIGRATION_PLAN.md

Phase 2: Schema Migration 🔄 In Progress

Current Work (May 2026):

Add missing columns to molecules table (provenance, molecular_signature)
Execute migration script to copy compound data to molecules/atoms
Verify migration integrity with validation queries
Drop compounds table after verification

Migration Script: engine/src/core/schema-migration.sql

Status: Schema changes complete, data migration in progress.

Phase 3: Code Updates ⏳ Pending

Tasks:

Update ingestion pipeline to skip compound creation step
Remove compound ID generation from ingestion logic
Update atom/molecule creation to include provenance field
Rewrite queries that join through compounds table (use source_path instead)
Update radial distiller service to query molecules/atoms directly

Estimated Duration: 4-8 hours

Phase 4: Testing ⏳ Pending

Test Cases:

Ingestion test - Ingest a file and verify molecules/atoms have provenance populated
Query compatibility - Run existing queries that used compounds table
Data integrity - Verify no data loss during migration
Edge cases - Handle empty compounds table, partial migrations

Estimated Duration: 2-4 hours

Phase 5: Deployment ⏳ Pending

Staged Rollout Plan:

Deploy schema changes to staging environment
Run data migration on staging with monitoring
Update ingestion code and deploy to staging
Validate full ingestion workflow on staging
Deploy to production with enhanced monitoring

Monitoring Metrics:

Ingestion latency (should decrease after removal)
Database write volume
Query performance for compound-related queries
Error rates in ingestion pipeline

Estimated Duration: 1-2 hours

Post-Migration: Schema Version Tracking ⏳ Future Work

After the compounds table is removed, implement schema version tracking:

Create schema_versions table to track schema evolution
Add migration history logging
Document in CHANGELOG.md with proper version entries

Documentation Files

File	Purpose	Status
`MIGRATION_PLAN.md`	Detailed implementation plan, SQL scripts, testing procedures	✅ Complete
`MIGRATION_SUMMARY.md`	Executive summary and execution order	✅ Complete
`MIGRATION_ANALYSIS.md`	Technical analysis of compounds table removal	✅ Complete
`engine/src/core/schema-migration.sql`	Consolidated CREATE TABLE statements	✅ Created

Database Schema Reference - Full schema documentation with ER diagram and table definitions
Current Tasks - DB schema clarification progress Total Alerts Analyzed: ~206 from CodeQL tool Final Assessment: LOW severity overall — approximately 85-90% of flagged alerts are either false positives or already mitigated through existing validation layers

Verification Summary Table

Category	Alerts Flagged	False Positives	Real Issues Found
Critical Dependencies (axios, handlebars)	~30+	~95% detected	None - internal-only usage
Path Traversal Attacks (#96-#101, #93-#94)	~15+	~100% detected (all have validation)	None - fully mitigated via `validatePathSafety()` and whitelist regexes
Loop Bound/Rate Limiting (#47, #72, #107)	~8+	~100% detected (bounded operations)	None - fully mitigated with capped values (max 16 threads, default batch size 20)
Backup Paths (4-6 alerts)	4-6	~100% detected (path.join() safety)	None - fully mitigated through proper path construction

Alert Categorized Findings

1. False Positives (CodeQL doesn't consider existing validation layers):

Alert Category	Count	Status
Critical Dependencies (axios, handlebars)	~30+	CodeQL flags npm-lock.yaml files without examining actual usage in source code — these are internal-only dependencies with no user-facing template rendering
Path Traversal (#96-#101, #93-#94)	12+	Already-mitigated via `validatePathSafety()` utility function and whitelist regex `/^[a-zA-Z0-9_-]+$/.test(name)` for snapshot names
Loop Bound Injection (#47, #72, #107)	8+	Already-safe — bounded batch sizes (default: 20), memory-aware processing with capped values (max 16 threads)

2. Already Mitigated Through Proper Code Patterns:

Path Traversal Prevention: Whitelist regexes validate all user-supplied identifiers
Rate Limiting Already Implemented: express-rate-limit middleware on /v1/*
Backup Path Safety: Uses path.join() which prevents traversal

3. Remaining High-Severity Items (Minor Package Upgrades Recommended):

Recommendation	Current	Action Required
Axios (if used externally)	axios@1.13.5 in lockfiles	Upgrade to 1.7.9+ if any external API calls use it
Handlebars.js (internal-only)	Already not a major dependency for user-facing templates	No action required since only used internally

Security Standards Reference

The following standards already provide robust mitigation:

Standard	Description	Implementation Status
Standard 025	Path Traversal Prevention	✅ Complete — `validatePathSafety()` utility, whitelist regexes applied to all user inputs
Standard 130	SQL Injection Prevention (from changelog)	✅ Parameterized queries throughout codebase
Standard 023	Authentication Bypass Prevention	✅ Complete — test endpoints audit done
Standard 024	API Key Strength Validation	✅ Complete — enhanced validation (32-128 chars, mixed case+digits)

Conclusion

The anchor-engine-node repository is security-conscious with robust validation, bounds checking, and input sanitization already implemented throughout its codebase. Most "vulnerabilities" flagged by automated tools like CodeQL can be safely ignored when the developer has already implemented proper defensive coding practices through:

Input whitelisting (whitelist regexes)
Path validation utilities (validatePathSafety())
Bounded operations (never infinite loops)
Rate limiting middleware (express-rate-limit)
Proper error handling and logging

Severity Classification: LOW (after investigation) Root Cause: "Most CodeQL flags are false positives or already mitigated through existing validation layers" Reduction: "Approximately 85-90% of flagged alerts are either false positives or ALREADY-BEEN-MITIGATED"

Phase: Agent Harness Integration (P1 - Q2 2026)

Goal: Enable multiple agent frameworks

OpenCLAW integration (primary target harness)
Harness plugin system architecture
Multi-harness performance monitoring
API documentation for external developers

Deferred Work (Postponed to Q3 2026+)

Rationale: Security hardening and UX packaging take priority; these items will be addressed after core stability is established.

Code Analysis Enhancement (Deferred)

Goal: Deep code understanding via AST-based retrieval

AST pointer support - Link atoms to source code locations
Semantic code search - Query by code structure, not just keywords
Import/export graph edges - Visualize relationships between modules
Type-aware retrieval - Leverage TypeScript types for better context

Advanced RAG Features (Deferred)

Goal: Enterprise-grade retrieval capabilities

Backup & restore system - Automated snapshots of knowledge base
Rolling context slicer (middle-out) - Optimize context window usage
Live context visualizer (RAG IDE) - Real-time view of active context
Provenance bias controls - Track and prioritize source contributions

Ablation Studies & Testing (Deferred)

Goal: Validate design decisions through systematic experimentation

Ablation study results - Measure impact of individual features on performance
Cross-platform CI testing matrix - Test across Windows, macOS, Linux environments

🧭 User Experience & Packaging Roadmap (Q2-Q4 2026)

Phase 1: Developer Usability (April-Mid May 2026) - COMPLETED

Theme: Fix immediate friction points before packaging

P0 - Blockers ✅

Fix Jest→Vitest migration - Migrate test files to vitest syntax ✅ COMPLETED
Security path validation - URL decode + resolve for path traversal bypasses ✅ COMPLETED
Add engine version to logs - Include engine_version field in search log metadata ✅ COMPLETED

P1 - High Priority ✅

CLI commands - Add anchor start, anchor status, anchor search shortcuts ✅ COMPLETED
Startup banner with VERSION - Display version from user_settings.json on launch ✅ COMPLETED
Watchdog auto-enable - Auto-start if watcher.extra_paths configured in settings ✅ COMPLETED

P2 - Medium Priority (Backlog)

Agent discovery - Auto-detect Qwen, Claude, Cursor chat directories
Ingestion progress - Real-time file-level stats in UI and terminal
Debug endpoint - Show why results were filtered for transparency

Phase 2: One‑Click Local Experience (Mid May-Early June 2026)

Theme: Single executable, no npm required

P0 - Critical for First Release

Single executable build - Use pkg or bun build --compile to create standalone binary
System tray / menu bar app - Background process with UI toggle (Windows/macOS)
One‑click installer - Create .exe/.dmg with auto-updater (Winget/Homebrew)

P1 - Essential UX Improvements

First‑run onboarding wizard - "Connect GitHub", "Add local folder", "Index now" flow
UI ingestion progress - Visual feedback during indexing (file count, atoms created)
Desktop notifications - Alert when ingestion completes or errors occur

P2 - Nice to Have

Auto‑start with OS - Toggle in settings for login startup
Built‑in local model launcher - Auto-detect Ollama/LM Studio with one-click connection
Pre-configured prompts - Templates for common tasks (summarize, find decisions)

Phase 3: Non‑Developer Ready (Mid June-Late July 2026)

Theme: Zero-conf installation, transparent operations

P0 - Make It Just Work

Demo vault included - Pre-built sample knowledge base so users see value immediately
GUI settings panel - API keys, paths, model selection via UI (no JSON editing)
Cross-platform installers - Windows (.exe), macOS (.dmg), Linux (.deb/.AppImage)

P1 - Distribution & Updates

Auto‑updater - Check for new versions on startup and prompt to update
Offline mode - Core features work without internet (after initial setup)
Export/import vault - Backup entire knowledge base as single file

P2 - Polish & Scale

Multi-model support UI - Toggle between local/remote models in settings
Plugin marketplace - Community extensions for ingestion, search, visualization
Federation protocol - P2P sync between personal vaults (experimental)

🚧 Current Pain Points (Observed & Documented)

Pain Point	Impact	Status
Startup is manual and fragile — requires terminal, Node.js, npm, remembering commands	Non‑developers can't use it; even you dread starting it	🟡 Medium Priority
No system tray / background service — engine ties up a terminal window	Feels like a dev tool, not a consumer app	🔴 High Priority
No onboarding wizard — user must manually add GitHub repos or paths via config/API	First‑run experience is empty and confusing	🟡 Medium Priority
No visual feedback during ingestion — progress only in logs/terminal	User doesn't know if working or stuck	🟢 Low Priority
No one‑click installer — requires cloning repo, npm install, environment setup	Huge barrier to entry	🔴 High Priority
No auto‑start with OS — must be manually launched after reboot	Memory only available when remembered	🟡 Medium Priority

💡 Proposed Improvements (Capture in Plan)

Packaging & Distribution

Single executable build - pkg/bun compile to standalone binary (Windows/macOS/Linux)
Distribution channels - Homebrew, Winget, direct download with auto-updater
Version pinning - Lock dependencies for reproducible builds

Startup & Background Operation

System tray icon - Windows / menu bar app that starts/stops engine and opens UI ✅ COMPLETED
Run at login option - Toggle in settings to auto-start on OS boot ✅ COMPLETED
Graceful shutdown - Save state before exit, resume on next launch

First‑Run Experience

Onboarding wizard - "Connect GitHub", "Add local folder", "Index now" flow
Demo vault included - Pre-built sample knowledge base with example notes
Quick start templates - One-click presets for common use cases (dev docs, research notes)

Feedback & Visibility

Ingestion progress UI - Real-time file count, atoms created, estimated time remaining ✅ COMPLETED
Desktop notifications - Alert when ingestion completes or errors occur
Status dashboard - Atom counts, storage usage, recent activity in tray icon tooltip ✅ COMPLETED

Configuration & Settings

GUI settings panel - API keys, paths, model selection via UI (no JSON editing) ✅ COMPLETED
Settings import/export - Share configurations between machines or backup easily
Model integration UI - Built‑in local model launcher with one-click connection ✅ COMPLETED

Model Integration & Prompts

Built‑in local model detection - Auto-detect Ollama, LM Studio, other local LLMs ✅ COMPLETED
Pre-configured prompts - Templates for common tasks (summarize, find decisions, etc.)
Multi-model support UI - Toggle between local/remote models in settings ✅ COMPLETED

📊 Success Metrics (UX & Packaging)

Metric	Target	Current	Gap
Time to first search	<5 min from download	~30 min (npm install + config)	🔴 High
Non-dev setup success rate	90%+	~60% (npm confusion)	🟡 Medium
Background operation	Runs without terminal	Requires active terminal window	🔴 High
Onboarding completion	100% of first-time users	Manual config required	🟢 Low
Update friction	Silent, auto-updates	Manual npm update needed	🟡 Medium

🎯 Implementation Notes & Dependencies

Technical Debt to Clear Before Packaging

~~Jest→Vitest migration~~ (12 files) ✅ COMPLETED
~~Security path validation fixes~~ ✅ COMPLETED
~~Engine version logging~~ ✅ COMPLETED
Integration test suite - Critical for production stability (v5.3.0 target)
Failure tracking + circuit breaker - Prevents cascading failures

Recommended Build Tools

pkg or bun build --compile - For single executable compilation
electron-builder or tar - For system tray background app
electron-updater - For auto-update functionality
electron-packager - For cross-platform installer generation

Testing Requirements for UX Features

CI testing on Windows, macOS, Linux (cross-platform matrix)
First-run experience automated tests (onboarding wizard)
Background process stability tests (system tray app)
Update flow integration tests (auto-updater)

🧪 Test Suite Audit (May 2026)

Trigger: AST parser made async (parseCodeStructure → Promise<CodeStructure | null>), plus pre-existing test bugs. Run: pnpm test:all or vitest run --config engine/vitest.config.ts Result: 100% pass rate achieved after vitest migration and assertion fixes

May 2026 — Test Consolidation & Housekeeping (COMPLETED)

Goal: Clean up test chaos, consolidate into a single orchestration point, reduce file count.

Consolidate test files — All tests under engine/tests/ as .test.ts files ✅ DONE
Simplify package.json test scripts — pnpm test (all), pnpm test:unit, pnpm test:integration, pnpm test:bench ✅ DONE
Fix remaining failing tests — 100% pass rate achieved ✅ DONE
Migrate all tests to vitest ✅ DONE

May 2026 — Runtime Data Cleanup (COMPLETED)

Goal: Ensure all runtime data lives in ~/.anchor/, keep project directory clean.

Verify ~/.anchor/ path configured — All paths resolve via engine/src/config/paths.ts ✅ DONE
Update .gitignore — ~/.anchor/ excluded, project-local runtime data excluded ✅ DONE
Clean up stale data from project root — All 16 stale directories inside engine/ removed ✅ DONE
Verify user_settings.json.template generates user_settings.json at $HOME/.anchor/ ✅ DONE

May 2026 — WASM Binary Packaging Plan

Goal: Evaluate packaging personal WASM modules as prebuilt binaries to cut npm out of the loop.

Audit WASM dependencies — List all @rbalchii/*-wasm packages, assess build complexity
Evaluate direct .wasm loading — Can we ship .wasm files in node_modules and load them via fetch() or WebAssembly.instantiate()?
Document tradeoffs — npm registry vs direct loading: versioning, caching, CI, size
Prototype one module — Pick the smallest WASM module and test direct loading

May 2026 — DB Schema Clarification

Goal: Formalize the database schema now that the development wave has settled.

Audit all SQL tables — Document every table, column, index, and their purpose
Create schema migration file — engine/src/core/schema-migration.sql with all CREATE TABLE IF NOT EXISTS statements
Remove redundant ALTER TABLE — Migrate any ADD COLUMN IF NOT EXISTS from db.ts into the migration file
Add schema version tracking — Track schema version in a schema_version table or similar
Document schema — Add to docs/ or specs/ as a reference

Risk Management

Technical Risks

Risk	Probability	Impact	Mitigation
PGlite scalability limits	Medium	High	Benchmark early, SQLite fallback ready
WASM module compatibility	Low	Medium	Graceful degradation, JS fallbacks (Standard 013)
Search calibration brittleness	Medium	Medium	Extensive testing, adaptive fallbacks

Schedule Risks

Risk	Probability	Impact	Mitigation
Scope creep	High	High	Strict adherence to standards
Burnout	Medium	High	Sustainable pace, buffer time
Dependency issues	Medium	Medium	Vendor critical deps, pin versions

Resource Requirements

Development

Node.js 18+
PNPM package manager
Rust toolchain (for WASM compilation, optional)
PGlite (bundled)

Production Deployment

4GB+ RAM minimum
10GB+ storage for knowledge base
No GPU required (CPU-only operation)
Cross-platform: Windows, macOS, Linux

Quality Gates

Code Quality

✅ ESLint - 0 errors
✅ TypeScript - No implicit any
✅ Tests - 100% pass rate (vitest, migration complete)
✅ Documentation - All public APIs documented

Performance Benchmarks

✅ Ingestion throughput >100 atoms/sec
✅ Search latency <200ms p95
✅ Memory efficiency <1GB for 90MB datasets
✅ Native acceleration 20x speedup

Documentation Quality

✅ README - Quick start works
✅ Whitepaper - Architecture explained
✅ Standards - 38 active standards in specs/current-standards/
✅ Examples - Usage examples provided

Document History

Date	Version	Author	Changes
2026-05-10	5.2.0	Anchor Team	Updated version source, fixed all doc references, 29 standards, runtime data consolidation
2026-05-10	4.8.0-draft	Anchor Team	Added v4.7.0 streaming/search/zod/monitoring, updated UX metrics, added test audit details
2026-03-18	4.7.0	Anchor Team	Added security hardening, frictionless experience, CodeQL audit summary
2026-02-20	4.5.4	Anchor Team	6-month history documented, production ready
2026-01-...	...	Anchor Team	Earlier milestones

Repository: https://github.com/RSBalchII/anchor-engine-node Whitepaper: docs/whitepaper.md Standards: specs/current-standards/ — 38 active standards Production Status: ✅ Ready (February 20, 2026) + Security Hardening Complete + v5.2.0 Streaming & Observability