Local LLM Co-Pilot with Ollama

This guide explains how to set up a local LLM as a co-pilot for Claudiomiro using Ollama.

What is the Local LLM Co-Pilot?

The Local LLM Co-Pilot is an optional feature that runs a local language model alongside the main AI (Claude, Codex, etc.) to:

• Reduce token consumption - Offload trivial operations to the local model
• Decrease latency - Process well-defined tasks locally without API calls
• Lower costs - Handle repetitive tasks without using paid API tokens

Important: The local LLM does NOT replace the main AI. It acts as an auxiliary module handling specific, low-ambiguity tasks like:

• Topic classification
• Section extraction from markdown
• Completion detection
• Dependency analysis

When Ollama is unavailable, Claudiomiro automatically falls back to heuristic-based methods (regex, keyword matching) - so everything works without Ollama installed.

Prerequisites

• Claudiomiro installed and working
• Ollama installed on your machine
• A compatible model pulled in Ollama

Step-by-Step Setup

1. Install Ollama

macOS

brew install ollama

Or download from ollama.ai

Linux

curl -fsSL https://ollama.ai/install.sh | sh

Windows

Download the installer from ollama.ai/download

2. Start the Ollama Server

ollama serve

The server runs on http://localhost:11434 by default.

3. Pull a Model

Choose a model optimized for coding tasks:

# Recommended for coding (7B parameters, good balance of speed/quality)
ollama pull qwen2.5-coder:7b

# Alternative options
ollama pull codellama:7b
ollama pull deepseek-coder:6.7b
ollama pull mistral:7b

Tip: Smaller models (7B) are faster but less accurate. Larger models (13B, 34B) are more accurate but slower. For the co-pilot use case, 7B models work well since tasks are well-defined.

4. Verify Installation

# List available models
ollama list

# Test the model
ollama run qwen2.5-coder:7b "Hello, write a simple JavaScript function"

Usage

Enable Local LLM Co-Pilot

Set the CLAUDIOMIRO_LOCAL_LLM environment variable to your model name:

# Enable with qwen2.5-coder
CLAUDIOMIRO_LOCAL_LLM=qwen2.5-coder:7b claudiomiro --prompt="Add user authentication"

# Enable with codellama
CLAUDIOMIRO_LOCAL_LLM=codellama:7b claudiomiro --prompt="Refactor database queries"

# Enable with deepseek-coder
CLAUDIOMIRO_LOCAL_LLM=deepseek-coder:6.7b claudiomiro --prompt="Fix security vulnerabilities"

Persistent Configuration

Option 1: Use the interactive config manager (recommended)

claudiomiro --config

This opens an interactive menu where you can set and persist all configuration options.

Option 2: Quick set via command line

claudiomiro --config CLAUDIOMIRO_LOCAL_LLM=qwen2.5-coder:7b

Option 3: Add to your shell profile

Add to your shell profile (~/.bashrc, ~/.zshrc):

export CLAUDIOMIRO_LOCAL_LLM=qwen2.5-coder:7b

Then restart your terminal or run source ~/.zshrc.

Configuration is stored in claudiomiro.config.json in the package installation directory and loaded automatically on startup.

Disable Local LLM (Default)

The local LLM is disabled by default. To explicitly disable:

# Any of these disable the local LLM
unset CLAUDIOMIRO_LOCAL_LLM
CLAUDIOMIRO_LOCAL_LLM= claudiomiro ...
CLAUDIOMIRO_LOCAL_LLM=false claudiomiro ...

Configuration Options

Environment Variables

Variable	Default	Description
CLAUDIOMIRO_LOCAL_LLM	(not set)	Model name to enable (e.g., qwen2.5-coder:7b). Not set = disabled.
OLLAMA_HOST	localhost	Ollama server hostname
OLLAMA_PORT	11434	Ollama server port
OLLAMA_TIMEOUT	30000	Request timeout in milliseconds
CLAUDIOMIRO_LLM_CACHE	true	Enable response caching
CLAUDIOMIRO_LLM_CACHE_SIZE	1000	Maximum cache entries

Example: Remote Ollama Server

CLAUDIOMIRO_LOCAL_LLM=qwen2.5-coder:7b \
OLLAMA_HOST=192.168.1.100 \
OLLAMA_PORT=11434 \
claudiomiro --prompt="Add payment processing"

Recommended Models

Model	Parameters	Best For	Speed
qwen2.5-coder:7b	7B	General coding	Fast
codellama:7b	7B	Code generation	Fast
deepseek-coder:6.7b	6.7B	Code understanding	Fast
codellama:13b	13B	Complex tasks	Medium
qwen2.5-coder:14b	14B	Better accuracy	Medium

Note: The co-pilot handles well-defined tasks, so smaller/faster models are usually sufficient.

How It Works

When enabled, the local LLM handles specific tasks:

1. Topic Classification - Categorizes content into topics (api, database, authentication, etc.)
2. Section Extraction - Extracts specific sections from markdown documents
3. Summarization - Creates concise summaries of code/documentation
4. Completion Detection - Determines if a task is fully completed
5. Dependency Analysis - Identifies explicit and implicit task dependencies

The main AI (Claude, Codex, etc.) still handles:

• Code generation
• Complex reasoning
• Architecture decisions
• Bug fixing
• Code review

Fallback Behavior

If Ollama is unavailable or returns an error, Claudiomiro automatically falls back to:

• Keyword-based classification - Uses weighted keyword matching
• Regex section extraction - Pattern-based markdown parsing
• Heuristic completion detection - String matching for completion markers
• Regex dependency parsing - Pattern-based dependency extraction

This means Claudiomiro works identically whether or not Ollama is running.

Testing the Integration

Use the --test-local-llm command to verify your Ollama setup:

# Interactive test (will prompt for input)
CLAUDIOMIRO_LOCAL_LLM=qwen2.5-coder:7b claudiomiro --test-local-llm

# Test with a specific prompt
CLAUDIOMIRO_LOCAL_LLM=qwen2.5-coder:7b claudiomiro --test-local-llm --prompt="Write a hello world in JavaScript"

This command will:

1. Check if Ollama server is running
2. Verify the model is available
3. Send your prompt to the local LLM
4. Display the response

Troubleshooting

Ollama Not Found

Error: Connection refused

Solution: Ensure Ollama server is running:

ollama serve

Model Not Found

Error: Model not found

Solution: Pull the model first:

ollama pull qwen2.5-coder:7b

Slow Responses

Solutions:

1. Use a smaller model (7B instead of 13B/34B)
2. Check system resources (RAM, CPU)
3. Increase timeout:

   OLLAMA_TIMEOUT=60000 claudiomiro ...
   

Connection to Remote Server

If connecting to a remote Ollama server:

1. Ensure the server is accessible:

   curl http://192.168.1.100:11434/api/tags
   

2. Check firewall rules allow port 11434

3. On the Ollama server, ensure it listens on all interfaces:

   OLLAMA_HOST=0.0.0.0 ollama serve
   

Verify Local LLM Status

Check if the local LLM is active programmatically:

const { getLocalLLMService } = require('claudiomiro/src/shared/services/local-llm');

const llm = getLocalLLMService();
await llm.initialize();

console.log(llm.getStatus());
// { initialized: true, available: true, fallbackMode: false, model: 'qwen2.5-coder:7b' }

Performance Tips

1. Keep Ollama running - Starting the server has overhead; keep it running in the background
2. Use caching - Response caching is enabled by default, reducing repeated API calls
3. Choose the right model - 7B models offer the best speed/quality balance for co-pilot tasks
4. Monitor memory - 7B models need ~8GB RAM, 13B models need ~16GB RAM

Notes

• The local LLM is completely optional - Claudiomiro works fine without it
• No data is sent externally when using the local LLM (all processing is local)
• The local LLM only handles auxiliary tasks; the main AI remains primary
• Response caching prevents redundant local model calls