Active Learning Administrator Guide

Active learning trains a classifier on the annotations you already have and reorders the queue so annotators see the instances it is least sure about first. This page is the administrator's reference for configuring it.

Table of Contents

1. Overview
2. Basic Configuration
3. Advanced Configuration
4. LLM Integration
5. Model Persistence
6. Multi-Schema Support
7. Monitoring and Metrics
8. Best Practices
9. Troubleshooting
10. Examples

Overview

Uncertain instances carry more information than confident ones, so a fixed annotation budget spent on them produces a better model than the same budget spent in dataset order. The cost is that your labeled set is no longer a random sample of the corpus, which matters if you intend to report distribution statistics over it.

The active learning cycle

1. Training: A machine learning classifier is trained on existing annotations
2. Prediction: The model predicts confidence scores for unannotated instances
3. Reordering: Instances are reordered based on uncertainty (lowest confidence first)
4. Annotation: Annotators work on the most uncertain instances
5. Retraining: The model is retrained periodically as new annotations are added

Basic Configuration

Enabling Active Learning

Add the active_learning section to your YAML configuration file:

active_learning:
  enabled: true
  schema_names: ["sentiment", "topic"]
  min_annotations_per_instance: 2
  min_instances_for_training: 10
  update_frequency: 5
  max_instances_to_reorder: 50

Core Parameters

Parameter	Description	Default	Recommended
enabled	Enable/disable active learning	false	true
schema_names	List of annotation schemas to use	[]	All schemas
min_annotations_per_instance	Minimum annotations needed per instance	1	2-3
min_instances_for_training	Minimum instances needed before training	10	20-50
update_frequency	How often to retrain (in annotations)	5	5-10
max_instances_to_reorder	Maximum instances to reorder	100	50-200

Example Basic Configuration

# Basic active learning setup
active_learning:
  enabled: true
  schema_names: ["sentiment"]
  min_annotations_per_instance: 2
  min_instances_for_training: 20
  update_frequency: 10
  max_instances_to_reorder: 100
  random_sample_percent: 20
  resolution_strategy: "majority_vote"

Advanced Configuration

Classifier Configuration

Choose and configure your machine learning classifier:

active_learning:
  enabled: true
  classifier_name: "sklearn.ensemble.RandomForestClassifier"
  classifier_kwargs:
    n_estimators: 100
    max_depth: 10
    random_state: 42
  vectorizer_name: "sklearn.feature_extraction.text.TfidfVectorizer"
  vectorizer_kwargs:
    max_features: 1000
    ngram_range: [1, 2]
    stop_words: "english"

Naming a classifier or a vectorizer

Both take either a short name or a fully qualified import path. These are the same thing:

active_learning:
  classifier: {name: logistic}
  vectorizer: {name: tfidf}

active_learning:
  classifier: {name: sklearn.linear_model.LogisticRegression}
  vectorizer: {name: sklearn.feature_extraction.text.TfidfVectorizer}

A dotted path is imported as written, so any scikit-learn-compatible estimator works. A bare word that is not one of the short names below is a config error naming the ones that are — it used to validate clean and then fall back to LogisticRegression at the first training run, with the failure only in the log.

Supported Classifiers

Short name	Class	Use Case	Pros	Cons
logistic	LogisticRegression	Binary/multi-class	Fast, interpretable	Linear only
random_forest	RandomForestClassifier	Complex patterns	Robust, handles non-linear	Slower training
svm	SVC	High-dimensional data	Good with sparse data	Memory intensive
—	MultinomialNB	Text classification	Very fast	Assumes independence

Supported Vectorizers

Short name	Class	Use Case	Pros	Cons
count	CountVectorizer	Simple text features	Fast, simple	No word importance
tfidf	TfidfVectorizer	Text with word importance	Better performance	Slightly slower
sentence-transformers	sentence-transformers embeddings	Semantic similarity	Captures meaning	Downloads a model
—	HashingVectorizer	Large datasets	Memory efficient	No feature names

A class with no short name is reached by its import path.

Resolution Strategies

When multiple annotators label the same instance, choose how to resolve conflicts:

active_learning:
  resolution_strategy: "majority_vote"  # Options: majority_vote, consensus, random

Strategy	Description	Use When
majority_vote	Most common label wins	Multiple annotators, clear disagreements
consensus	All annotators must agree	High-quality requirements
random	Randomly select one annotation	Quick testing, simple workflows

LLM Integration

Enabling LLM Support

Integrate Large Language Models for advanced confidence scoring:

active_learning:
  enabled: true
  llm_enabled: true
  llm_config:
    endpoint_url: "http://localhost:8000"
    model_name: "llama-2-7b"
    use_mock: false
    max_tokens: 100
    temperature: 0.1

LLM Configuration Options

Parameter	Description	Default	Example
endpoint_url	VLLM endpoint URL	Required	http://localhost:8000
model_name	Model name on server	Required	llama-2-7b
use_mock	Use mock for testing	false	true
max_tokens	Maximum response tokens	100	50-200
temperature	Response randomness	0.1	0.0-1.0

Mock Mode for Testing

Use mock mode during development and testing:

active_learning:
  llm_enabled: true
  llm_config:
    use_mock: true
    endpoint_url: "http://localhost:8000"  # Not used in mock mode
    model_name: "test-model"

Model Persistence

Enabling Model Persistence

Save trained models for reuse and analysis:

active_learning:
  enabled: true
  model_persistence_enabled: true
  model_save_directory: "/path/to/models"
  model_retention_count: 2

Persistence Configuration

Parameter	Description	Default	Recommended
model_persistence_enabled	Enable model saving	false	true
model_save_directory	Directory to save models	Required	/models/
model_retention_count	Number of models to keep	2	3-5

Training history

Potato records every training run: what was trained, on how many annotations, how long it took, and where the model was saved. There is nothing to configure. The records go into the project's SQLite database alongside the rest of Potato's state, and you can read them at GET /admin/active-learning/stats.

Is the ordering being served?

active_learning.enabled: true means the pool is being reordered. Whether anyone is served that order is a separate question, decided by assignment_strategy, and the two came apart silently: two servers with the same data and the same annotations, one serving the ranked pool and one serving the file order, logged an identical Reordered N instances line and reported the same stats.

The reorder line names the strategy and whether the order will be used, and GET /admin/active-learning/stats carries assignment_strategy and ordering_is_served. enabled: true under the wrong strategy is the likeliest way to misconfigure this feature.

Earlier versions documented database_enabled and a database_config block with MySQL connection details. Neither key did anything: the code behind them created no tables and saved no metrics. Both are ignored if you still have them in a config file, and you can delete them.

Multi-Schema Support

Schema Cycling

Configure active learning to cycle through multiple annotation schemas:

active_learning:
  enabled: true
  schema_names: ["sentiment", "topic", "urgency"]
  min_annotations_per_instance: 2
  min_instances_for_training: 15

Schema-Specific Configuration

Configure different parameters for each schema:

active_learning:
  enabled: true
  schema_names: ["sentiment", "topic"]
  schema_configs:
    sentiment:
      min_annotations_per_instance: 3
      classifier_name: "sklearn.linear_model.LogisticRegression"
    topic:
      min_annotations_per_instance: 2
      classifier_name: "sklearn.ensemble.RandomForestClassifier"

Monitoring and Metrics

Accessing Metrics

The admin interface exposes these metrics:

# Get active learning statistics
from potato.active_learning_manager import get_active_learning_manager

manager = get_active_learning_manager()
stats = manager.get_stats()

print(f"Training count: {stats['training_count']}")
print(f"Models trained: {stats['models_trained']}")
print(f"Last training time: {stats['last_training_time']}")
print(f"LLM enabled: {stats['llm_enabled']}")

Key Metrics

Metric	Description	What to Monitor
training_count	Number of training cycles	Training frequency
models_trained	Schemas with trained models	Coverage across schemas
last_training_time	Time since last training	Training recency
llm_enabled	LLM integration status	LLM availability
training_accuracy	Model accuracy scores	Model performance

Performance Monitoring

Monitor training performance and adjust parameters:

active_learning:
  enabled: true
  # Adjust these based on performance monitoring
  update_frequency: 5      # Increase if training is too frequent
  min_instances_for_training: 20  # Decrease if training is too slow
  max_instances_to_reorder: 50    # Adjust based on dataset size

Best Practices

Configuration Best Practices

1. Start Simple: Begin with basic configuration and add complexity gradually
2. Monitor Performance: Track training times and model accuracy
3. Balance Parameters: Adjust update_frequency and min_instances_for_training based on your workflow
4. Use Appropriate Classifiers: Choose classifiers based on your data characteristics
5. Enable Persistence: Save models for analysis and debugging

Workflow Best Practices

1. Sufficient Initial Data: Ensure you have enough initial annotations before enabling active learning
2. Regular Monitoring: Check metrics regularly to ensure optimal performance
3. Quality Control: Use resolution strategies appropriate for your quality requirements
4. Scalability: Adjust parameters for large datasets
5. Testing: Use mock mode during development and testing

Performance Optimization

1. Fast Classifiers: Use fast classifiers for real-time annotation workflows
2. Feature Limits: Limit vectorizer features to maintain speed
3. Update Frequency: Balance between responsiveness and computational cost
4. Memory Management: Monitor memory usage with large datasets

Troubleshooting

Common Issues

Training Not Triggering

Problem: Active learning training is not being triggered.

Solutions:

• Check min_instances_for_training is not too high
• Verify min_annotations_per_instance is met
• Ensure update_frequency is appropriate
• Check that annotations are being added correctly

active_learning:
  min_instances_for_training: 10  # Reduce if too high
  min_annotations_per_instance: 1  # Reduce if too high
  update_frequency: 5  # Reduce for more frequent training

Slow Training

Problem: Training is taking too long.

Solutions:

• Use faster classifiers (LogisticRegression, MultinomialNB)
• Limit vectorizer features
• Increase update_frequency
• Use simpler vectorizers

active_learning:
  classifier_name: "sklearn.linear_model.LogisticRegression"
  vectorizer_kwargs:
    max_features: 500  # Reduce feature count
  update_frequency: 10  # Train less frequently

LLM Integration Issues

Problem: LLM integration is not working.

Solutions:

• Verify VLLM endpoint is running and accessible
• Check endpoint URL and model name
• Use mock mode for testing
• Verify network connectivity

active_learning:
  llm_config:
    use_mock: true  # Use mock for testing
    endpoint_url: "http://localhost:8000"  # Verify URL
    model_name: "llama-2-7b"  # Verify model name

Debug Mode

Enable debug logging for troubleshooting:

import logging
logging.getLogger('potato.active_learning_manager').setLevel(logging.DEBUG)

Examples

Basic Sentiment Analysis

# Basic sentiment analysis with active learning
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: What is the sentiment of this text?
    labels:
      - positive
      - negative
      - neutral

active_learning:
  enabled: true
  schema_names: ["sentiment"]
  min_annotations_per_instance: 2
  min_instances_for_training: 20
  update_frequency: 10
  classifier_name: "sklearn.linear_model.LogisticRegression"
  vectorizer_name: "sklearn.feature_extraction.text.TfidfVectorizer"
  vectorizer_kwargs:
    max_features: 1000
    stop_words: "english"
  resolution_strategy: "majority_vote"
  random_sample_percent: 20

Multi-Schema Classification

# Multi-schema classification with active learning
annotation_schemes:
  - annotation_type: radio
    name: sentiment
    description: What is the sentiment?
    labels: [positive, negative, neutral]

  - annotation_type: multiselect
    name: topics
    description: What topics are mentioned?
    labels: [politics, technology, sports, entertainment]

active_learning:
  enabled: true
  schema_names: ["sentiment", "topics"]
  min_annotations_per_instance: 2
  min_instances_for_training: 30
  update_frequency: 15
  classifier_name: "sklearn.ensemble.RandomForestClassifier"
  classifier_kwargs:
    n_estimators: 100
    max_depth: 10
  vectorizer_name: "sklearn.feature_extraction.text.TfidfVectorizer"
  vectorizer_kwargs:
    max_features: 2000
    ngram_range: [1, 2]
  resolution_strategy: "majority_vote"
  random_sample_percent: 15
  model_persistence_enabled: true
  model_save_directory: "./models"

Advanced LLM Integration

# Advanced configuration with LLM integration
active_learning:
  enabled: true
  schema_names: ["sentiment", "intent"]
  min_annotations_per_instance: 3
  min_instances_for_training: 50
  update_frequency: 20

  # Traditional ML classifier
  classifier_name: "sklearn.ensemble.RandomForestClassifier"
  classifier_kwargs:
    n_estimators: 200
    max_depth: 15
  vectorizer_name: "sklearn.feature_extraction.text.TfidfVectorizer"
  vectorizer_kwargs:
    max_features: 3000
    ngram_range: [1, 3]

  # LLM integration
  llm_enabled: true
  llm_config:
    endpoint_url: "http://localhost:8000"
    model_name: "llama-2-7b"
    use_mock: false
    max_tokens: 150
    temperature: 0.1

  # Model persistence
  model_persistence_enabled: true
  model_save_directory: "/data/potato/models"
  model_retention_count: 5

  resolution_strategy: "majority_vote"
  random_sample_percent: 10
  max_instances_to_reorder: 200

Query Strategies

Potato supports multiple query strategies beyond basic uncertainty sampling. For the mathematical formulations and citations, see Active Learning Query Strategies.

Strategy	Description	Config Value
Uncertainty Sampling	Select least-confident instances	uncertainty
Diversity Sampling	Maximize feature-space coverage	diversity
BADGE	Uncertainty-weighted diversity	badge
BALD	Ensemble disagreement	bald
Hybrid	Weighted combination	hybrid

active_learning:
  query_strategy: "hybrid"
  hybrid_weights:
    uncertainty: 0.7
    diversity: 0.3

Cold-Start with LLM Selection

Before enough annotations exist for classifier training, Potato can use an LLM to identify the most informative instances. Instances with moderate LLM confidence (near the decision boundary) are prioritized.

active_learning:
  cold_start_strategy: "llm"
  cold_start_batch_size: 20
  llm:
    enabled: true
    endpoint_url: "http://localhost:8080/v1/chat/completions"
    model_name: "your-model"

Sentence-Transformer Embeddings

For tasks where TF-IDF features are insufficient, use pre-trained sentence-transformer embeddings:

active_learning:
  vectorizer_name: "sentence-transformers"
  vectorizer_params:
    model_name: "all-MiniLM-L6-v2"

Requires: pip install sentence-transformers

Probability Calibration

Raw predict_proba outputs from sklearn classifiers are often poorly calibrated. Potato wraps classifiers with CalibratedClassifierCV by default:

active_learning:
  calibrate_probabilities: true  # default

Set to false to disable calibration (e.g., for debugging or when using RandomForest which has better-calibrated probabilities natively).

Configuration Reference (New Fields)

Field	Type	Default	Description
query_strategy	string	"uncertainty"	Query strategy: uncertainty, diversity, badge, bald, hybrid
hybrid_weights	dict	{uncertainty: 0.7, diversity: 0.3}	Weights for hybrid strategy (must sum to 1.0)
bald_params	dict	{n_estimators: 5, bootstrap_fraction: 0.8}	BALD ensemble parameters
calibrate_probabilities	bool	true	Wrap classifier with CalibratedClassifierCV
cold_start_strategy	string	"random"	Cold-start strategy: random, llm
cold_start_batch_size	int	20	Instances to sample for LLM cold-start
classifier_params	dict	{}	Extra parameters passed to classifier constructor
vectorizer_params	dict	{}	Extra parameters passed to vectorizer constructor
use_icl_ensemble	bool	false	Blend ICL predictions with classifier
annotation_routing	bool	false	Enable LLM-based annotation routing

Architecture Overview

The active learning system uses a modular ActiveLearningManager class (singleton pattern) with these key components:

• Asynchronous training: Model training runs in background threads to avoid blocking annotation workflows
• Modular classifiers/vectorizers: Pluggable sklearn classifiers and vectorizers configured via YAML
• LLM integration: Optional VLLM endpoint integration for advanced confidence scoring, with mock mode for testing
• Model persistence: Pickle-based model storage with configurable retention policies and training history tracking
• Schema cycling: Balanced training across multiple annotation schemas

Key source files:

• potato/active_learning_manager.py — Main manager class
• potato/ai/llm_active_learning.py — LLM integration module

Where to go next

• Active Learning Query Strategies — Detailed strategy reference with citations
• AI Support — LLM endpoint configuration