Monitor Lifecycle and Capacity Management Design

July 7, 2025 · View on GitHub

Overview

This document describes the monitor lifecycle management and capacity management systems that govern how monitors are promoted, demoted, and maintained across the NTP Pool monitoring infrastructure.

Monitor Lifecycle

Lifecycle States

Global Monitor Lifecycle (monitors.status)

Registration → Monitor created with pending status
Activation → Manual approval changes status to testing or active
Server Assignment → API creates server_scores entries as candidate
Promotion Pipeline → candidate → testing → active per server
Maintenance → Ongoing health and constraint evaluation
Deactivation → Status changes to paused or deleted

Server-Specific Lifecycle (server_scores.status)

Assignment → Monitor assigned to server as candidate
Testing Phase → Promoted to testing to collect performance data
Active Phase → Promoted to active based on performance
Demotion → Demoted due to constraints, health, or capacity limits
Removal → Removed from server assignment

State Transitions

Promotion Flow

Available Pool → Candidate → Testing → Active
     ↑              ↓         ↓       ↓
     └──────────────┴─────────┴───────┘
           (Demotion paths)

Promotion Criteria

Available → Candidate: Assignment policy (handled by API)
Candidate → Testing: Health check, constraint validation, capacity available
Testing → Active: Performance metrics meet thresholds, globally active status

Demotion Triggers

Constraint violations: Network, account, diversity violations
Health issues: Poor performance metrics, connectivity problems
Global status changes: Monitor becomes pending, paused, or deleted
Capacity management: Excess monitors beyond targets

Promotion Helper Architecture

Centralized promotion logic ensures consistent behavior across all rules:

type promotionResult struct {
    promoted bool
    reason   string
}

func attemptPromotion(monitor *Monitor, targetStatus Status,
    workingCounts *workingCounts, emergency bool) promotionResult {
    // Unified promotion logic with:
    // - Emergency override handling
    // - Constraint checking
    // - Capacity validation
    // - Working count updates
}

Capacity Management

Target Definitions

Active Monitor Targets

Default: 7 active monitors per server
Minimum: Never reduce below 1 (safety mechanism)
Emergency override: Zero active triggers emergency mode

Testing Monitor Targets (Dynamic)

Base target: 5 testing monitors per server
Dynamic adjustment: Increases when active monitors below target
Formula: targetTesting = 5 + max(0, targetActive - actualActive)

Example: Server with 3 active monitors (target 7)

Gap: 7 - 3 = 4 missing active monitors
Testing target: 5 + 4 = 9 testing monitors
Allows larger pipeline when rebuilding active pool

Globally Active Requirements

Minimum: 4 globally active monitors in testing pool per server
Purpose: Ensures promotion pipeline for server-active status
Bootstrap: Promotes globally active candidates when requirement not met

Capacity Enforcement Rules

Rule 1.5: Active Excess Demotion

Demotes excess healthy active monitors when count exceeds target:

Safety Checks:

Never reduce to 0 active monitors
Respects emergency override conditions
Reserves demotion budget for constraint violations

Implementation:

if workingActiveCount > targetActiveMonitors &&
   workingActiveCount > 1 &&
   !emergencyOverride &&
   limits.activeRemovals > demotionsSoFar {

    // Calculate available budget after reserving for constraints
    constraintDemotionsNeeded := countConstraintViolations()
    reservedDemotions := min(constraintDemotionsNeeded, remainingBudget)
    availableDemotions := remainingBudget - reservedDemotions

    // Demote worst performers
    demotionsNeeded := min(excessActive, availableDemotions)
}

Rule 2.5: Testing Pool Management

Manages testing pool size based on dynamic targets:

Dynamic Calculation:

baseTestingTarget := 5
activeGap := max(0, targetActiveMonitors - len(activeMonitors))
dynamicTestingTarget := baseTestingTarget + activeGap

if len(testingMonitors) > dynamicTestingTarget {
    // Demote excess testing monitors to candidate
}

Rule 5: Candidate Promotion with Performance-Based Replacement

Promotes candidates to testing using two complementary approaches:

Phase 1: Capacity-Based Promotion (existing logic):

testingCapacity := max(0, dynamicTestingTarget - workingTestingCount)
promotionsNeeded := min(min(changesRemaining, 2), testingCapacity)

Phase 2: Performance-Based Replacement (new logic): When testing pool is at capacity, compares candidate performance with existing testing monitors and replaces worse performers:

// Only attempt replacement if we have budget remaining
remainingBudget := promotionLimit - capacityPromotions
if remainingBudget > 0 && len(candidates) > 0 && len(testingMonitors) > 0 {
    // Find better-performing candidates that can replace worse testing monitors
    // Respects all constraints and account limits
    replacementChanges := attemptTestingReplacements(...)
}

Performance Comparison Logic:

Health status takes priority (healthy always beats unhealthy)
Among monitors with equal health, RTT determines performance (lower is better)
Only replaces when candidate significantly outperforms testing monitor

Constraint Compliance:

Tests replacement scenarios with temporary account limits
Validates both demotion and promotion against all constraints
Maintains testing pool size (1 out, 1 in)

Change Limits System

Per-Status-Group Limits

Independent limits for each transition type prevent competition:

type changeLimits struct {
    activeRemovals  int // active → testing demotions
    testingRemovals int // testing → candidate demotions
    promotions      int // testing → active, candidate → testing
}

Limit Calculation

func calculateChangeLimits(currentActiveMonitors, blockedMonitors int) changeLimits {
    base := 2 // Increased from 1 for better throughput
    if blockedMonitors > 1 {
        base = 3 // Expedite cleanup when many blocked
    }
    if currentActiveMonitors == 0 {
        base = 4 // Bootstrap mode
    }

    return changeLimits{
        activeRemovals:  base,
        testingRemovals: base,
        promotions:      base,
    }
}

Benefits

Independent processing per status group
Higher throughput (base 2 vs previous 1)
Dynamic testing pool sizing works correctly
Maintained safety with per-group limits

Working Count Tracking

Mathematical Consistency

The system maintains accurate counts throughout the selection process:

type workingCounts struct {
    active  int
    testing int
}

// Initialize with current counts
working := workingCounts{
    active:  len(activeMonitors),
    testing: len(testingMonitors),
}

// Update after each status change decision
func (wc *workingCounts) applyChange(change statusChange) {
    // Decrement from source status
    switch change.fromStatus {
    case ServerScoresStatusActive:
        wc.active--
    case ServerScoresStatusTesting:
        wc.testing--
    }

    // Increment to target status
    switch change.toStatus {
    case ServerScoresStatusActive:
        wc.active++
    case ServerScoresStatusTesting:
        wc.testing++
    }
}

Critical Pattern

Update working counts immediately after each change decision, not after database execution. This prevents counting errors that led to monitor target overruns.

Rule Execution Order

Optimized Sequence

The rules execute in a specific order to maximize efficiency:

Rule 1 (Immediate Blocking): Remove monitors that should be blocked immediately
Rule 2 (Gradual Constraint Removal): Gradual removal of candidateOut monitors
Rule 1.5 (Active Excess Demotion): Demote excess healthy active monitors
Rule 3 (Testing to Active Promotion): Promote from testing to active
Rule 5 (Candidate to Testing Promotion): Promote candidates to testing and replace worse-performing testing monitors
Rule 2.5 (Testing Pool Management): Demote excess testing monitors
Rule 6 (Bootstrap Promotion): Bootstrap case promotions

Order Rationale

Rule 2.5 after Rule 5: Ensures Rule 2.5 sees all pending promotions from Rule 5
Rule 1.5 before Rule 3: Ensures promotion counts are accurate after active demotions
Constraint rules first: Clears violations before capacity optimizations

Constraint bypassing: Allows promotions despite constraint violations
Capacity respect: Still respects capacity limits even in emergency
Unified handling: All promotion functions accept emergencyOverride parameter

Implementation Pattern

emergencyOverride := len(activeMonitors) == 0

// Level 1: Emergency safety (allows constraint violations)
if !emergencyOverride && hasConstraintViolation() {
    return false  // Block only if not emergency
}

// Level 2: Capacity limits (respected even in emergencies)
if workingCount >= targetCount {
    return false  // Always respect capacity
}

Bootstrap Logic

Bootstrap Conditions

Zero active monitors on server
All monitors in candidate status
Need to establish initial monitoring

Bootstrap Behavior

Promotes multiple candidates to testing simultaneously
Respects constraint checking during promotion
Uses emergency override for faster recovery
Targets minimum viable monitoring coverage

Implementation

if len(activeMonitors) == 0 && len(testingMonitors) == 0 {
    // Bootstrap mode: promote up to target testing count
    promoteCount := min(targetTestingCount, len(healthyCandidates))
    // Apply emergency override for constraint bypassing
    emergencyOverride := true
}