Timeout Configuration

July 30, 2026 · View on GitHub

Overview

Esperanto provides flexible timeout configuration across all provider types with intelligent defaults and multiple configuration methods. This allows you to optimize for different use cases, from real-time applications requiring quick responses to batch processing with longer-running operations.

Default Timeouts

Different provider types have optimized default timeouts based on typical operation duration:

Provider Type	Default Timeout	Typical Use Case
Language Models	60 seconds	Chat completions, text generation
Embedding	60 seconds	Text embeddings, semantic search
Reranking	60 seconds	Search result reranking
Speech-to-Text	300 seconds (5 min)	Audio transcription
Text-to-Speech	300 seconds (5 min)	Audio generation

Configuration Methods

Configure timeouts using three methods with clear priority hierarchy:

1. Config Dictionary (Highest Priority)

The most explicit and recommended method - set timeout in the config dictionary:

from esperanto.factory import AIFactory

# LLM with custom timeout
model = AIFactory.create_language(
    "openai",
    "gpt-4",
    config={"timeout": 120.0}  # 2 minutes
)

# Embedding with custom timeout
embedder = AIFactory.create_embedding(
    "openai",
    "text-embedding-3-small",
    config={"timeout": 90.0}  # 1.5 minutes
)

# Speech-to-Text with longer timeout for large files
transcriber = AIFactory.create_speech_to_text(
    "openai",
    config={"timeout": 600.0}  # 10 minutes
)

2. Direct Parameters (STT/TTS Only)

For Speech-to-Text and Text-to-Speech providers, you can pass timeout as a direct parameter:

# Text-to-Speech with direct timeout parameter
speaker = AIFactory.create_text_to_speech(
    "elevenlabs",
    timeout=180.0  # 3 minutes
)

# Speech-to-Text with direct timeout parameter
transcriber = AIFactory.create_speech_to_text(
    "openai",
    timeout=450.0  # 7.5 minutes
)

Note: This method is only available for STT and TTS providers. For consistency, using the config dictionary is recommended for all provider types.

3. Environment Variables

Set global defaults for all instances of a provider type:

# Set environment variables
export ESPERANTO_LLM_TIMEOUT=90          # 90 seconds for all LLM providers
export ESPERANTO_EMBEDDING_TIMEOUT=120   # 2 minutes for all embedding providers
export ESPERANTO_RERANKER_TIMEOUT=75     # 75 seconds for all reranker providers
export ESPERANTO_STT_TIMEOUT=600         # 10 minutes for all STT providers
export ESPERANTO_TTS_TIMEOUT=400         # 6.5 minutes for all TTS providers

# These will use environment variable defaults
model = AIFactory.create_language("openai", "gpt-4")  # Uses ESPERANTO_LLM_TIMEOUT
embedder = AIFactory.create_embedding("voyage", "voyage-2")  # Uses ESPERANTO_EMBEDDING_TIMEOUT

Priority Order

Configuration resolves in this priority order (highest to lowest):

Config parameter - Explicit timeout in config dict
Environment variable - Global default for provider type
Provider type default - Built-in default (60s or 300s)

# Example: Final timeout will be 150 seconds (config overrides env var)
# Even if ESPERANTO_LLM_TIMEOUT=90 is set
model = AIFactory.create_language(
    "openai",
    "gpt-4",
    config={"timeout": 150.0}  # This takes precedence
)

Validation

All timeout values are validated with clear error messages to prevent common mistakes:

Valid Range

Minimum: 1 second
Maximum: 3600 seconds (1 hour)

Type Checking

Timeouts must be numeric (int or float):

# ✅ Valid timeouts
AIFactory.create_language("openai", "gpt-4", config={"timeout": 30})      # int
AIFactory.create_language("openai", "gpt-4", config={"timeout": 30.5})    # float
AIFactory.create_language("openai", "gpt-4", config={"timeout": 120.0})   # float

# ❌ Invalid timeouts - will raise ValueError
AIFactory.create_language("openai", "gpt-4", config={"timeout": "30"})    # string
AIFactory.create_language("openai", "gpt-4", config={"timeout": -1})      # negative
AIFactory.create_language("openai", "gpt-4", config={"timeout": 0})       # zero
AIFactory.create_language("openai", "gpt-4", config={"timeout": 4000})    # too large

Error Messages

# TypeError for non-numeric values
try:
    model = AIFactory.create_language("openai", "gpt-4", config={"timeout": "invalid"})
except ValueError as e:
    print(e)  # "Timeout must be a number (int or float), got str"

# ValueError for out-of-range values
try:
    model = AIFactory.create_language("openai", "gpt-4", config={"timeout": -1})
except ValueError as e:
    print(e)  # "Timeout must be between 1 and 3600 seconds, got -1"

Production Use Cases

Real-time Applications

Short timeouts for interactive applications:

# Chatbot with quick response requirement
chatbot = AIFactory.create_language(
    "groq",
    "mixtral-8x7b-32768",  # Fast inference
    config={"timeout": 30.0}  # 30 seconds max
)

# Real-time embedding for instant search
search_embedder = AIFactory.create_embedding(
    "voyage",
    "voyage-2",
    config={"timeout": 20.0}  # 20 seconds
)

Batch Processing

Longer timeouts for bulk operations:

# Batch embedding of large datasets
batch_embedder = AIFactory.create_embedding(
    "openai",
    "text-embedding-3-large",
    config={"timeout": 300.0}  # 5 minutes for large batches
)

# Process thousands of documents
documents = load_documents()  # Large dataset
embeddings = batch_embedder.embed(documents)

Audio Processing

Extended timeouts for long audio files:

# Transcribe hour-long meetings
transcriber = AIFactory.create_speech_to_text(
    "openai",
    config={"timeout": 900.0}  # 15 minutes
)

# Transcribe large audio file
with open("meeting.mp3", "rb") as audio_file:
    transcript = transcriber.transcribe(audio_file)

# Generate long-form audio content
speaker = AIFactory.create_text_to_speech(
    "elevenlabs",
    timeout=600.0  # 10 minutes for long narrations
)

Background Jobs

Maximum timeout for async background processing:

# Long-running document analysis
analyzer = AIFactory.create_language(
    "anthropic",
    "claude-sonnet-5",
    config={"timeout": 3600.0}  # 1 hour max
)

# Process in background
async def process_document(doc):
    messages = [{"role": "user", "content": f"Analyze this document: {doc}"}]
    result = await analyzer.achat_complete(messages)
    return result

Microservices Architecture

Different timeouts for different service tiers:

# API Gateway Configuration
class AIServiceConfig:
    """Centralized timeout configuration"""

    # Quick operations
    QUICK_TIMEOUT = 15.0

    # Standard operations
    STANDARD_TIMEOUT = 60.0

    # Long operations
    LONG_TIMEOUT = 300.0

    @staticmethod
    def create_quick_llm(provider, model):
        return AIFactory.create_language(
            provider, model,
            config={"timeout": AIServiceConfig.QUICK_TIMEOUT}
        )

    @staticmethod
    def create_standard_embedder(provider, model):
        return AIFactory.create_embedding(
            provider, model,
            config={"timeout": AIServiceConfig.STANDARD_TIMEOUT}
        )

    @staticmethod
    def create_batch_transcriber(provider):
        return AIFactory.create_speech_to_text(
            provider,
            config={"timeout": AIServiceConfig.LONG_TIMEOUT}
        )

# Usage in services
quick_chat = AIServiceConfig.create_quick_llm("groq", "llama3-8b-8192")
standard_embedder = AIServiceConfig.create_standard_embedder("openai", "text-embedding-3-small")
batch_transcriber = AIServiceConfig.create_batch_transcriber("openai")

Load Testing

Adjust timeouts for different load scenarios:

# Normal load
normal_model = AIFactory.create_language(
    "openai", "gpt-4",
    config={"timeout": 60.0}
)

# High load - shorter timeout to fail fast
high_load_model = AIFactory.create_language(
    "openai", "gpt-4",
    config={"timeout": 30.0}  # Fail faster under load
)

# Low priority background tasks - longer timeout
background_model = AIFactory.create_language(
    "openai", "gpt-4",
    config={"timeout": 180.0}  # More patient for background work
)

Environment-Specific Configuration

Development

More generous timeouts for debugging:

# .env.development
ESPERANTO_LLM_TIMEOUT=300
ESPERANTO_EMBEDDING_TIMEOUT=180
ESPERANTO_STT_TIMEOUT=900

Production

Tighter timeouts for reliability:

# .env.production
ESPERANTO_LLM_TIMEOUT=60
ESPERANTO_EMBEDDING_TIMEOUT=45
ESPERANTO_STT_TIMEOUT=300

Testing

Very short timeouts to catch issues early:

# .env.test
ESPERANTO_LLM_TIMEOUT=10
ESPERANTO_EMBEDDING_TIMEOUT=10
ESPERANTO_STT_TIMEOUT=30

Loading Environment-Specific Config

import os
from dotenv import load_dotenv

# Load environment-specific config
env = os.getenv("ENVIRONMENT", "development")
load_dotenv(f".env.{env}")

# Models automatically use environment timeouts
model = AIFactory.create_language("openai", "gpt-4")
# Uses timeout from ESPERANTO_LLM_TIMEOUT in loaded env file

Timeout Handling Strategies

Retry with Exponential Backoff

import time

def create_model_with_retry(provider, model_name, max_retries=3):
    """Create model with retry logic for timeout errors"""

    for attempt in range(max_retries):
        try:
            # Increase timeout with each retry
            timeout = 60.0 * (2 ** attempt)  # 60s, 120s, 240s

            model = AIFactory.create_language(
                provider, model_name,
                config={"timeout": min(timeout, 3600.0)}  # Cap at 1 hour
            )

            # Test the model
            test_messages = [{"role": "user", "content": "test"}]
            model.chat_complete(test_messages)

            return model

        except TimeoutError:
            if attempt < max_retries - 1:
                wait_time = 2 ** attempt
                print(f"Timeout on attempt {attempt + 1}, retrying in {wait_time}s...")
                time.sleep(wait_time)
            else:
                raise

# Usage
model = create_model_with_retry("openai", "gpt-4")

Fallback to Faster Model

def create_model_with_fallback(primary_config, fallback_config):
    """Try primary model, fall back to faster model on timeout"""

    try:
        # Try primary model with shorter timeout
        model = AIFactory.create_language(
            **primary_config,
            config={"timeout": 30.0}
        )
        return model, "primary"

    except TimeoutError:
        # Fall back to faster model
        print("Primary model timeout, using fallback...")
        model = AIFactory.create_language(
            **fallback_config,
            config={"timeout": 60.0}
        )
        return model, "fallback"

# Usage
primary = {"provider": "anthropic", "model_name": "claude-sonnet-5"}
fallback = {"provider": "groq", "model_name": "mixtral-8x7b-32768"}

model, used = create_model_with_fallback(primary, fallback)
print(f"Using {used} model")

Circuit Breaker Pattern

from datetime import datetime, timedelta

class CircuitBreaker:
    """Circuit breaker for timeout protection"""

    def __init__(self, failure_threshold=5, timeout_duration=60):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.timeout_duration = timeout_duration
        self.last_failure_time = None
        self.state = "closed"  # closed, open, half-open

    def call(self, func, *args, **kwargs):
        if self.state == "open":
            if datetime.now() - self.last_failure_time > timedelta(seconds=self.timeout_duration):
                self.state = "half-open"
            else:
                raise Exception("Circuit breaker is OPEN")

        try:
            result = func(*args, **kwargs)
            self.on_success()
            return result
        except TimeoutError:
            self.on_failure()
            raise

    def on_success(self):
        self.failure_count = 0
        self.state = "closed"

    def on_failure(self):
        self.failure_count += 1
        self.last_failure_time = datetime.now()
        if self.failure_count >= self.failure_threshold:
            self.state = "open"

# Usage
breaker = CircuitBreaker(failure_threshold=3, timeout_duration=60)

model = AIFactory.create_language("openai", "gpt-4", config={"timeout": 30.0})

try:
    response = breaker.call(
        model.chat_complete,
        [{"role": "user", "content": "Hello"}]
    )
except Exception as e:
    print(f"Circuit breaker prevented call: {e}")

Monitoring and Logging

Track Timeout Performance

import time
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class TimedModel:
    """Wrapper to track model performance"""

    def __init__(self, model):
        self.model = model
        self.call_times = []

    def chat_complete(self, messages):
        start_time = time.time()
        try:
            response = self.model.chat_complete(messages)
            elapsed = time.time() - start_time

            self.call_times.append(elapsed)
            logger.info(f"Call completed in {elapsed:.2f}s")

            # Warn if approaching timeout
            if hasattr(self.model, 'timeout') and elapsed > self.model.timeout * 0.8:
                logger.warning(f"Call took {elapsed:.2f}s, approaching timeout of {self.model.timeout}s")

            return response

        except TimeoutError as e:
            elapsed = time.time() - start_time
            logger.error(f"Timeout after {elapsed:.2f}s")
            raise

    def get_stats(self):
        if not self.call_times:
            return None

        return {
            "count": len(self.call_times),
            "avg": sum(self.call_times) / len(self.call_times),
            "min": min(self.call_times),
            "max": max(self.call_times)
        }

# Usage
base_model = AIFactory.create_language("openai", "gpt-4", config={"timeout": 60.0})
timed_model = TimedModel(base_model)

# Make calls
timed_model.chat_complete([{"role": "user", "content": "Hello"}])

# Get statistics
stats = timed_model.get_stats()
print(f"Average response time: {stats['avg']:.2f}s")

Best Practices

1. Set Appropriate Defaults

# ✅ Good - Reasonable defaults for use case
chat_model = AIFactory.create_language(
    "openai", "gpt-4",
    config={"timeout": 60.0}  # Standard chat timeout
)

# ❌ Bad - Unnecessarily long timeout
chat_model = AIFactory.create_language(
    "openai", "gpt-4",
    config={"timeout": 3600.0}  # 1 hour for simple chat?
)

2. Use Environment Variables for Global Defaults

# Set sensible defaults in environment
export ESPERANTO_LLM_TIMEOUT=60
export ESPERANTO_EMBEDDING_TIMEOUT=45

# Override only when needed
standard_model = AIFactory.create_language("openai", "gpt-4")  # Uses env default

long_running_model = AIFactory.create_language(
    "openai", "gpt-4",
    config={"timeout": 300.0}  # Override for specific use case
)

3. Match Timeout to Provider Speed

# Fast providers - shorter timeout
groq_model = AIFactory.create_language(
    "groq", "mixtral-8x7b-32768",
    config={"timeout": 30.0}  # Groq is very fast
)

# Slower providers - longer timeout
anthropic_model = AIFactory.create_language(
    "anthropic", "claude-sonnet-5",
    config={"timeout": 120.0}  # Claude can be slower
)

4. Consider Content Length

# Short prompts - short timeout
quick_chat = AIFactory.create_language(
    "openai", "gpt-4",
    config={"timeout": 30.0}
)

# Long documents or complex analysis - longer timeout
document_analyzer = AIFactory.create_language(
    "openai", "gpt-4",
    config={"timeout": 180.0}
)

5. Document Your Timeout Strategy

class ModelFactory:
    """
    Centralized model creation with documented timeout strategy:

    - Quick operations (chat, simple queries): 30s
    - Standard operations (analysis, generation): 60s
    - Long operations (batch, complex tasks): 300s
    - Audio operations: 600s
    """

    TIMEOUT_QUICK = 30.0
    TIMEOUT_STANDARD = 60.0
    TIMEOUT_LONG = 300.0
    TIMEOUT_AUDIO = 600.0

    @staticmethod
    def create_chat_model():
        return AIFactory.create_language(
            "openai", "gpt-4",
            config={"timeout": ModelFactory.TIMEOUT_QUICK}
        )

    @staticmethod
    def create_analysis_model():
        return AIFactory.create_language(
            "anthropic", "claude-sonnet-5",
            config={"timeout": ModelFactory.TIMEOUT_STANDARD}
        )