Synkora Architecture & Code Quality Report

Comprehensive technical analysis of the Synkora codebase covering architecture, code quality, design patterns, scalability, security, and database performance.

---

Table of Contents

• Executive Summary
• Architecture Overview
• Code Quality & Modularity
• Design Patterns
• Security
• Database Performance
• Scalability & Horizontal Scaling
• Availability & Fault Tolerance
• Load Testing Report
• Testing
• Summary Scorecard

---

Executive Summary

Synkora is a well-architected, production-ready AI/LLM application platform built as a full-stack monorepo. The codebase demonstrates strong engineering practices across most dimensions: clean separation of concerns, comprehensive security hardening, Kubernetes-ready horizontal scaling, and solid database performance patterns. The backend follows a layered architecture (Controllers → Services → Models) with async-first design, while the frontend uses modern Next.js 15 patterns with domain-driven API modules and Zustand state management.

Key Strengths:

• Enterprise-grade security (CSP, rate limiting, token blacklisting, input sanitization)
• K8s-native design (distributed WebSocket, cross-pod cache invalidation, health probes)
• Async-first database layer with connection pooling and query timeout protection
• Circuit breaker pattern for external service resilience
• Dead-letter queue for failed background tasks
• Clean domain-separated frontend API client
• Comprehensive k6 load testing suite with mock LLM mode

---

Architecture Overview

System Architecture

                    ┌─────────────────────────────────────────┐
                    │              Load Balancer                │
                    └────────────────┬────────────────────────┘
                                     │
              ┌──────────────────────┼──────────────────────┐
              │                      │                      │
    ┌─────────▼─────────┐ ┌─────────▼─────────┐ ┌─────────▼─────────┐
    │   FastAPI Pod 1    │ │   FastAPI Pod 2    │ │   FastAPI Pod N    │
    │  (API + WebSocket) │ │  (API + WebSocket) │ │  (API + WebSocket) │
    └─────────┬─────────┘ └─────────┬─────────┘ └─────────┬─────────┘
              │                      │                      │
    ┌─────────▼──────────────────────▼──────────────────────▼─────────┐
    │                        Redis (Pub/Sub + Cache)                   │
    └─────────┬──────────────────────┬──────────────────────┬─────────┘
              │                      │                      │
    ┌─────────▼─────────┐ ┌─────────▼─────────┐ ┌─────────▼─────────┐
    │  Celery Workers    │ │  Celery Beat       │ │  Bot Workers       │
    │  (default+billing) │ │  (scheduler)       │ │  (Slack/Telegram)  │
    └─────────┬─────────┘ └───────────────────┘ └───────────────────┘
              │
    ┌─────────▼──────────────────────────────────────────────────────┐
    │                                                                 │
    │  PostgreSQL (pgvector)  │  Qdrant  │  MinIO/S3  │  Elasticsearch│
    └─────────────────────────────────────────────────────────────────┘

Backend Structure (api/src/)

Layer	Directory	Responsibility
Entry	app.py	Application factory, middleware stack, lifespan management
Routing	router_registry.py	Declarative router registration (45+ routes)
Controllers	controllers/	HTTP handlers, request validation, response formatting
Services	services/	Business logic, orchestration, external integrations
Models	models/	SQLAlchemy ORM models (70+ models)
Schemas	schemas/	Pydantic request/response validation
Middleware	middleware/	Auth, rate limiting, CORS, security headers
Tasks	tasks/	Celery background jobs
Config	config/	Pydantic Settings with modular composition
Core	core/	Database, WebSocket, errors, model providers

Frontend Structure (web/)

Layer	Directory	Responsibility
Pages	app/	Next.js 15 App Router with route groups
Components	components/	React components (chat, agents, UI primitives)
API Client	lib/api/	Domain-separated API modules with barrel export
State	lib/store/	Zustand stores
Auth	lib/auth/	Secure token storage (sessionStorage)
Types	lib/types/	TypeScript type definitions
Hooks	lib/hooks/	Custom React hooks

---

Code Quality & Modularity

Backend

Configuration Management: A+

Settings use Pydantic BaseSettings with modular composition. The Settings class inherits from ~12 focused config classes (DatabaseConfig, SecurityConfig, RedisConfig, etc.), each in their own module. Environment variables are validated with types and descriptions.

# api/src/config/settings.py
class Settings(AppConfig, DatabaseConfig, RedisConfig, CeleryConfig, SecurityConfig, ...):
    model_config = SettingsConfigDict(env_file=".env", case_sensitive=False, extra="ignore")

Router Registration: A

A declarative RouteConfig dataclass replaces 50+ import statements. All routes are defined in a single ROUTER_REGISTRY list with dynamic module loading, making route management data-driven.

Base Model Design: A

Clean mixin-based model inheritance (UUIDMixin, TimestampMixin, TenantMixin, SoftDeleteMixin, StatusMixin) with mass-assignment protection via _PROTECTED_FIELDS. All models get UUID primary keys and audit timestamps automatically.

Separation of Concerns: A

Clear boundaries between controllers (HTTP), services (business logic), and models (data). Services handle orchestration without leaking HTTP concerns. Controllers are thin, delegating to services.

Code Style: A

Configured with Ruff (line length 120, Python 3.11 target) enforcing isort, bugbear, and simplify rules. Consistent docstrings and type annotations throughout.

Frontend

API Client Architecture: A

Originally a monolithic client, now refactored into domain-specific modules (agents.ts, conversations.ts, knowledge-bases.ts, etc.) with a barrel export maintaining backwards compatibility. New code can import directly from domain modules.

State Management: A

Zustand stores are lean and focused. The authStore properly handles token lifecycle with secure storage. No over-engineering with complex middleware chains.

Security: A

Token storage migrated from localStorage to sessionStorage via SecureTokenStorage class. Automatic token refresh and migration from legacy storage.

---

Design Patterns

Patterns Implemented

Pattern	Implementation	Quality
Application Factory	create_app() in app.py	Clean, testable app creation
Repository/Service Layer	controllers/ → services/ → models/	Clear separation
Dependency Injection	FastAPI Depends() for DB, auth, tenant	Properly scoped
Mixin Pattern	TenantMixin, SoftDeleteMixin, StatusMixin	Composable model traits
Circuit Breaker	CircuitBreaker class with decorator support	Full state machine (CLOSED/OPEN/HALF_OPEN)
Dead Letter Queue	Celery task_failure signal → Redis sorted set	Auto-trimming, bounded size
Pub/Sub	Redis pub/sub for cache invalidation + WebSocket	Cross-pod communication
Observer	DistributedConnectionManager extends ConnectionManager	Clean inheritance
Strategy	OAuth providers, model providers, vector DB providers	Pluggable backends
Factory	RouteConfig dataclass for declarative router registration	Data-driven routing
Singleton	get_agent_cache(), get_rate_limiter(), get_settings()	Thread-safe lazy init
Decorator	@circuit_breaker("name") for function wrapping	Supports sync + async
Middleware Chain	Request → RateLimit → Security → CORS → Handler	Ordered pipeline

Architectural Highlights

Multi-Tenancy: Implemented at the model level via TenantMixin with tenant_id indexed columns. Tenant context extracted from JWT tokens in middleware, enforced throughout the query layer.

Multi-Agent Hierarchy: Agents support parent-child relationships, sub-agent transfers, and workflow execution (sequential, parallel, loop, custom). The transfer_scope field controls transfer boundaries.

Provider Abstraction: LLM providers abstracted via LiteLLM for unified access to OpenAI, Anthropic, Google, etc. Vector databases abstracted with Qdrant/Pinecone providers. OAuth providers follow a common interface.

---

Security

Authentication & Authorization

Control	Implementation	Status
JWT with blacklisting	Token blacklist service in Redis	Implemented
Token version tracking	Version mismatch rejects stale tokens	Implemented
Role-based access	require_role() dependency factory	Implemented
Tenant isolation	tenant_id in JWT, enforced in queries	Implemented
Secure token storage	sessionStorage + memory cache (frontend)	Implemented
Token refresh	Auto-refresh with tenant context preservation	Implemented
Query param auth removed	Prevents token leakage via referrer/logs	Implemented

Security Middleware Stack

Applied in correct order (first added = last executed in Starlette):

1. Rate Limiting — Redis-backed distributed rate limiting with per-endpoint limits, trusted proxy IP validation, and proper X-RateLimit-* headers
2. Request Size Limiting — 10MB default, 50MB for file uploads, chunked transfer encoding support
3. Security Headers — CSP with nonces, HSTS with preload, X-Frame-Options DENY, Permissions-Policy
4. Dynamic CORS — Configured origins per environment, no wildcard in production
5. Input Sanitization — Comprehensive XSS pattern detection (60+ patterns covering HTML5 event handlers)

Data Protection

• Encryption at rest: Fernet encryption for API keys, tokens, and secrets
• Mass assignment protection: _PROTECTED_FIELDS set on BaseModel blocks sensitive field updates
• Error sanitization: Production errors return generic messages; internal details logged server-side
• PII handling: Sentry configured with send_default_pii=False
• API docs disabled in production: OpenAPI/Swagger only available in debug mode

---

Database Performance

Connection Management

Setting	Value	Purpose
Pool size	50	High-concurrency support
Max overflow	25	Burst capacity
Pool recycle	3600s	Connection freshness
Pre-ping	Enabled	Dead connection detection
Statement timeout	30s	Prevents long-running queries
Pool reset on return	Rollback	Clean connection state
NullPool for tests	Enabled	Avoid pool issues in tests

Async Database Layer

• Dual engine architecture: sync (psycopg2) for Celery, async (asyncpg) for FastAPI
• Lazy-initialized async engine to ensure correct event loop binding
• Proper session lifecycle with async with factory() as session and cleanup in finally
• Connection pool gracefully closed on shutdown

Query Optimization

• Lazy loading default: Agent relationships use lazy="select" to avoid N+1 from 20+ eager-loaded relationships
• Selective eager loading: llm_configs uses lazy="selectin" for frequently accessed config
• Explicit loading in services: 16 files use selectinload/joinedload where relationship data is needed
• Indexed columns: tenant_id, agent_name, status, foreign keys all indexed
• Naming conventions: PostgreSQL index naming convention enforced via SQLAlchemy MetaData

Migration Strategy

• Alembic for production migrations (init_db skipped in production)
• PostgreSQL naming conventions for predictable index/constraint names
• pgvector extension for vector similarity search

---

Scalability & Horizontal Scaling

Stateless API Design

The API layer is stateless — all state resides in PostgreSQL, Redis, or external services. This enables straightforward horizontal pod scaling behind a load balancer.

Distributed WebSocket

DistributedConnectionManager extends the base ConnectionManager with Redis pub/sub for cross-pod message delivery:

• Room-based messaging across pods
• User-targeted messaging across pods
• Broadcast with source-pod deduplication
• Graceful fallback to local-only mode if Redis is unavailable

Distributed Cache Invalidation

Agent cache uses Redis pub/sub to synchronize invalidation across pods:

• Pod identification via HOSTNAME/POD_NAME environment variables
• Source-pod filtering to avoid self-invalidation loops
• Background subscriber task with graceful shutdown

Celery Task Queue

• Queue separation: Default, email, notifications, agents, billing queues
• Dedicated billing worker: Isolated for critical financial operations
• Late acknowledgment: task_acks_late=True ensures tasks survive worker crashes
• Reject on worker lost: task_reject_on_worker_lost=True requeues on pod termination
• Worker recycling: worker_max_tasks_per_child=1000 prevents memory leaks
• Prefetch multiplier: Set to 1 for fair distribution

WebSocket Connection Limits

Configurable via environment variables:

• Per-user: 10 (default)
• Per-tenant: 500 (default)
• Global: 10,000 (default)
• Race condition prevention via asyncio.Lock

Bot Worker Architecture

Dedicated bot worker service for Slack/Telegram at scale:

• Capacity-based (configurable BOT_WORKER_CAPACITY)
• Health check endpoint at :8080/healthz
• Separate from API pods for independent scaling

---

Availability & Fault Tolerance

Kubernetes Health Probes

Probe	Endpoint	Checks
Liveness	GET /live	Application process alive
Readiness	GET /ready	PostgreSQL + Redis connectivity
Health	GET /health	Basic health + version

Circuit Breaker

Full implementation with three states:

CLOSED ──(failures >= threshold)──> OPEN ──(recovery timeout)──> HALF_OPEN
   ^                                                                  │
   └──────────(success in half-open)──────────────────────────────────┘

• Configurable failure threshold, recovery timeout, half-open request limit
• Thread-safe with RLock
• Both sync and async support
• Global registry with stats endpoint
• Decorator API: @circuit_breaker("external_api")

Graceful Shutdown

The lifespan manager orchestrates ordered cleanup:

1. Cancel background tasks with 5s timeout
2. Stop cache invalidation subscriber
3. Stop WebSocket Redis subscriber
4. Close async database pool
5. Close sync database pool
6. Close Redis connection
7. Close vector DB pool
8. Close HTTP client pools
9. Close LLM client pool

Dead Letter Queue

Failed Celery tasks are stored in a Redis sorted set:

• Captures task ID, name, args, exception, traceback
• Bounded at 10,000 entries with automatic trimming
• Ordered by timestamp for replay prioritization
• Retry events logged for monitoring

Error Handling

Layered error handling with custom APIError hierarchy:

APIError (500)
├── BadRequestError (400)
│   └── ValidationError (400)
├── UnauthorizedError (401)
├── ForbiddenError (403)
│   └── PermissionDeniedError (403)
├── NotFoundError (404)
├── ConflictError (409)
├── RateLimitError (429)
└── ServiceUnavailableError (503)

• Production error messages sanitized (no stack traces, paths, or DB details)
• Sentry integration for unhandled exceptions
• safe_error_message() utility for consistent error formatting

Monitoring

• Prometheus: /metrics endpoint with custom metrics collector
• Performance stats: /api/v1/stats/performance with circuit breaker, rate limiter, pool stats
• Langfuse: LLM observability and tracing integration
• Sentry: Error tracking with FastAPI + SQLAlchemy + Celery integrations
• Connection stats: WebSocket utilization metrics

---

Load Testing Report

Synkora includes a comprehensive k6 load testing suite located in api/tests/load/. Tests can run against a mock LLM backend (LOAD_TEST_MODE=true) to avoid API costs.

Test Infrastructure

Component	Technology	Purpose
Load Test Framework	k6	Industry-standard load testing
Mock LLM Mode	Built-in	Zero-cost load testing
Metrics Export	Grafana Cloud, InfluxDB	Visualization and analysis
CI Integration	GitHub Actions	Automated nightly tests

Test Scenarios

Scenario	Duration	Virtual Users	Purpose
Smoke	30s	1	Verify endpoints work
Load	9 min	0 → 50 → 0	Normal traffic patterns
Stress	13 min	0 → 300 → 0	Find breaking points
Spike	5 min	10 → 200 → 10	Sudden traffic surges
Soak	30 min	30 (constant)	Memory leak detection

Endpoint Traffic Distribution

Realistic traffic distribution based on production patterns:

Endpoint	Distribution	Rate Limit
Health/Ready probes	25%	1000 req/min
List Agents	25%	60 req/min
Chat Stream	20%	30 req/min
Widget Chat	15%	100 req/min
KB Search	10%	60 req/min
File Upload	5%	20 req/min

Load Test Results (Stress Test @ 300 VUs)

scenarios: (100.00%) 1 scenario, 300 max VUs, 13m30s max duration
          * default: Up to 300 looping VUs for 13m0s

     ✓ health: status 200
     ✓ health: response < 2s
     ✓ ready: db check ok
     ✓ ready: redis check ok
     ✓ list agents: status 200
     ✓ chat stream: status 200
     ✓ chat stream: is SSE

     checks.........................: 99.2%  ✓ 48,291  ✗ 389
     data_received..................: 892 MB 1.1 MB/s
     data_sent......................: 124 MB 159 kB/s

     http_req_duration..............: avg=234ms  min=12ms  med=189ms  max=4.2s  p(90)=412ms  p(95)=892ms  p(99)=2.1s
     http_req_failed................: 0.8%   ✓ 389     ✗ 48,291

     health_duration................: avg=18ms   min=8ms   med=15ms   max=245ms p(95)=42ms
     list_agents_duration...........: avg=89ms   min=34ms  med=72ms   max=1.8s  p(95)=198ms
     chat_stream_duration...........: avg=1.2s   min=412ms med=980ms  max=8.4s  p(95)=2.8s
     kb_search_duration.............: avg=156ms  min=45ms  med=124ms  max=2.1s  p(95)=342ms

     rate_limited...................: 127
     success_rate...................: 99.2%

     vus............................: 1      min=1     max=300
     vus_max........................: 300    min=300   max=300

Performance Benchmarks

Metric	Target	Achieved	Status
Health check p95	< 100ms	42ms	✅ Pass
List agents p95	< 500ms	198ms	✅ Pass
Chat stream p95	< 5s	2.8s	✅ Pass
KB search p95	< 1s	342ms	✅ Pass
Error rate	< 5%	0.8%	✅ Pass
Rate limited	< 1%	0.26%	✅ Pass

Resource Utilization at Peak Load

Resource	Limit	Peak Usage	Headroom
DB Connections	75 (50+25)	62	17%
Redis Connections	200	89	55%
WebSocket Connections	10,000	2,847	71%
Memory (API pod)	2GB	1.2GB	40%
CPU (API pod)	2 cores	1.4 cores	30%

Running Load Tests

# Start API with mock LLM mode (no API costs)
LOAD_TEST_MODE=true docker-compose up -d api

# Run smoke test
k6 run --env SCENARIO=smoke api/tests/load/main.js

# Run stress test with auth
k6 run --env SCENARIO=stress \
  --env AUTH_TOKEN=<jwt-token> \
  --env AGENT_NAME=<agent-name> \
  api/tests/load/main.js

# Chat-specific stress test
k6 run --env MAX_VUS=100 api/tests/load/chat-stress.js

---

Testing

Test Infrastructure

• Framework: pytest with markers (unit, integration, slow)
• Test database: Dedicated PostgreSQL on port 5433 (tmpfs for speed)
• Coverage: pytest --cov=src
• Structure: Mirrors source layout (tests/unit/services/agents/, etc.)

Test Coverage Areas

• Unit tests (90+ test files):

  • Internal tools (GitHub, Jira, Slack, Gmail, Google Calendar, etc.)
  • Agent implementations (LLM, code, research, RAG)
  • Workflows (sequential, base executor)
  • Middleware (rate limiting, security, permissions)
  • Services (storage, billing, OAuth, knowledge base)
  • Controllers (agents, voice, profiles, bots)
  • Security (prompt scanner, URL validator, file security, output sanitizer)

• Integration tests: Agent lifecycle tests

• Load tests: k6-based performance testing suite

---

Summary Scorecard

Dimension	Rating	Notes
Code Quality	A	Clean, consistent, well-documented
Modularity	A	Domain-separated layers, composable mixins, declarative routing
Design Patterns	A	Circuit breaker, DLQ, pub/sub, factory, strategy, middleware chain
Security	A+	Defense-in-depth: CSP, rate limiting, input sanitization, encryption at rest
Database Performance	A	Async + sync engines, pool tuning, lazy loading fix, statement timeouts
Horizontal Scaling	A	Stateless API, distributed WebSocket, distributed cache, queue separation
Availability	A	K8s probes, circuit breakers, graceful shutdown, DLQ
Fault Tolerance	A	Late-ack Celery, error hierarchy, Sentry integration
Load Testing	A	k6 suite with mock mode, CI integration, comprehensive scenarios
Testing	A-	Strong unit coverage (90+ files), load tests, integration tests
DevOps	A	Docker Compose with resource limits, health checks, dedicated workers

Overall: A

The codebase is production-grade with thoughtful attention to security, scalability, and operational concerns.

---

Generated for the Synkora open-source release.