PII-Redactor

July 8, 2026 ยท View on GitHub

Middleware-first PII redaction service for conversational systems.

This service redacts PII before text reaches an LLM, then rehydrates placeholders in the model response before returning text to end users.

V1 Scope

  • Mandatory entities: names, email, phone
  • Token format: <fn_#>, <mn1_#>, <mn2_#>, <ln_#>, <em_#>, <ph_#>
  • Isolation key: thread_id + session_id + visitor_id + client_id + assistant_id
  • API surface: REST only (/redact, /rehydrate, /session/end, /allowlist/refresh, /health)
  • Security: API key (raw or SHA-256 hash verification)
  • Detection backend: tuned deterministic heuristics by default on the slm branch
  • Default failure policy: fail-closed (per-request override available)
  • SLM defaults:
    • PII_REDACTOR_USE_GLINER=false
    • PII_REDACTOR_USE_PRESIDIO=false
    • PII_REDACTOR_REQUIRE_GLINER=false
    • PII_REDACTOR_REQUIRE_PRESIDIO=false

Token Policy

  • Tokens are scoped per isolated chat context.
  • Numeric suffixes increment per entity as new distinct values are seen in scope (<fn_1>, <fn_2>, ...).
  • Existing tokens are not overwritten by later distinct values.
  • Re-registering the same normalized value reuses its original token.
  • new_user=true still advances active_user_index for flow tracking.

Quick Start

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt
cp .env.example .env
uvicorn src.server:app --host 0.0.0.0 --port 8000 --reload

Notes:

  • Start the server from repo root so .env is auto-loaded.
  • If your runtime already injects environment variables, set PII_REDACTOR_LOAD_DOTENV=false.
  • requirements.txt is intentionally slim on the slm branch. Use requirements-full.txt only for GLiNER/Presidio experiments.

SLM Image

The slm branch is the small-image line. It excludes GLiNER, Presidio, spaCy, and their model/runtime dependencies from requirements.txt; the Docker image defaults to heuristic-only redaction.

Build normally:

docker build -t pii-redactor:slm .

Verify the running image through /health:

{
  "presidio_enabled": false,
  "gliner_enabled": false,
  "name_detection_mode": "heuristic"
}

Rollback path: deploy the previous full-detector branch/image, or install from requirements-full.txt and explicitly set PII_REDACTOR_USE_GLINER=true / PII_REDACTOR_USE_PRESIDIO=true.

Example

curl -X POST http://localhost:8000/redact \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: change-me' \
  -d '{
    "thread_id": "thread_abc123",
    "session_id": "s1",
    "visitor_id": "v1",
    "client_id": "c1",
    "assistant_id": "a1",
    "message": "My name is Jinbad Profut and my email is jin@test.com",
    "previous_assistant_message": "What is your first name?",
    "non_name_allowlist": ["Windsor", "Shadow Hills", "Old Redwood Village"],
    "failure_mode": "closed"
  }'

If you see {"detail":"Server is missing API key configuration"}:

  • Ensure .env contains either PII_REDACTOR_API_KEY or PII_REDACTOR_API_KEY_SHA256.
  • Restart uvicorn after editing .env.
  • Keep PII_REDACTOR_REQUIRE_API_KEY=true for normal operation.

Build Image Guide

If you are embedding this into another app image or running strict offline in Docker, use:

Offline Transcript Cleaning

To clean exported transcript files in-place for review, run:

./myenv/bin/python scripts/clean_transcripts.py path/to/transcript.txt

This writes a sibling file named transcript_cleaned.txt and redacts user-provided PII in both:

  • user messages
  • assistant messages that repeat that same user-provided PII

Optional allowlist inputs:

  • --community-tree path/to/community_tree.json
  • --floor-plans path/to/floor_plans.json

Integration Guide

Use this order in your app:

  1. (Optional, recommended) Call /allowlist/refresh when community/plan data changes for an assistant.
  2. Call /redact with the raw user message before sending content to your LLM.
  3. Send redacted message to the LLM.
  4. Call /rehydrate with the LLM output.
  5. Render clean text to end users.
  6. Call /session/end when the thread ends.

Required scope fields on every request:

  • thread_id (must start with thread_)
  • session_id
  • visitor_id
  • client_id
  • assistant_id (optional; defaults to <client_id>_chat_001 when omitted/blank)

Request Notes

  • /redact
    • new_user=true advances token profile (*_1 -> *_2, etc.) for the same thread scope.
    • previous_assistant_message improves prompted-name handling.
    • failure_mode supports closed or open (default inherits server setting).
    • include_replacements=true only returns raw replacements when PII_REDACTOR_ALLOW_RAW_REPLACEMENTS=true.
  • /rehydrate
    • Use failure_mode="closed" for user-facing flows.
    • Use failure_mode="open" only for internal redacted-only tooling.
  • /allowlist/refresh
    • Stores a per-client_id+assistant_id non-name allowlist in local cache files.
    • Rewrites cache file only when extracted term content changes.
    • Supports direct terms or selector-based extraction from arbitrary JSON payloads.

Failure Policy

  • Server default is fail-closed (PII_REDACTOR_FAIL_CLOSED_DEFAULT=true).
  • In fail-closed mode, unavailable redaction/rehydration returns HTTP 503.
  • In fail-open mode, service returns passthrough text.
  • Persistence write failures are retried with cooldown-based recovery before the service stays blocked.
  • Transient persistence save failures report degraded_nonblocking; existing in-memory scopes can continue, but missing-vault loads, queue-full saves, and auth/schema/config failures still fail closed.

Logging

  • PII_REDACTOR_LOG_LEVEL controls application logging and falls back to shared LOG_LEVEL when unset.
  • PII_REDACTOR_LOG_FORMAT accepts json or text and falls back to shared LOG_FORMAT.
  • PII_REDACTOR_LOG_JSON=true or LOG_JSON=true enables JSON logs when no explicit format is set.
  • PII_REDACTOR_ACCESS_LOGS=false by default suppresses Uvicorn 2xx request access logs.
  • INFO logs are reserved for startup and operational state changes; successful request-level timing and 2xx request details are DEBUG.
  • WARNING/ERROR logs remain visible at INFO level and cover saturation, persistence pressure, blocked requests, rejected allowlist refreshes, and unexpected endpoint failures.
  • Production observability should use LOG_FORMAT=json so Grafana/Loki can extract level, logger, environment, app_role, request_id, client_id, assistant_id, and message.

Memory + Persistence Behavior

  • In-memory scope cache is bounded:
    • PII_REDACTOR_MAX_ACTIVE_SCOPES (default 15)
    • PII_REDACTOR_VAULT_TTL_SECONDS (default 3600)
  • If persistence is configured:
    • Writes are queued asynchronously (non-blocking request path)
    • Queue pressure or persistence health can force fail-closed behavior
    • Rehydrate resolves memory first, then persistence fallback

Local Allowlist Cache

  • PII_REDACTOR_ALLOWLIST_CACHE_ENABLED=true enables local per-assistant cache.
  • Cache key: client_id + assistant_id.
  • Cache file writes are atomic and content-hash based (unchanged refreshes do not rewrite files).
  • /redact automatically merges:
    • cached allowlist terms (if present)
    • request non_name_allowlist terms (if provided)

Refresh Payload Selectors

Use selectors to extract terms from varying JSON schemas or table-shaped payloads:

  • selector: path expression with support for:
    • . key traversal
    • * wildcard child selection
    • ** recursive descent
    • [index] list index
  • include:
    • values: collect string values
    • keys: collect object keys
    • both: collect both

Examples:

  • Floor plans (extract only name fields):
{
  "client_id": "c1",
  "assistant_id": "a1",
  "payload": {"rows":[{"name":"Cypress II"},{"name":"Hampton II"}]},
  "selectors": [{"selector":"**.name","include":"values"}],
  "source_version": "fp_2026-04-14T10:00:00Z"
}
  • Community tree (extract keys):
{
  "client_id": "c1",
  "assistant_id": "a1",
  "payload": {"Windsor":{"Old Redwood Village":[]}},
  "selectors": [{"selector":"**","include":"keys"}],
  "source_version": "community_2026-04-14T10:00:00Z"
}

Persistence Mode Selector

Use PII_REDACTOR_PERSISTENCE_MODE:

  • none
    • In-memory only.
    • No DB integration.
  • internal
    • Redactor process owns DB credentials/config.
    • Current internal implementation supports Supabase (PII_REDACTOR_INTERNAL_STORE_IMPL=supabase).
  • external
    • Host app controls persistence implementation.
    • Provide PII_REDACTOR_EXTERNAL_STORE_FACTORY=<module>:<callable> or inject store in-process.

Internal Supabase Required Env

When PII_REDACTOR_PERSISTENCE_MODE=internal and PII_REDACTOR_INTERNAL_STORE_IMPL=supabase, set:

  • PII_REDACTOR_SUPABASE_URL
  • PII_REDACTOR_SUPABASE_SERVICE_ROLE_KEY
  • PII_REDACTOR_SUPABASE_TABLE (default pii_vault_snapshots)
  • PII_REDACTOR_PERSISTENCE_MASTER_KEY (required for encrypted payloads)
  • PII_REDACTOR_PERSISTENCE_KEY_VERSION (for key rotation)
  • PII_REDACTOR_SUPABASE_REQUEST_TIMEOUT_SECONDS (default 15)

Recommended with fail-closed:

  • PII_REDACTOR_REQUIRE_PERSISTENCE=true
  • PII_REDACTOR_PERSISTENCE_BLOCK_ON_ERROR=true

External Mode Required Env

When PII_REDACTOR_PERSISTENCE_MODE=external, set:

  • PII_REDACTOR_EXTERNAL_STORE_FACTORY=<module>:<callable>

Factory callable contract:

  • Returns object with methods: load(scope), save(scope, snapshot, expires_at_epoch, key_version), delete(scope)
  • Can accept zero args or one settings arg.

Deployment Patterns

Single-instance mode (simplest):

  • Run redactor side-by-side with your chat backend.
  • In-memory scope cache handles active threads.

Multi-instance mode (recommended for scale):

  • Use a shared persistence backend so any instance can rehydrate.
  • Keep thread_id stable per conversation.
  • Limit concurrent /redact and /rehydrate work with PII_REDACTOR_REDACT_MAX_CONCURRENCY, PII_REDACTOR_REHYDRATE_MAX_CONCURRENCY, and PII_REDACTOR_CONCURRENCY_ACQUIRE_TIMEOUT_SECONDS. Saturated requests return 503 quickly instead of accumulating indefinitely.
  • Monitor /health fields:
    • redact_active, rehydrate_active
    • redact_max_concurrency, rehydrate_max_concurrency
    • redact_saturated_count, rehydrate_saturated_count
    • persistence_queue_depth, persistence_queue_max
    • persistence_blocking_requests
    • status
    • persistence_status
    • persistence_state
    • persistence_enabled
    • persistence_healthy
    • persistence_worker_alive
    • persistence_worker_restart_count
    • persistence_last_worker_restart_at
    • persistence_last_error_type
    • persistence_last_error_category
    • persistence_last_error_status_code
    • persistence_last_error_operation
    • persistence_last_error_at
    • persistence_last_success_at
    • persistence_unhealthy_since
    • persistence_recovery_attempts
    • persistence_next_recovery_at
    • persistence_queue_depth
    • performance_metrics

Name Tuning Hooks

  • previous_assistant_message (optional): improves one-word name handling by only treating single-word replies as names when prior assistant text asked for a name.
  • non_name_allowlist (optional): per-request city/community/domain terms that should not be treated as person names.
  • Environment defaults:
    • PII_REDACTOR_NON_NAME_TERMS (CSV)
    • PII_REDACTOR_NON_NAME_TERMS_JSON_PATH (JSON tree path; keys/values are flattened into non-name terms)

Supabase Persistence Guidance

Use encrypted persistence behind a vault-store interface:

  • Encrypt values before writing to Supabase (AES-GCM with per-record nonce)
  • Keep encryption keys outside DB (environment/KMS)
  • Store key version metadata for rotation (PII_REDACTOR_PERSISTENCE_KEY_VERSION)
  • Add TTL and explicit delete paths for session end
  • Never log raw PII

Suggested table schema for internal Supabase mode:

create table if not exists public.pii_vault_snapshots (
  scope_key text primary key,
  thread_id text not null,
  session_id text not null,
  visitor_id text not null,
  client_id text not null,
  assistant_id text not null,
  key_version text not null,
  expires_at timestamptz not null,
  payload jsonb not null,
  updated_at timestamptz not null default now()
);

create index if not exists idx_pii_vault_snapshots_scope
  on public.pii_vault_snapshots (client_id, assistant_id, visitor_id, session_id, thread_id);

create index if not exists idx_pii_vault_snapshots_expires_at
  on public.pii_vault_snapshots (expires_at);

Example .env (Internal Supabase)

PII_REDACTOR_PERSISTENCE_MODE=internal
PII_REDACTOR_INTERNAL_STORE_IMPL=supabase
PII_REDACTOR_REQUIRE_PERSISTENCE=true
PII_REDACTOR_PERSISTENCE_BLOCK_ON_ERROR=true
PII_REDACTOR_PERSISTENCE_RECOVERY_COOLDOWN_SECONDS=30
PII_REDACTOR_SUPABASE_URL=https://YOUR_PROJECT.supabase.co
PII_REDACTOR_SUPABASE_SERVICE_ROLE_KEY=YOUR_SERVICE_ROLE_KEY
PII_REDACTOR_SUPABASE_TABLE=pii_vault_snapshots
PII_REDACTOR_PERSISTENCE_MASTER_KEY=LONG_RANDOM_MASTER_KEY
PII_REDACTOR_PERSISTENCE_KEY_VERSION=v1
PII_REDACTOR_SUPABASE_REQUEST_TIMEOUT_SECONDS=15

Notes

  • PII-redactor-plan.v2.md is preserved as the planning reference.
  • Runtime health endpoint includes detector status and persistence diagnostics so you can verify detector load and persistence recovery state.
  • On the slm branch, Presidio/GLiNER are not installed by default and the engine starts directly in heuristic mode.
  • If you install requirements-full.txt, PII_REDACTOR_REQUIRE_GLINER=true or PII_REDACTOR_REQUIRE_PRESIDIO=true still makes startup fail if the required detector is unavailable.
  • For full-detector strict air-gap mode, keep:
    • PII_REDACTOR_GLINER_ALLOW_REMOTE_DOWNLOAD=false
    • PII_REDACTOR_PRESIDIO_MINIMAL_RECOGNIZERS=true