Data Lifecycle

November 26, 2025 ยท View on GitHub

SerialMemory provides comprehensive data management capabilities including export, deletion, retention policies, and audit logging.

Data Export

Full Workspace Export

Export all tenant data including memories, entities, relationships, and optionally events.

Via API:

curl -X POST "http://localhost:5001/tenant/export" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "format": "json",
    "includeEntities": true,
    "includeRelationships": true,
    "activeOnly": true
  }'

Response:

{
  "exportId": "550e8400-e29b-41d4-a716-446655440000",
  "status": "processing",
  "estimatedCompletionAt": "2024-01-15T10:05:00Z"
}

Export Formats

FormatDescription
jsonFull JSON export with all metadata
csvTabular export (memories only)
graphmlKnowledge graph in GraphML format

Encrypted Export

For sensitive data, enable AES-256 encryption:

{
  "format": "json",
  "encrypt": true,
  "encryptionKey": "your-secret-key-32-chars-minimum"
}

Tenant Deletion

Requesting Deletion

Tenant owners can request account deletion. This initiates a 30-day grace period.

Via API:

curl -X DELETE "http://localhost:5001/tenant" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"confirmationPhrase": "DELETE MY ACCOUNT"}'

Deletion Timeline

  1. Day 0: Deletion requested, tenant marked "pending_deletion"
  2. Days 1-30: Grace period - can cancel deletion
  3. Day 30: Permanent deletion begins
    • All memories deleted
    • All entities deleted
    • All relationships deleted
    • API keys revoked
    • User accounts removed

Canceling Deletion

During the grace period:

curl -X POST "http://localhost:5001/tenant/deletion/cancel" \
  -H "Authorization: Bearer <token>"

Data Retention

Memory Confidence Decay

Memories have a confidence score that decays over time:

confidence = initial_confidence * 0.5^(days / half_life)

Default half-life: 90 days

Memories below threshold (default 0.1) are candidates for archival.

Memory Layers

LayerDescriptionTypical Retention
L0_RAWRaw input data30 days
L1_CONTEXTContextual understanding90 days
L2_SUMMARYSummarized information180 days
L3_KNOWLEDGEExtracted knowledge365 days
L4_HEURISTICLearned patternsIndefinite

Archival

Low-confidence, low-access memories are archived:

  • Moved to cold storage
  • Not included in search results
  • Can be restored if needed

Admin Audit Log

All administrative actions are logged in a tamper-evident hash chain.

Logged Actions

ActionDescription
tenant_createdNew tenant signup
api_key_createdAPI key generated
api_key_revokedAPI key revoked
data_exportedExport requested
deletion_requestedTenant deletion initiated
deletion_cancelledDeletion cancelled
settings_changedTenant settings modified

Hash Chain Integrity

Each audit log entry includes:

{
  "id": "550e8400-e29b-41d4-a716-446655440001",
  "actionType": "api_key_created",
  "tenantId": "550e8400-e29b-41d4-a716-446655440000",
  "actorId": "admin@example.com",
  "timestamp": "2024-01-15T10:00:00Z",
  "details": { "keyId": "...", "keyName": "Production" },
  "previousHash": "abc123...",
  "currentHash": "def456..."
}

The hash chain ensures:

  • No entries can be deleted
  • No entries can be modified
  • Tampering is detectable

Verifying Integrity

curl "http://localhost:5001/admin/audit/verify" \
  -H "Authorization: Bearer <admin-token>"

Returns:

{
  "status": "valid",
  "entriesChecked": 1523,
  "lastVerifiedAt": "2024-01-15T10:00:00Z"
}

Event Sourcing

All memory mutations are stored as immutable events:

EventDescription
MemoryCreatedNew memory added
MemoryUpdatedContent modified
MemoryMergedMemories combined
MemoryInvalidatedMemory soft-deleted
MemoryDecayedConfidence decreased
MemoryReinforcedMemory validated
MemoryArchivedMoved to cold storage

Event Replay

Events can be replayed to reconstruct state at any point in time:

# Export events for replay
curl "http://localhost:5001/tenant/export" \
  -H "Authorization: Bearer <token>" \
  -d '{"includeEvents": true}'

GDPR Compliance

SerialMemory supports GDPR compliance through:

  1. Data Portability: Full export in machine-readable format
  2. Right to Erasure: Tenant deletion with 30-day grace period
  3. Data Minimization: Confidence decay reduces stale data
  4. Audit Trail: Complete log of data access and modifications
  5. Consent: API key authentication represents consent

Next Steps