Data sync components for LFX One

This repository contains tools and services for synchronizing data between LFX v1 and LFX One (v2) platforms. This solution uses Meltano for data extraction and loading, a WAL listener for real-time PostgreSQL change streaming, and a sync helper service that handles data mapping and ingestion into the v2 ecosystem.

Overview

This repository serves two distinct purposes:

1. Real-time streaming replication. PostgreSQL WAL events (via wal-listener) and DynamoDB Streams are replicated in real time—alongside periodic Meltano backfills—into a v1-objects NATS KV bucket. LFX One wrapper services subscribe to this bucket to drive indexing pipelines (OpenSearch via the indexer service) and access-control pipelines (OpenFGA via fga-sync), without needing to integrate directly with ITX eventing.

2. Bidirectional sync for "core" resources. Projects and committees are fully synced in both directions between LFX v1 and LFX One. This gives LFX One a self-contained stack for these entity types, which simplifies developer environment stand-up by removing the dependency on the highly-interconnected LFX/Salesforce/ITX stack.

ITX-hosted resources such as Meetings are handled by v2 "wrapper" services that sit in front of the ITX APIs and rely on the NATS KV replication above for eventing; they do not get their own native v2 entity storage. See the ITX wrappers component diagram in the Architecture Diagrams section for how this fits together.

Prerequisites

• Python 3.12 (managed automatically by uv)
• uv package manager installed
• Access to LFX v1 data sources (DynamoDB, PostgreSQL)
• LFX One platform running via Helm

Please see each component for further setup instructions.

Repository structure

This repository contains three main components:

Meltano

Data extraction and loading pipeline that extracts data from LFX v1 sources (DynamoDB for meetings, PostgreSQL for projects/committees) and loads it into NATS KV stores for processing by the v2 platform.

v1-sync-helper

Go service that monitors NATS KV stores for replicated v1 data and synchronizes it with the LFX v2 platform APIs, handling data transformation and conflict resolution.

Helm charts

Kubernetes deployment manifests for the custom app service and WAL listener component, providing scalable deployment options for production environments.

Research & guides

• Adding a new DynamoDB table — step-by-step checklist for onboarding a new DynamoDB table into the Meltano pipeline and stream consumer, with a worked example.
• Updating the Meltano catalog ConfigMap — how to regenerate and apply the schema cache when tables or columns change.

NATS API

The v1-sync-helper service provides a NATS request/reply function for querying v1-v2 ID mappings.

Request/Reply Subject

Subject	Description
lfx.lookup_v1_mapping	Bidirectional v1↔v2 mapping lookup function

Usage

Send a NATS request to lfx.lookup_v1_mapping with the mapping key as the payload. The service will respond with the corresponding mapping value or an error.

Request Format:

Subject: lfx.lookup_v1_mapping
Payload: <mapping_key>

Response Format:

• Success: The mapped value as a string
• Not Found: Empty string ("")
• Error: String prefixed with "error: " (e.g., "error: connection timeout")

Available Lookup Patterns

Note: While called "sfid", v1 committees and committee members actually store UUIDs in their "sfid" column, so references to {*_sfid} for these entities will contain UUIDs.

The following table shows the supported mapping key patterns and their expected response formats:

Direction	Lookup Key Pattern	Example Key	Response Format	Description
Projects
v1→v2	project.sfid.{v1_sfid}	project.sfid.a0941000002wBjEAAU	{v2_uuid}	Project SFID to UUID
v2→v1	project.uid.{v2_uuid}	project.uid.123e4567-e89b-12d3-a456-426614174000	{v1_sfid}	Project UUID to SFID
Committees
v1→v2	committee.sfid.{v1_sfid}	committee.sfid.123e4567-e89b-12d3-a456-426614174003	{v2_uuid}	Committee SFID to UUID
v2→v1	committee.uid.{v2_uuid}	committee.uid.123e4567-e89b-12d3-a456-426614174001	{project_sfid}:{committee_sfid}	Committee UUID to compound SFID
Committee Members
v1→v2	committee_member.sfid.{v1_sfid}	committee_member.sfid.123e4567-e89b-12d3-a456-426614174004	{committee_uuid}:{member_uuid}	Member SFID to compound UUID
v2→v1	committee_member.uid.{v2_member_uuid}	committee_member.uid.123e4567-e89b-12d3-a456-426614174002	{project_sfid}:{committee_sfid}:{member_sfid}	Member UUID to compound SFID

User SFID Lookup API

The service also provides NATS request/reply functions for resolving v1 platform user SFIDs by username or email. These lookups query the v1 platform PostgreSQL database live, so results always reflect current data.

Subject	Description
lfx.lookup_v1_user_sfid.by_username	Lookup v1 user SFID by username
lfx.lookup_v1_user_sfid.by_email	Lookup v1 user SFID by email

Request Format:

Subject: lfx.lookup_v1_user_sfid.by_username
Payload: <username>

Subject: lfx.lookup_v1_user_sfid.by_email
Payload: <email>

Response Format:

• Success: The v1 user SFID as a string
• Not Found: Empty string ("") — includes stale index detection
• Error: String prefixed with "error: " (e.g., "error: connection timeout")

Notes:

• Lookups query salesforce.merged_user and salesforce.alternate_email__c directly with case-insensitive, whitespace-trimmed matching
• If a username/email does not resolve to a live (non-deleted) record, the lookup returns an empty string (miss)
• Callers send raw UTF-8 usernames and emails; the service normalizes (trim, lowercase, NFC) internally

Backfill / Reindex

ACS grant backfill (--backfill-acs-project / --backfill-acs-org):

Two independent flags are available:

• --backfill-acs-project — merges ACS user grants into v2 project settings (Writers, Auditors, MeetingCoordinators).
• --backfill-acs-org — merges ACS org grants into v2 b2b_org settings (Writers/Auditors). Requires MEMBER_SERVICE_URL.

Both flags are additive-only (existing v2 entries are never removed), support --dry-run to preview without writing, and are idempotent. The flags are mutually exclusive — run each as a separate Job invocation. Requires full service credentials (Auth0, Heimdall, PROJECT_SERVICE_URL; MEMBER_SERVICE_URL for the org pass).

In production, apply the Job manifest manually (not ArgoCD-managed). A dry-run pass is recommended first — add --dry-run to the manifest args, apply, inspect logs, then re-apply without it:

kubectl --context lfx-v2-prod -n v1-sync-helper apply -f manifests/backfill-acs-job.yaml

Locally (with NATS and member-service port-forwarded):

lfx-v1-sync-helper --backfill-acs-project [--dry-run]
lfx-v1-sync-helper --backfill-acs-org [--dry-run]

Committee-member reverse-mapping repair (--backfill-committee-member-mappings):

Repairs committee-member reverse mappings (committee_member.uid.* in v1-mappings) that store the platform-community__c record sfid instead of the contact SFID (v1 API "MemberID") — the root cause of v1 committee-member deletes 404ing and leaving members on the committee / meeting invites (LFXV2-2673). Resolves the contact SFID from the v1-objects KV bucket; makes no calls to any other service. Writes are optimistic-concurrency guarded against the live sync-helper, idempotent, and safe to re-run.

Run once after deploying the fix, to repair already-affected mappings. A dry-run pass is recommended first:

kubectl --context lfx-v2-prod -n v1-sync-helper apply -f manifests/backfill-committee-member-mappings-job.yaml

Add --dry-run to the manifest args, apply, inspect logs (inspected/poisoned/fixed/already_ok/unresolved/malformed/tombstoned/conflicted counts), then re-apply without it for the live run.

Locally (with NATS port-forwarded):

lfx-v1-sync-helper --backfill-committee-member-mappings [--dry-run]

Unmapped-project backfill (--backfill-projects):

Re-emits v1 projects that have no v2 mapping (LFXV2-3220 removed the project allowlist, so projects created before 17 Aug 2026 and never modified since were never synced) by re-PUTting the existing v1-objects value and letting the running deployment's KV consumer create them. Because that consumer unconditionally ACKs project messages and hard-errors when a parent's mapping is missing, this backfill re-emits parent-before-child, level by level, rather than in one flat pass. Requires only NATS (NATS_URL, AUTH0_CLIENT_ID) — it makes no v2 API calls itself.

Formation-staged projects are scoped separately by default: a live run refuses to start if it finds Formation-staged candidates unless --exclude-stage-prefix, --include-stage-prefix, or --allow-formation is passed, so the two populations are never silently mixed into one run.

kubectl --context lfx-v2-prod -n v1-sync-helper apply -f manifests/backfill-projects-job.yaml

Add --dry-run to the manifest args first, apply, inspect logs (scanned/candidates/emitted/levels/formation_candidates/remaining_unmapped counts), then run the exclude-Formation pass without it:

lfx-v1-sync-helper --backfill-projects --exclude-stage-prefix Formation --check-slugs [--dry-run]

The include-Formation pass bulk-creates checklists in lfx-v2-formation-service on first sync, so do not run it until lfx-self-serve#1957's "re-emitting a create event for a project that already has a checklist creates nothing" acceptance criterion is confirmed and the dry-run's formation_candidates count has been shared with the Formation team:

lfx-v1-sync-helper --backfill-projects --include-stage-prefix Formation --check-slugs [--dry-run]

Unmapped-committee backfill (--backfill-committees):

Re-emits v1 committees that have no v2 mapping, by re-PUTting the existing v1-objects value and letting the running deployment's KV consumer create them. handleCommitteeUpdate permanently drops (ACKs, no redelivery) a committee create whose parent project has no live mapping — the same bug pattern as the project backfill above, but with a single dependency (parent project) rather than two. Run this after --backfill-projects has completed and settled; a candidate whose parent project is still unmapped is reported as skipped_parent_unmapped rather than resolved. Requires only NATS (NATS_URL, AUTH0_CLIENT_ID) — it makes no v2 API calls itself.

--check-committee-names skips candidates whose project UID + name already resolves to a v2 committee via NATS lfx.committee-api.name_to_uid (lfx-v2-committee-service PR #209) — a lost-mapping case that needs a mapping repair, not a duplicate create.

Prerequisite: lfx-v2-committee-service PR #209 must be merged and deployed to the target environment before this job is run. --check-committee-names is on by default in the shipped manifest, and without the name_to_uid subject that PR adds, every lookup gets "no responders" and the job (including a dry run) hard-errors before reporting any counters.

kubectl --context lfx-v2-prod -n v1-sync-helper apply -f manifests/backfill-committees-job.yaml

Apply the manifest as shipped (--dry-run and --check-committee-names are on by default), inspect logs (scanned/candidates/emitted/remaining_unmapped/skipped_name_conflict counts), then delete the completed dry-run Job, remove --dry-run from its args, and apply again for the live run — kubectl apply on an existing Job whose pod template changed is rejected with field is immutable:

kubectl --context lfx-v2-prod -n v1-sync-helper delete job backfill-committees

Locally:

lfx-v1-sync-helper --backfill-committees --check-committee-names [--dry-run]

Architecture Diagrams

Regarding the following sequence diagrams:

• "Projects API" is representative of the core resources that have bidirectional sync (projects, committees). ITX-hosted resources such as Meetings are handled by wrapper services that subscribe to the NATS KV bucket instead—see the component diagram below.

ITX wrappers component diagram

This diagram shows how the LFX One platform, the v1-sync-helper replication pipeline, and ITX-hosted services fit together at the component level.

flowchart TD
    %%{init: {'flowchart': {'defaultRenderer': 'elk' }}}%%

    user[User]

    subgraph lfxv2["LFX Platform (k8s)"]
        traefik[Traefik]
        heimdall[Heimdall]
        subgraph fga-sync
        fga-sync-update-access[update-access]
        fga-sync-access-check[access-check]
        end
        indexer
        query-svc[Query Service]
        opensearch[OpenSearch]
        openfga[OpenFGA]

        xyz-wrapper@{ shape: processes, label: "Entity services (wrappers)" }

        traefik -.->|calls authz middleware| heimdall
        traefik --->|"proxies all list (search) requests to"| query-svc
        heimdall -.->|checks relations via| openfga
        query-svc -->|queries from| opensearch
        query-svc -.->|checks access via NATS| fga-sync-access-check
        indexer -.->|stores to| opensearch
        fga-sync-update-access -.->|syncs relations to| openfga
        fga-sync-access-check -.->|checks access via| openfga

        traefik -->|proxies authorized resource create/get/put requests to| xyz-wrapper

        xyz-wrapper -.->|upsert via NATS| indexer
        xyz-wrapper -.->|push relations via NATS| fga-sync-update-access

        %%wal-listener
        v1-sync-helper
        v1-objects[(v1 replica<br />KV bucket)]
        %%wal-listener -.->|NATS stream| v1-sync-helper
        v1-sync-helper -.->|NATS KV operations| v1-objects

        v1-objects -.->|subscribes to bucket events via NATS| xyz-wrapper
    end

    subgraph itx-aws[ITX AWS]
        itx-api-gw[API Gateway]
        itx-svc-authz[Authorizer Lambda]
        itx-service-xyz@{ shape: processes, label: "ITX services (Lambdas)"}
        dynamodb[(DynamoDB)]

        itx-api-gw -.-> itx-svc-authz
        itx-api-gw --> itx-service-xyz
        itx-service-xyz --> dynamodb
    end

    third-party-svcs@{ shape: processes, label: "Third-party services (Zoom, etc)"}
    itx-service-xyz --> third-party-svcs

    xyz-wrapper -->|authorized<br />create/get/put| itx-api-gw

    dynamodb -.->|consumed by streams| v1-sync-helper

    user -->|old| PIS[PIS or User Service] -->|authorized create/get/put/list| itx-api-gw
    user -->|new| traefik

Data extraction/replication sequence diagram

sequenceDiagram
    participant lfx_v1 as LFX v1 API
    participant postgres as Platform Database<br/>(PostgreSQL)
    participant wal-listener
    participant dynamodb as DynamoDB
    participant dynamo-stream as dynamodb-stream-consumer
    participant meltano as Meltano<br/>(custom NATS<br/>exporter)
    participant v1_kv as "v1" NATS KV bucket
    participant v1-sync-helper

    Note over lfx_v1,v1-sync-helper: Live data sync
    lfx_v1 ->> postgres: create/update/delete
    postgres-)+wal-listener: WAL CDC event
    Note over v1-sync-helper: Note, this is a different handler than the KV<br />bucket-updates handler below
    wal-listener-)+v1-sync-helper: notification on "wal-listener" subject
    deactivate wal-listener
    v1-sync-helper-)-v1_kv: store record (or soft-deletion) by v1 ID
    lfx_v1 ->> dynamodb: create/update/delete (via ITX API)
    dynamodb-)+dynamo-stream: DynamoDB Streams event
    dynamo-stream-)+v1-sync-helper: notification on "dynamodb_streams" subject
    deactivate dynamo-stream
    v1-sync-helper-)-v1_kv: store record (or soft-deletion) by v1 ID

    Note over lfx_v1,v1_kv: Data backfill (full sync & incremental gap-fill)
    meltano->>meltano: scheduled task invoke (weekly/monthly)
    activate meltano
    meltano->>meltano: load state from S3<br/>(incremental state bookmark)
    meltano->>+postgres: query records >= LAST_SYNC<br/>(full re-sync also supported)
    postgres--)-meltano: results
    meltano->>+dynamodb: Scan tables >= LAST_MONTH<br/>(full re-scan also supported)
    dynamodb--)-meltano: results
    loop for each record
    meltano->>+v1_kv: fetch KV item by v1 ID
    v1_kv--)-meltano: KV item, soft-deletion, or empty
    alt KV item is soft-deleted: non-null sdc_deleted_at
    Note over meltano: Avoid potential race condition if an<br />in-progress Meltano batch has a recently-updated<br />item that was just deleted via CDC live data sync
    meltano->>meltano: skip record, log notice
    else KV item empty, or item timestamp < record timestamp
    meltano-)v1_kv: store record by v1 ID
    else item timestamp > record timestamp
    Note over meltano: Handle another race condition: a recently-updated<br />item is updated again during the Meltano sync
    meltano->>meltano: skip record, log notice
    end
    end
    meltano->>meltano: save state to S3
    deactivate meltano

LFX One data-loading sequence diagram

sequenceDiagram
    participant v1_kv as "v1" NATS KV bucket
    participant v1-sync-helper
    participant mapping-db as v1/v2<br/>mapping DB<br/>(NATS KV)
    participant projects-api
    participant projects-kv as Projects NATS kv bucket
    participant openfga as OpenFGA
    participant opensearch as OpenSearch

    v1_kv-)+v1-sync-helper: notification on KV bucket subject
    v1-sync-helper->>v1-sync-helper: check if delete (hard or soft) or upsert
    v1-sync-helper->>v1-sync-helper: check if upsert was by v1-sync-helper's M2M client ID
    v1-sync-helper->>+mapping-db: check for v1->v2 ID mapping
    mapping-db--)-v1-sync-helper: v2 ID, deletion tombstone, or empty
    alt deletion tombstone exists
    Note right of v1-sync-helper: Deletes that originated in v2 and synced<br/>to v1 must NOT be re-processed FROM v1
    v1-sync-helper->>v1-sync-helper: log notice and skip record
    else item upsert & last-modified-by v1-sync-helper
    Note right of v1-sync-helper: Creations or updates that originated in<br />v2 and synced to v1 must NOT be<br />re-processed FROM v1
    v1-sync-helper->>v1-sync-helper: log notice and skip record
    else item deleted & mapping empty
    v1-sync-helper->>v1-sync-helper: not expected, log warning and skip record
    else item deleted & mapping exists
    Note right of v1-sync-helper: This is a "delete" from v1
    Note over v1-sync-helper: No v1 principal available
    v1-sync-helper ->>+ projects-api: DELETE v2 id, on-behalf-of "v1 sync" app
    projects-api -) projects-kv: delete (async)
    projects-api -) openfga: clear access control (via fga-sync)
    projects-api -) opensearch: index deletion transection (via indexer)
    Note right of v1-sync-helper: if the DELETE fails, notify team and abort
    projects-api --)- v1-sync-helper: 204 (no body)
    v1-sync-helper -) mapping-db: tombstone 🪦 v1->v2 mapping
    v1-sync-helper -) mapping-db: tombstone 🪦 v2->v1 mapping
    else item upsert & NOT last-modified-by v1-sync-helper & mapping empty
    Note right of v1-sync-helper: This is a "create" from v1
    v1-sync-helper->>v1-sync-helper: impersonate v1 principal w/ Heimdall key
    v1-sync-helper ->>+ projects-api: create (POST) on-behalf-of "v1 sync" app
    projects-api -) projects-kv: create (async)
    projects-api -) openfga: update access control (via fga-sync)
    projects-api -) opensearch: index resource (via indexer)
    Note right of v1-sync-helper: if the POST fails, notify team and abort
    projects-api --)- v1-sync-helper: 201 created (Location header, no body)
    v1-sync-helper -) mapping-db: store v2 ID (from Location header) by v1 ID
    v1-sync-helper -) mapping-db: store v1 ID by v2 ID
    else item upsert & NOT last-modified-by v1-sync-helper & mapping exists
    Note right of v1-sync-helper: This is an "update" from v1
    v1-sync-helper ->>+ projects-api: GET by v2 ID
    projects-api ->>- v1-sync-helper: data w/ etag
    v1-sync-helper->>v1-sync-helper: impersonate v1 principal w/ Heimdall key
    v1-sync-helper->>v1-sync-helper: hydrate v1 data into v2 record
    Note over v1-sync-helper: If the hydrated v2 data is unchanged,<br/>log a notice and skip the update
    v1-sync-helper ->>+ projects-api: update (PUT) on-behalf-of "v1 sync" app, if-match: etag
    projects-api -) projects-kv: update (async)
    projects-api -) openfga: update access control (via fga-sync)
    projects-api -) opensearch: index updated transaction (via indexer)
    Note right of v1-sync-helper: if the PUT fails, notify team
    projects-api --)- v1-sync-helper: 204 (no body)
    end
    deactivate v1-sync-helper

LFX One to v1 bidirectional sync

Implemented for projects, committees, committee members, and project staff. The v1-sync-helper subscribes to indexer domain events (lfx.project.*, lfx.committee.*, lfx.committee_member.*) published after every successful OpenSearch write and mirrors the change to the v1 API — projects via the Project Service v1 API (/project-service/v1/projects), committees and members via the Project Service v2 API. Project staff uses a separate projects-api event subscription; see below.

Loop detection: if a non-tombstoned reverse mapping already exists for the v2 object, the event originated from v1 and the create is skipped to prevent duplicate v1 records. On the update and delete paths the loop is broken on the v1 side by shouldSkipSync, which detects v1 records whose lastmodifiedbyid matches the v1-sync-helper's own Auth0 client ID.

Project staff sync (GH-1802)

When V2_TO_V1_PROJECT_STAFF_SYNC_ENABLED=true, the service also subscribes to lfx.projects-api.project_settings.updated (published by project-service on every settings write, with before/after snapshots) and pushes executive_director / program_manager changes back to the v1 platform via PATCH {LFX_API_GW}project-service/v1/projects/{sfid} — the same v1 contract PCC's own staff edit dialog uses, including "None" to clear an assignment. This is the return path for the LFX One staff dialog (lfx-self-serve PUT /projects/{uid}/staff, which edits one role per save).

Only the roles that actually changed in the event are sent. The v1 PATCH is partial (PCC's other project dialogs send disjoint field subsets), so an unchanged role is omitted rather than re-sent from the v1-objects replica — that replica lags v1 by the WAL pipeline, and echoing it back would revert a newer v1 assignment when two single-role edits land inside the lag window.

A role assigned in v2 that cannot be resolved to a v1 contact — LFX One allows a manual staff entry with name + email and no username, and such an email often has no merged_user row — is logged and omitted, leaving the existing v1 assignment intact. "None" is written only when the role was genuinely cleared in v2. The converse of that omission: such an entry never reaches v1, so v1's field stays empty — and v1-empty is authoritative on the next v1→v2 sync (see the GH-179 paragraph below), meaning an unresolvable v2-only entry is not durable and is cleared from the staff card by the next v1 project update.

Removals propagate in the v1→v2 direction as well (GH-179): a role cleared in PCC arrives as an empty SFID field, which the settings sync treats as a deliberate clear — the settings PUT (a full-document replace) fires with the role omitted, storing it as cleared. Previously that write was gated on at least one non-nil settings field, so a removal that arrived with no other settings change was silently dropped and the stale person lingered on the LFX One staff card.

Staff-field direction coverage:

Field	v1→v2	v2→v1	Notes
executive_director	✅	✅	Resolves through B2C merged_user by username, then email; v1→v2 propagates removals (GH-179)
program_manager	✅	✅	Resolves through B2C merged_user by username, then email; v1→v2 propagates removals (GH-179)
opportunity_owner	✅	❌	SFDC-owned; resolves through the B2B user store; one-way only; v1→v2 propagates removals (GH-179)

Echo/loop guards:

• Compare-before-write (primary). Each changed role is compared against the current v1-objects record before writing. A PCC-originated edit reaches v2 only after the v1-objects update that triggered the v1→v2 sync, so the record read back already carries the new value and the event no-ops.
• Auth0 M2M attribution. The v1 write sets lastmodifiedbyid = "{AUTH0_CLIENT_ID}@clients", which shouldSkipSync skips when the change replicates back through WAL → v1-objects KV, so our own v1 write never bounces into v2.
• Heimdall origin skip (partial). Events whose actor is {HEIMDALL_CLIENT_ID}@clients are dropped on receipt. This only covers the v1→v2 writes that fall back to the service account (empty/platform/unknown-Salesforce principals): v1→v2 normally impersonates the real v1 user, so the actor is usually that user's LFID.

Delivery: the subscription is core NATS (matching lfx.user_profile.updated), so there is no redelivery. A transient failure — v1 DB unavailable during contact resolution, or a 5xx from the gateway — is logged as an error and the event is dropped; recovering that project's staff state requires a fresh edit or a manual backfill.

sequenceDiagram
    participant lfx_v1 as LFX v1 API
    participant v1-sync-helper
    participant mapping-db as v1/v2<br/>mapping DB<br/>(NATS KV)
    participant opensearch as OpenSearch

    opensearch -)+ v1-sync-helper: v2 create/update/delete events (via indexer)
    alt transaction includes on-behalf-of "v1 sync" app
    v1-sync-helper->>v1-sync-helper: log notice and ignore
    else creates NOT on-behalf-of "v1 sync"
    v1-sync-helper->>+lfx_v1: create in v1
    lfx_v1->>-v1-sync-helper: data w/ ID
    v1-sync-helper -) mapping-db: store v1 ID (from data) by v2 ID
    v1-sync-helper -) mapping-db: store v2 ID by v1 ID
    else updates NOT on-behalf-of "v1 sync"
    v1-sync-helper->>+mapping-db: check for v2->v1 ID mapping
    mapping-db--)-v1-sync-helper: v1 ID
    v1-sync-helper->>+lfx_v1: update in v1
    lfx_v1->>-v1-sync-helper: data w/ ID
    else deletes NOT on-behalf-of "v1 sync"
    v1-sync-helper->>+mapping-db: check for v2->v1 ID mapping
    mapping-db--)-v1-sync-helper: v1 ID
    v1-sync-helper->>+lfx_v1: delete in v1
    lfx_v1->>-v1-sync-helper: 204 (no content)
    v1-sync-helper -) mapping-db: delete v1->v2 mapping
    v1-sync-helper -) mapping-db: delete v2->v1 mapping
    end
    deactivate v1-sync-helper

Combined sequence diagram

Several of the sequence diagram participants are shared in the previous diagrams. This next diagram combines the previous diagrams to help show how the data sync works holistically (in its expected, final target state).

sequenceDiagram
    participant lfx_v1 as LFX v1 API
    participant postgres as Platform Database<br/>(PostgreSQL)
    participant wal-listener
    participant dynamodb as DynamoDB
    participant dynamo-stream as dynamodb-stream-consumer
    participant meltano as Meltano<br/>(custom NATS<br/>exporter)
    participant v1_kv as "v1" NATS KV bucket
    participant v1-sync-helper
    participant mapping-db as v1/v2<br/>mapping DB<br/>(NATS KV)
    participant projects-api
    participant projects-kv as Projects NATS kv bucket
    participant openfga as OpenFGA
    participant opensearch as OpenSearch

    Note over lfx_v1,v1-sync-helper: Live data sync
    lfx_v1 ->> postgres: create/update/delete
    postgres-)+wal-listener: WAL CDC event
    Note over v1-sync-helper: Note, this is a different handler than the KV<br />bucket-updates handler below
    wal-listener-)+v1-sync-helper: notification on "wal-listener" subject
    deactivate wal-listener
    v1-sync-helper-)-v1_kv: store record (or soft-deletion) by v1 ID
    lfx_v1 ->> dynamodb: create/update/delete (via ITX API)
    dynamodb-)+dynamo-stream: DynamoDB Streams event
    dynamo-stream-)+v1-sync-helper: notification on "dynamodb_streams" subject
    deactivate dynamo-stream
    v1-sync-helper-)-v1_kv: store record (or soft-deletion) by v1 ID

    Note over lfx_v1,v1_kv: Data backfill (full sync & incremental gap-fill)
    meltano->>meltano: scheduled task invoke (weekly/monthly)
    activate meltano
    meltano->>meltano: load state from S3<br/>(incremental state bookmark)
    meltano->>+postgres: query records >= LAST_SYNC<br/>(full re-sync also supported)
    postgres--)-meltano: results
    meltano->>+dynamodb: Scan tables >= LAST_MONTH<br/>(full re-scan also supported)
    dynamodb--)-meltano: results
    loop for each record
    meltano->>+v1_kv: fetch KV item by v1 ID
    v1_kv--)-meltano: KV item, soft-deletion, or empty
    alt KV item is soft-deleted: non-null sdc_deleted_at
    Note over meltano: Avoid potential race condition if an<br />in-progress Meltano batch has a recently-updated<br />item that was just deleted via CDC live data sync
    meltano->>meltano: skip record, log notice
    else KV item empty, or item timestamp < record timestamp
    meltano-)v1_kv: store record by v1 ID
    else item timestamp > record timestamp
    Note over meltano: Handle another race condition: a recently-updated<br />item is updated again during the Meltano sync
    meltano->>meltano: skip record, log notice
    end
    end
    meltano->>meltano: save state to S3
    deactivate meltano

    Note over v1_kv,opensearch: Process watched "v1 KV bucket" item-update notification
    v1_kv-)+v1-sync-helper: notification on KV bucket subject
    v1-sync-helper->>v1-sync-helper: check if delete or upsert
    v1-sync-helper->>v1-sync-helper: check if upsert was by v1-sync-helper's M2M client ID
    v1-sync-helper->>+mapping-db: check for v1->v2 ID mapping
    mapping-db--)-v1-sync-helper: v2 ID, deletion tombstone, or empty
    alt deletion tombstone exists
    Note right of v1-sync-helper: Deletes that originated in v2 and synced<br/>to v1 must NOT be re-processed FROM v1
    v1-sync-helper->>v1-sync-helper: log notice and skip record
    else item upsert & last-modified-by v1-sync-helper
    Note right of v1-sync-helper: Creations or updates that originated in<br />v2 and synced to v1 must NOT be<br />re-processed FROM v1
    v1-sync-helper->>v1-sync-helper: log notice and skip record
    else item deleted & mapping empty
    v1-sync-helper->>v1-sync-helper: not expected, log warning and skip record
    else item deleted & mapping exists
    Note right of v1-sync-helper: This is a "delete" from v1
    Note over v1-sync-helper: No v1 principal available
    v1-sync-helper ->>+ projects-api: DELETE v2 id, on-behalf-of "v1 sync" app
    projects-api -) projects-kv: delete (async)
    projects-api -) openfga: clear access control (via fga-sync)
    projects-api -) opensearch: index deletion transection (via indexer)
    Note right of v1-sync-helper: if the DELETE fails, notify team and abort
    projects-api --)- v1-sync-helper: 204 (no body)
    v1-sync-helper -) mapping-db: tombstone 🪦 v1->v2 mapping
    v1-sync-helper -) mapping-db: tombstone 🪦 v2->v1 mapping
    else item upsert & NOT last-modified-by v1-sync-helper & mapping empty
    Note right of v1-sync-helper: This is a "create" from v1
    v1-sync-helper->>v1-sync-helper: impersonate v1 principal w/ Heimdall key
    v1-sync-helper ->>+ projects-api: create (POST) on-behalf-of "v1 sync" app
    projects-api -) projects-kv: create (async)
    projects-api -) openfga: update access control (via fga-sync)
    projects-api -) opensearch: index resource (via indexer)
    Note right of v1-sync-helper: if the POST fails, notify team and abort
    projects-api --)- v1-sync-helper: 201 created (Location header, no body)
    v1-sync-helper -) mapping-db: store v2 ID (from Location header) by v1 ID
    v1-sync-helper -) mapping-db: store v1 ID by v2 ID
    else item upsert & NOT last-modified-by v1-sync-helper & mapping exists
    Note right of v1-sync-helper: This is an "update" from v1
    v1-sync-helper ->>+ projects-api: GET by v2 ID
    projects-api ->>- v1-sync-helper: data w/ etag
    v1-sync-helper->>v1-sync-helper: impersonate v1 principal w/ Heimdall key
    v1-sync-helper->>v1-sync-helper: hydrate v1 data into v2 record
    Note over v1-sync-helper: If the hydrated v2 data is unchanged,<br/>log a notice and skip the update
    v1-sync-helper ->>+ projects-api: update (PUT) on-behalf-of "v1 sync" app, if-match: etag
    projects-api -) projects-kv: update (async)
    projects-api -) openfga: update access control (via fga-sync)
    projects-api -) opensearch: index updated transaction (via indexer)
    Note right of v1-sync-helper: if the PUT fails, notify team
    projects-api --)- v1-sync-helper: 204 (no body)
    end
    deactivate v1-sync-helper

    Note over lfx_v1,opensearch: Process v2 events
    opensearch -)+ v1-sync-helper: v2 create/update/delete events (via indexer)
    alt transaction includes on-behalf-of "v1 sync" app
    v1-sync-helper->>v1-sync-helper: log notice and ignore
    else creates NOT on-behalf-of "v1 sync"
    v1-sync-helper->>+lfx_v1: create in v1
    lfx_v1->>-v1-sync-helper: data w/ ID
    v1-sync-helper -) mapping-db: store v1 ID (from data) by v2 ID
    v1-sync-helper -) mapping-db: store v2 ID by v1 ID
    else updates NOT on-behalf-of "v1 sync"
    v1-sync-helper->>+mapping-db: check for v2->v1 ID mapping
    mapping-db--)-v1-sync-helper: v1 ID
    v1-sync-helper->>+lfx_v1: update in v1
    lfx_v1->>-v1-sync-helper: data w/ ID
    else deletes NOT on-behalf-of "v1 sync"
    v1-sync-helper->>+mapping-db: check for v2->v1 ID mapping
    mapping-db--)-v1-sync-helper: v1 ID
    v1-sync-helper->>+lfx_v1: delete in v1
    lfx_v1->>-v1-sync-helper: 204 (no content)
    v1-sync-helper -) mapping-db: tombstone 🪦 v2->v1 mapping
    v1-sync-helper -) mapping-db: tombstone 🪦 v1->v2 mapping
    end
    deactivate v1-sync-helper

License

Copyright The Linux Foundation and each contributor to LFX.

This project’s source code is licensed under the MIT License. A copy of the license is available in LICENSE.

This project’s documentation is licensed under the Creative Commons Attribution 4.0 International License (CC-BY-4.0). A copy of the license is available in LICENSE-docs.