mxr

Overview

The sync engine orchestrates data flow between providers and local core-mail state. It runs inside the daemon as a continuous background process.

Immediate sync guarantee:

• envelope + body are written to SQLite during sync
• Tantivy is updated during the same batch and committed before sync completes
• label associations, label counts, threading, and cursor maintenance happen in the same flow

Semantic chunk prep is intentionally not part of mxr-sync itself. The daemon performs that as a post-sync platform step using the message ids that sync upserted.

Sync lifecycle

Initial sync (first time an account is added)

1. Authenticate with provider
2. Fetch all labels/folders → upsert into labels table
3. Fetch messages in batches (newest first, paginated)
   • Gmail: messages.list with maxResults=100, paging through all messages
   • Store each batch of envelopes + bodies in SQLite
   • Populate label associations
   • Index each batch in Tantivy with body text
   • Commit the lexical batch
   • Parse List-Unsubscribe header and store on envelope
4. Recalculate label counts
5. Rethread accounts that do not have stable native thread ids
6. Store the initial sync cursor (Gmail: latest historyId)
7. Log sync completion in sync_log

After the daemon receives the sync outcome, it ingests semantic chunks for the newly upserted messages. If semantic retrieval is disabled, that ingest stops after chunk persistence.

Initial sync for a large mailbox (10k+ messages) may take several minutes. The daemon should:

• Sync in batches, committing each batch to SQLite (no single giant transaction)
• Make lexical search and reads available as batches arrive
• Show sync progress via the IPC protocol (TUI displays a progress indicator)

Delta sync (subsequent syncs)

1. Read stored sync cursor for account
2. Call provider's sync_messages(cursor) → get SyncBatch
   • Gmail: history.list with startHistoryId → returns only changes since last sync
   • This is what makes Gmail sync fast: typically a handful of API calls even for active inboxes
3. Apply SyncBatch to local store:
   • Upsert new/modified envelopes
   • Upsert eagerly fetched bodies
   • Update label associations
   • Delete messages marked as deleted
   • Apply label changes
   • Parse List-Unsubscribe on new messages
4. Update Tantivy index (add new docs, reindex changed docs, remove deleted)
5. Commit the lexical batch
6. Update label counts (unread_count, total_count)
7. Rethread when provider capabilities require it
8. Update sync cursor in accounts table
9. Daemon ingests semantic chunks for upserted messages
10. Notify connected clients of changes via IPC
11. Log sync in sync_log

Current lifecycle split:

• mxr-sync
  • SQLite persistence
  • lexical index updates
  • counts/threading/cursor maintenance
• daemon post-sync platform work
  • semantic chunk persistence for changed messages
  • optional embedding generation when semantic is enabled
  • rules execution on newly upserted messages

Sync loop timing

async fn sync_loop(store: Store, providers: Vec<Box<dyn MailSyncProvider>>) {
    let mut interval = tokio::time::interval(Duration::from_secs(60)); // configurable
    loop {
        interval.tick().await;
        for provider in &providers {
            if let Err(e) = sync_account(&store, provider).await {
                tracing::error!(account = %provider.account_id(), "Sync failed: {}", e);
                // Store error in sync_log, notify TUI
            }
        }
    }
}

Default: sync every 60 seconds. Configurable per-account. The TUI can also trigger immediate sync via mxr sync or a keybinding.

Error handling

Sync errors should NOT crash the daemon. They should:

• Be logged in sync_log with the error message
• Be reported to connected TUI clients (show a status indicator)
• Be retried on the next sync cycle
• Escalate to user notification if errors persist (e.g., auth expired, needs re-auth)

Conflict resolution

For the v1, use a simple strategy: last-write-wins with provider as authority.

If a message was modified both locally and remotely between syncs:

• Remote state wins for server-managed metadata (labels, read status)
• Local-only state is preserved (snooze, draft progress, saved searches)

This is simpler than full CRDT-style conflict resolution and correct for the common case. The live provider-truth surfaces today are provider IDs, sync cursors, and explicit capability flags. ProviderMeta remains a reserved/dormant escape hatch, not active runtime conflict state.

Cursor invalidation

If a Gmail cursor becomes invalid/not-found (for example, historyId too old), the current sync path:

1. logs the invalidation
2. resets the stored cursor to Initial
3. retries once as a full sync

This keeps recovery boring and explicit instead of leaving the account stuck on a bad provider cursor.

Repair behavior

Current repair paths:

• bad Gmail cursor -> reset to Initial and retry once
• label-capable account with messages but empty message_labels -> reset cursor and rebuild associations through full sync
• lexical index drift on daemon startup -> rebuild Tantivy from SQLite
• single-message GetBody on a missing, legacy, or suspicious best-effort body row -> provider hydrate and persist

Note the boundary:

• lexical repair is mandatory core-mail repair
• semantic readiness is optional platform work layered on top

Eager body fetch

Envelopes and bodies are always fetched together during sync. The SyncBatch contains Vec<SyncedMessage> where each SyncedMessage pairs an Envelope with a MessageBody.

• Gmail: batch_get_messages uses bounded concurrent messages.get(format=full) requests to get headers + body during sync
• IMAP: BODY.PEEK[] fetches the full RFC822 message; both envelope and body are parsed from it

When the user opens a message, the TUI batches body reads through ListBodies. That path reads SQLite only — no provider hydration, no network call, no normal loading state.

GetBody remains an explicit repair-capable path for single-message reads such as CLI mxr cat: if a missing, legacy, or suspicious best-effort body row is detected, the daemon may hydrate from the provider and persist the repaired row. That repair behavior is intentionally outside the TUI bulk preview path.

This approach means:

• Opening any correctly synced message is instant (pure SQLite read)
• Single-message CLI reads can repair local body drift instead of treating the cached view as final truth
• Full-text lexical search works immediately after sync (body text indexed at sync time)
• Offline access works for all synced messages
• Storage grows proportionally to mailbox size (all bodies stored)

What happens after new mail syncs in?

In normal operation:

1. the message is in SQLite immediately
2. lexical search is fresh after the sync batch commit
3. semantic chunks are persisted for the changed message
4. embeddings are generated only if semantic retrieval is enabled

So:

• mxr search ... --mode lexical is fresh as soon as sync completes
• mxr search ... --mode hybrid or --mode semantic depends on semantic enablement/profile readiness

Snooze wake loop

Runs alongside the sync loop in the daemon.

async fn snooze_waker(store: Store, sync_engine: SyncEngine) {
    let mut interval = tokio::time::interval(Duration::from_secs(60));
    loop {
        interval.tick().await;
        let now = Utc::now();
        let due = store.get_due_snoozes(now).await;
        for snoozed in due {
            // 1. Re-apply INBOX label on provider
            //    (Gmail: POST /messages/{id}/modify addLabelIds: ["INBOX"])
            sync_engine.unsnooze(&snoozed).await;

            // 2. Restore local labels
            store.restore_labels(&snoozed).await;

            // 3. Remove from snoozed table
            store.remove_snooze(&snoozed.message_id).await;

            // 4. Notify connected TUI clients
            notify_clients(Event::MessageUnsnoozed {
                message_id: snoozed.message_id
            });
        }
    }
}

Snooze keybinding flow

User presses Z on a message
  → Snooze menu appears:
    t = tomorrow 9am
    n = next Monday 9am
    w = this weekend (Saturday 10am)
    e = tonight (6pm)
    c = custom datetime prompt
  → User selects option
  → Daemon:
    1. Records current labels for the message
    2. Removes INBOX label on Gmail (archive)
    3. Inserts row into snoozed table
    4. Message disappears from inbox view

When snooze wakes, the message reappears in both mxr and Gmail's web UI. This is critical for inbox-zero workflows — the state must be consistent across clients.

For folder-backed providers, archive/restore style mutations should reconcile through provider sync instead of optimistic local label edits. That keeps store/search aligned with provider truth even when a move materializes as delete+create.

Attachment handling

Attachments are metadata-only until downloaded:

User views message with attachments
  → TUI shows: [1] invoice.pdf (2.3 MB)  [2] receipt.png (145 KB)
  → User presses 'a' then '1'
  → Daemon calls provider.fetch_attachment(message_id, attachment_id)
  → Raw bytes saved to configurable download directory
  → local_path updated in attachments table
  → User can then open with 'o' (xdg-open) or see the file path

Download directory default: ~/mxr/attachments/ (configurable).

Sync diagnostics

The sync_log table provides a history of sync operations for debugging:

-- Recent sync history for an account
SELECT started_at, finished_at, status, messages_synced, error_message
FROM sync_log
WHERE account_id = ?
ORDER BY started_at DESC
LIMIT 20;

mxr doctor reads this and reports:

• Last successful sync per account
• Any recurring errors
• Sync duration trends
• Cursor validity