Client Guide: Indexing Resources in LFX

This guide explains how to send messages to the LFX Indexer Service to index your resources in OpenSearch.

Table of Contents

• Overview
• Message Format
• Using indexing_config
• Template Support
• Field Reference
• Examples
• Best Practices

Overview

The LFX Indexer Service processes messages from NATS and indexes resources into OpenSearch for search and discovery. Clients should provide complete indexing metadata via the indexing_config field to ensure proper access control and search behavior.

Message Format

All indexing messages follow the IndexerMessageEnvelope structure:

type IndexerMessageEnvelope struct {
    Action         string                 // "created", "updated", or "deleted"
    Headers        map[string]string      // Authentication headers
    Data           any                    // Resource data (map or string for deletes)
    Tags           []string               // Optional: fields to index as tags
    IndexingConfig *IndexingConfig        // Required for create/update; omit for delete
}

Publishing Messages

Publish to NATS subjects:

• Object-specific: lfx.index.<object_type> (e.g., lfx.index.project)

Using indexing_config

Create and update messages must include the indexing_config field to provide complete indexing metadata. Delete messages omit indexing_config — the indexer only needs the object ID to remove the document. indexing_config gives you full control over how your resources are indexed and ensures proper access control.

Benefits:

• Full control over indexing behavior
• Explicit access control configuration
• Works for any object type
• Consistent indexing across all clients

Example:

{
  "action": "created",
  "headers": {
    "authorization": "Bearer <token>"
  },
  "data": {
    "uid": "proj-123",
    "name": "My Project",
    "slug": "my-project",
    "description": "A sample project"
  },
  "tags": ["uid", "slug"],
  "indexing_config": {
    "object_id": "proj-123",
    "public": true,
    "access_check_object": "project:proj-123",
    "access_check_relation": "viewer",
    "history_check_object": "project:proj-123",
    "history_check_relation": "historian",
    "sort_name": "my project",
    "name_and_aliases": ["My Project", "my-project"],
    "parent_refs": ["org:org-456"],
    "tags": ["featured", "active"],
    "fulltext": "My Project - A sample project for demonstration"
  }
}

The server will:

• Use the provided config directly
• Set server-side fields: latest, timestamps, and principals
• Index the document with your exact specifications

Template Support in indexing_config

To reduce message payload size and eliminate redundancy, indexing_config supports template variables that reference fields from the data object using {{ field_name }} syntax.

Benefits:

• Reduced Payload Size: Avoid repeating the same values multiple times
• Single Source of Truth: Data field values are the canonical source
• Less Error-Prone: No risk of mismatched values between data and config
• Type Preservation: Templates can preserve original data types

Template Syntax:

Templates use double curly braces: {{ field_name }}

Features:

1. Simple Templates: "{{ uid }}" - Entire field value (preserves type)
2. Embedded Templates: "project:{{ uid }}" - Template within a string
3. Nested Fields: "{{ parent.id }}" - Dot notation for nested objects
4. Arrays: ["{{ uid }}", "{{ parent_id }}"] - Templates in arrays
5. Multiple Templates: "{{ name }} - {{ description }}" - Multiple in one string
6. Escaping: "\{{ not_a_template }}" - Use backslash to escape

Example with Templates:

{
  "action": "created",
  "headers": {
    "authorization": "Bearer eyJhbGc..."
  },
  "data": {
    "uid": "proj-123",
    "name": "My Project",
    "slug": "my-project",
    "parent_id": "org-456",
    "metadata": {
      "organization": {
        "id": "org-789"
      }
    }
  },
  "tags": ["uid", "slug"],
  "indexing_config": {
    "object_id": "{{ uid }}",
    "public": true,
    "access_check_object": "project:{{ uid }}",
    "access_check_relation": "viewer",
    "history_check_object": "project:{{ uid }}",
    "history_check_relation": "historian",
    "sort_name": "{{ name }}",
    "name_and_aliases": ["{{ name }}", "{{ slug }}"],
    "parent_refs": ["org:{{ parent_id }}", "org:{{ metadata.organization.id }}"],
    "fulltext": "{{ name }} - Project in organization {{ metadata.organization.id }}"
  }
}

Equivalent without templates (notice the redundancy):

{
  "action": "created",
  "headers": {
    "authorization": "Bearer eyJhbGc..."
  },
  "data": {
    "uid": "proj-123",
    "name": "My Project",
    "slug": "my-project",
    "parent_id": "org-456",
    "metadata": {
      "organization": {
        "id": "org-789"
      }
    }
  },
  "tags": ["uid", "slug"],
  "indexing_config": {
    "object_id": "proj-123",
    "public": true,
    "access_check_object": "project:proj-123",
    "access_check_relation": "viewer",
    "history_check_object": "project:proj-123",
    "history_check_relation": "historian",
    "sort_name": "My Project",
    "name_and_aliases": ["My Project", "my-project"],
    "parent_refs": ["org:org-456", "org:org-789"],
    "fulltext": "My Project - Project in organization org-789"
  }
}

Template Rules:

1. Field Resolution: Templates reference fields in the data object
2. Nested Access: Use dot notation (e.g., {{ parent.id }}) for nested objects
3. Type Preservation: Simple templates ("{{ field }}") preserve the original type
4. String Conversion: Embedded templates convert values to strings
5. Error Handling: Missing fields return an error during processing
6. Escaping: Use \{{ }} to include literal curly braces in values
7. Whitespace: Whitespace inside templates is trimmed (e.g., {{ uid }} → uid)

Type Preservation Example:

{
  "data": {
    "uid": "proj-123",
    "count": 42,
    "active": true
  },
  "indexing_config": {
    "object_id": "{{ uid }}",           // Becomes: "proj-123" (string)
    "sort_name": "Count: {{ count }}",  // Becomes: "Count: 42" (string - embedded)
    ...
  }
}

Best Practices:

• Use templates for repeated values (e.g., UIDs in multiple FGA fields)
• Use nested field access to avoid flattening your data structure
• Escape templates when you need literal {{ }} in your data
• Ensure referenced fields exist in data to avoid processing errors

Field Reference

Required Fields (All Messages)

Field	Type	Description
action	string	Operation type: "created", "updated", or "deleted"
headers	object	Authentication headers (must include authorization for V2)
data	object/string	Resource data (object for create/update, string ID for delete)

Top-Level Fields

Field	Type	Required	Description
tags	array	No	Field names from data to index as searchable tags
indexing_config	object	Yes (create/update)	Indexing metadata controlling access control, search, and sort behavior. Not required for delete.

IndexingConfig Fields

Required indexing_config Fields

Field	Type	Description	Example
object_id	string	Unique identifier for the resource	"proj-123"
access_check_object	string	FGA object for access checks	"project:proj-123"
access_check_relation	string	FGA relation for access checks	"viewer"
history_check_object	string	FGA object for history checks	"project:proj-123"
history_check_relation	string	FGA relation for history checks	"historian"

Optional indexing_config Fields

Field	Type	Description	Example
public	boolean	Whether resource is publicly accessible	true
sort_name	string	Normalized name for sorting	"my project"
name_and_aliases	array	Names and aliases for search	["My Project", "MP"]
parent_refs	array	References to parent resources	["org:org-456"]
tags	array	Additional static tags for the document	["featured", "active"]
fulltext	string	Full-text search content	"searchable content"
contacts	array	Contact information for the resource (see Contact structure below)	See example below

Contact Structure

Each contact in the contacts array supports the following fields:

Field	Type	Description	Example
lfx_principal	string	LFX principal ID for the contact	"user:user-123"
name	string	Display name of the contact	"John Doe"
emails	array	Email addresses for the contact	["john@example.com"]
bot	boolean	Whether the contact is a bot account	false
profile	object	Additional profile information	{"title": "Developer"}

Example contacts array:

{
  "contacts": [
    {
      "lfx_principal": "user:user-123",
      "name": "John Doe",
      "emails": ["john@example.com", "jdoe@example.com"],
      "bot": false,
      "profile": {
        "title": "Project Lead",
        "organization": "ACME Corp"
      }
    },
    {
      "name": "Jane Smith",
      "emails": ["jane@example.com"]
    }
  ]
}

Note: Contacts support template expansion, allowing you to reference fields from the data object:

{
  "data": {
    "uid": "proj-123",
    "owner_id": "user-456",
    "owner_name": "Alice Johnson",
    "owner_email": "alice@example.com"
  },
  "indexing_config": {
    "object_id": "{{ uid }}",
    "contacts": [
      {
        "lfx_principal": "{{ owner_id }}",
        "name": "{{ owner_name }}",
        "emails": ["{{ owner_email }}"]
      }
    ]
  }
}

Server-Side Fields (Automatically Set)

These fields are always set by the server and should not be included in your message:

Field	Description	Set By
latest	Boolean flag (always true)	Server
created_at	Timestamp for create actions	Server (from message timestamp)
updated_at	Timestamp for update actions	Server (from message timestamp)
deleted_at	Timestamp for delete actions	Server (from message timestamp)
created_by	Principal(s) who created the resource	Server (from auth headers)
updated_by	Principal(s) who updated the resource	Server (from auth headers)
deleted_by	Principal(s) who deleted the resource	Server (from auth headers)
created_by_principals	Principal IDs only	Server (extracted from tokens)
updated_by_principals	Principal IDs only	Server (extracted from tokens)
deleted_by_principals	Principal IDs only	Server (extracted from tokens)
created_by_emails	Email addresses only	Server (extracted from tokens)
updated_by_emails	Email addresses only	Server (extracted from tokens)
deleted_by_emails	Email addresses only	Server (extracted from tokens)

Examples

Example 1: Create Resource

NATS Subject: lfx.index.project

{
  "action": "created",
  "headers": {
    "authorization": "Bearer eyJhbGc..."
  },
  "data": {
    "uid": "proj-123",
    "name": "My Project",
    "slug": "my-project",
    "parent_id": "org-456"
  },
  "tags": ["uid", "slug"],
  "indexing_config": {
    "object_id": "proj-123",
    "public": true,
    "access_check_object": "project:proj-123",
    "access_check_relation": "viewer",
    "history_check_object": "project:proj-123",
    "history_check_relation": "historian",
    "sort_name": "my project",
    "name_and_aliases": ["My Project", "my-project"],
    "parent_refs": ["org:org-456"],
    "tags": ["featured"],
    "fulltext": "My Project - A sample project"
  }
}

Example 2: Update Resource

NATS Subject: lfx.index.project

{
  "action": "updated",
  "headers": {
    "authorization": "Bearer eyJhbGc..."
  },
  "data": {
    "uid": "proj-123",
    "name": "Updated Project Name",
    "slug": "my-project"
  },
  "indexing_config": {
    "object_id": "proj-123",
    "public": false,
    "access_check_object": "project:proj-123",
    "access_check_relation": "viewer",
    "history_check_object": "project:proj-123",
    "history_check_relation": "historian",
    "sort_name": "updated project name",
    "name_and_aliases": ["Updated Project Name"]
  }
}

Example 3: Delete Resource

NATS Subject: lfx.index.project

{
  "action": "deleted",
  "headers": {
    "authorization": "Bearer eyJhbGc..."
  },
  "data": "proj-123"
}

Note: For delete operations, indexing_config is not needed. The server only requires the resource ID.

Best Practices

Access Control

1. FGA Pattern: Use the pattern <type>:<id>#<relation> for FGA queries

   • Example: project:proj-123#viewer

2. Access vs History: Distinguish between access and history permissions

   • access_check_*: Who can view the current state
   • history_check_*: Who can view the change history

3. Public Flag: Set appropriately to enable/disable public access

   • true: Resource is publicly searchable
   • false: Resource requires authorization

Search Optimization

1. sort_name: Normalize for consistent sorting

   • Lowercase
   • Remove special characters
   • Example: "My Project!" → "my project"

2. name_and_aliases: Include all searchable variations

   • Full name
   • Short name
   • Common abbreviations
   • Slug/URL-friendly name

3. parent_refs: Maintain resource hierarchy

   • Use format: <type>:<id>
   • Example: ["org:org-456", "team:team-789"]

4. tags: Add categorical tags

   • Keep tags concise
   • Use consistent naming conventions
   • Example: ["featured", "active", "verified"]

5. fulltext: Construct comprehensive search text

   • Include name, description, and other searchable content
   • Keep it concise but complete

Authentication

1. V2 Messages: Always include authorization header

   "headers": {
     "authorization": "Bearer <jwt-token>"
   }
   

2. Token Requirements:

   • Must be a valid JWT token
   • Validated against Heimdall service
   • Used to extract principal information for audit fields

Error Handling

The indexer service will reply with:

• "OK" on success
• "ERROR: <details>" on failure

Common errors:

• Missing required indexing_config fields
• Invalid action type
• Missing authorization header
• Malformed FGA queries

Performance Considerations

1. Message Size: indexing_config increases payload size

   • Average overhead: ~500-1000 bytes
   • Consider compression for high-volume scenarios

2. Processing Speed: indexing_config is required and drives all indexing decisions

   • Reduced server CPU usage

3. Batch Operations: Use NATS queue groups for load balancing

   • Service uses queue group: lfx.indexer.queue
   • Messages distributed across instances

Go Client Example

For Go services, use the public types package:

import (
    "github.com/linuxfoundation/lfx-v2-indexer-service/pkg/types"
    "github.com/linuxfoundation/lfx-v2-indexer-service/pkg/constants"
)

// Create message with indexing config
publicFlag := true
envelope := &types.IndexerMessageEnvelope{
    Action: constants.ActionCreated,
    Headers: map[string]string{
        "authorization": "Bearer " + token,
    },
    Data: map[string]any{
        "id":   "proj-123",
        "name": "My Project",
    },
    IndexingConfig: &types.IndexingConfig{
        ObjectID:             "proj-123",
        Public:               &publicFlag,
        AccessCheckObject:    "project:proj-123",
        AccessCheckRelation:  "viewer",
        HistoryCheckObject:   "project:proj-123",
        HistoryCheckRelation: "historian",
        SortName:             "my project",
        NameAndAliases:       []string{"My Project"},
        Tags:                 []string{"featured"},
    },
}

// Marshal and publish to NATS
data, _ := json.Marshal(envelope)
nc.Publish("lfx.index.project", data)

Troubleshooting

Issue: Document missing server-side fields

Problem: latest, created_at, or principal fields are missing in OpenSearch.

Solution: This was a bug fixed in recent versions. Ensure you're using the latest indexer service version. These fields are always set by the server automatically.

Issue: "indexing_config is required" error

Problem: Sending create/update messages without an indexing_config block.

Solution: Add indexing_config to your message. The indexer is data-agnostic and requires clients to provide complete indexing metadata.

Issue: Access control not working

Problem: Users can't access indexed resources.

Solution: Verify FGA configuration:

• Check access_check_object matches your FGA object pattern
• Verify access_check_relation is configured in FGA
• Ensure public flag is set correctly

Issue: Search not finding resources

Problem: Resources don't appear in search results.

Solution: Improve searchability fields:

• Add comprehensive name_and_aliases
• Set meaningful fulltext content
• Add relevant tags
• Verify public flag for public searches

Additional Resources

• Architecture Overview - System architecture and design patterns
• README - Project overview and setup instructions
• OpenFGA Documentation - Fine-Grained Authorization reference