Next generation of AcoustID fingerprint index

Key changes from the first version

• Written in Zig instead of C++
• Supports updates and deletes
• HTTP API that can be used for managing multiple indexes
• Fast and concurrent write operations using WAL and in-memory segments
• Simplified internal file format, using msgpack for serialization wherever possible
• StreamVByte compression for efficient storage of posting lists with SIMD-accelerated decoding

Building

Building from source code:

zig build

Running tests:

zig build unit-tests --summary all
zig build e2e-tests --summary all

Test runner environment variables:

• TEST_VERBOSE=false - Disable verbose output (default: true)
• TEST_FAIL_FIRST=true - Stop on first failure
• TEST_FILTER=substring - Filter tests by name

Running server:

zig build run -- --dir /tmp/fpindex --port 8080 --log-level debug

HTTP API

The API supports both JSON and MessagePack formats. Use Content-Type: application/json or Content-Type: application/vnd.msgpack for requests, and Accept: application/json or Accept: application/vnd.msgpack for responses. JSON is the default if no headers are specified.

Index Management

Check if index exists

Returns HTTP status 200 if the index exists, 404 if not found.

HEAD /:indexname

Get index information

Returns detailed information about an index including version, metadata, and statistics.

GET /:indexname

Response:

{
  "version": 1,
  "metadata": {
    "key1": "value1",
    "key2": "value2"
  },
  "stats": {
    "min_doc_id": 1,
    "max_doc_id": 999,
    "num_segments": 2,
    "num_docs": 1000
  }
}

Create index

Creates a new index. Returns empty JSON object {} on success.

PUT /:indexname

Delete index

Deletes an index and all its data. Returns empty JSON object {} on success.

DELETE /:indexname

Get segment information

Returns detailed information about index segments (memory and file segments).

GET /:indexname/_segments

Response:

{
  "segments": [
    {
      "kind": "memory",
      "version": 1,
      "merges": 0,
      "min_doc_id": 1,
      "max_doc_id": 100
    },
    {
      "kind": "file", 
      "version": 2,
      "merges": 1,
      "min_doc_id": 101,
      "max_doc_id": 999
    }
  ]
}

Create index snapshot

Creates a downloadable tar archive containing complete index data for backup.

GET /:indexname/_snapshot

Fingerprint Operations

Bulk update operations

Performs multiple insert/delete operations atomically. This is the preferred method for bulk operations.

POST /:indexname/_update

Request:

{
  "changes": [
    {"insert": {"id": 12345, "hashes": [100, 200, 300, 400, 500]}},
    {"insert": {"id": 67890, "hashes": [150, 250, 350]}},
    {"delete": {"id": 11111}}
  ],
  "metadata": {
    "key1": "value1",
    "key2": "value2"
  }
}

Parameters:

• changes (required): Array of insert/delete operations to perform
• metadata (optional): Key-value pairs of string metadata to associate with the index

Response: Empty JSON object {}

Search for fingerprints

Searches for fingerprints by finding documents that contain subsets of the query hashes. Returns results ranked by the number of matching hashes (score).

POST /:indexname/_search

Request:

{
  "query": [100, 200, 300, 400, 500],
  "timeout": 1000,
  "limit": 20
}

Parameters:

• query (required): Array of 32-bit unsigned integers representing hash values
• timeout (optional): Search timeout in milliseconds (default: 500, max: 10000)
• limit (optional): Maximum number of results to return (default: 40, min: 1, max: 100)

Response:

{
  "results": [
    {"id": 12345, "score": 5},
    {"id": 67890, "score": 3}
  ]
}

Score calculation: The score represents the number of query hashes that match hashes in the fingerprint. Higher scores indicate better matches.

Minimum score threshold: Results must have at least (query_length + 19) / 20 matching hashes and at least 10% of the query hashes.

Check if fingerprint exists

Returns HTTP status 200 if the fingerprint exists, 404 if not found.

HEAD /:indexname/:fpid

Get fingerprint information

Returns metadata about a fingerprint. Note: Original hashes cannot be retrieved as they are stored in an optimized format.

GET /:indexname/:fpid

Response:

{
  "version": 1
}

Insert/update single fingerprint

Inserts or updates a single fingerprint. For bulk operations, prefer using /_update.

PUT /:indexname/:fpid

Request:

{
  "hashes": [100, 200, 300, 400, 500]
}

Response: Empty JSON object {}

Delete single fingerprint

Deletes a single fingerprint. For bulk operations, prefer using /_update.

DELETE /:indexname/:fpid

Response: Empty JSON object {}

System Utilities

Health check

Returns "OK" if the service is healthy.

GET /_health

Index health check

Returns "OK" if the specific index is ready and healthy.

GET /:indexname/_health

Prometheus metrics

Returns metrics in Prometheus format for monitoring.

GET /_metrics

Example metrics:

# TYPE aindex_search_hits_total counter
aindex_search_hits_total 1250
# TYPE aindex_search_misses_total counter  
aindex_search_misses_total 45
# TYPE aindex_search_duration_seconds histogram
aindex_search_duration_seconds_bucket{le="0.005"} 100

Error Responses

All endpoints return structured error responses in case of failures:

{
  "error": "IndexNotFound"
}

Common error codes:

• 400 Bad Request: Invalid request body, missing parameters, or malformed data
• 404 Not Found: Index or fingerprint not found
• 415 Unsupported Media Type: Invalid Content-Type header
• 500 Internal Server Error: Server-side processing error
• 503 Service Unavailable: Index not ready (still loading)