Multimodal Microservice Embedder

The Multimodal Microservice Embedder generates embeddings for both text and images via an external microservice. It works with any embedding service that implements the standard API, enabling cross-modal similarity search where text and image embeddings exist in the same vector space.

Features

• Cross-modal search: Find images similar to text queries and vice versa
• Model-agnostic: Works with any embedding service implementing the standard API
• Configurable dimensions: Support for different embedding models and vector sizes
• Batch processing: Embed multiple texts (up to 100) or images (up to 20) in one request
• Cloud Run IAM auth: Optional Google Cloud Run authentication

Configuration

Configure the embedder via environment variables:

Service Connection

# Option 1: Set host and port separately (recommended)
MULTIMODAL_EMBEDDER_HOST=multimodal-embedder  # Service hostname (default)
MULTIMODAL_EMBEDDER_PORT=8000                  # Service port (default)

# Option 2: Set full URL directly (overrides host:port)
MULTIMODAL_EMBEDDER_URL=http://multimodal-embedder:8000

Vector Dimensions

The vector size must match the embedding model used by your microservice:

# Default (768) - works with CLIP ViT-L-14, many transformer models
MULTIMODAL_EMBEDDER_VECTOR_SIZE=768

# Other common dimensions
MULTIMODAL_EMBEDDER_VECTOR_SIZE=512   # CLIP ViT-B-32, smaller models
MULTIMODAL_EMBEDDER_VECTOR_SIZE=1024  # Larger models
MULTIMODAL_EMBEDDER_VECTOR_SIZE=1536  # OpenAI-compatible models

Authentication

# API key for the microservice (optional)
MULTIMODAL_EMBEDDER_API_KEY=your-api-key

Embedding Weights

When combining text and image embeddings for multimodal annotations:

# Weight for text component (default: 0.3)
MULTIMODAL_TEXT_WEIGHT=0.3

# Weight for image component (default: 0.7)
MULTIMODAL_IMAGE_WEIGHT=0.7

Example Configurations

Local Docker Compose (Default)

No configuration needed - uses defaults:

• Host: multimodal-embedder
• Port: 8000
• Vector size: 768

External Cloud Service

MULTIMODAL_EMBEDDER_HOST=embeddings.example.com
MULTIMODAL_EMBEDDER_PORT=443
MULTIMODAL_EMBEDDER_URL=https://embeddings.example.com
MULTIMODAL_EMBEDDER_VECTOR_SIZE=768
MULTIMODAL_EMBEDDER_API_KEY=sk-your-api-key

Google Cloud Run

MULTIMODAL_EMBEDDER_URL=https://embedder-abc123-uc.a.run.app
MULTIMODAL_EMBEDDER_VECTOR_SIZE=768
# IAM auth is auto-detected for Cloud Run URLs

Custom Model with Different Dimensions

# Using a model with 512-dimensional embeddings
MULTIMODAL_EMBEDDER_VECTOR_SIZE=512

API Specification

Any microservice implementing these endpoints will work with the embedder:

Endpoint	Method	Description	Request	Response
/embeddings	POST	Single text	{"text": "..."}	{"embeddings": [[...]]}
/embeddings/image	POST	Single image	{"image": "<base64>"}	{"embeddings": [[...]]}
/embeddings/batch	POST	Batch text (max 100)	{"texts": [...]}	{"embeddings": [[...], ...]}
/embeddings/image/batch	POST	Batch images (max 20)	{"images": [...]}	{"embeddings": [[...], ...]}

Response Format

All endpoints return embeddings in the same format:

{
  "embeddings": [
    [0.123, -0.456, 0.789, ...]  // Vector of configured dimension
  ]
}

For batch endpoints, the array contains one vector per input item.

Usage in Code

from opencontractserver.pipeline.embedders.multimodal_microservice import (
    CLIPMicroserviceEmbedder
)

embedder = CLIPMicroserviceEmbedder()

# Check configured vector size
print(f"Vector size: {embedder.vector_size}")

# Text embedding
text_vector = embedder.embed_text("contract clause about payment terms")

# Image embedding (base64-encoded)
image_vector = embedder.embed_image(base64_image_data)

# Batch operations
text_vectors = embedder.embed_texts_batch(["text1", "text2", "text3"])
image_vectors = embedder.embed_images_batch([base64_img1, base64_img2])

Error Handling

The embedder distinguishes between retriable and non-retriable errors:

Error Type	HTTP Status	Behavior
Client error	4xx	Not retried (invalid input)
Server error	5xx	Retried with backoff
Timeout	-	Not retried
Connection error	-	Not retried

Timeouts

• Single text/image: 30 seconds
• Batch text: 60 seconds
• Batch images: 120 seconds

Supported File Types

• PDF
• TXT
• DOCX

Building a Compatible Microservice

To build your own embedding microservice, implement the four endpoints above. Example using FastAPI:

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class TextRequest(BaseModel):
    text: str

class ImageRequest(BaseModel):
    image: str  # base64

@app.post("/embeddings")
def embed_text(req: TextRequest):
    vector = your_model.encode_text(req.text)
    return {"embeddings": [vector.tolist()]}

@app.post("/embeddings/image")
def embed_image(req: ImageRequest):
    vector = your_model.encode_image(req.image)
    return {"embeddings": [vector.tolist()]}

# ... batch endpoints similarly

See Also

• Pipeline Overview
• Docling Parser - Extracts images from PDFs