SIE SDK

Python client SDK for the SIE inference server.

Installation

pip install sie-sdk

Quick Start

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")

# Encode text. Results are TypedDicts — access fields by key.
result = client.encode("BAAI/bge-m3", Item(text="Hello world"))
print(result["dense"].shape)  # (1024,)

# Score items against a query with a reranker model.
# Scores come back sorted by relevance (rank 0 = most relevant).
scores = client.score(
    "BAAI/bge-reranker-v2-m3",
    query=Item(text="What is machine learning?"),
    items=[
        Item(id="doc-1", text="Machine learning is a subfield of AI."),
        Item(id="doc-2", text="Python is a programming language."),
    ],
)
for entry in scores["scores"]:
    print(entry["item_id"], entry["score"])

Generation prompts and guard verdicts

generate and stream_generate treat text-only prompts as raw continuation input. They do not render a chat template, including model settings such as enable_thinking or guardian_config. Already-rendered prompts stay unchanged. Native requests with images render the prompt and images as one user turn.

Use chat_completions or stream_chat_completions with messages for chat, instruction-based structured output (response_format), and guard checks:

answer = client.chat_completions(
    "Qwen/Qwen3-4B-Instruct-2507",
    [{"role": "user", "content": "Write a haiku about the sea."}],
    max_completion_tokens=64,
)
print(answer["choices"][0]["message"]["content"])

The worker renders the selected model's template and applies served template settings with operator configuration taking precedence over request kwargs. Granite Guardian's shipped risk dimension is harm; prose requesting a different dimension does not change that setting. Its configured threshold produces Yes (unsafe) or No (safe). A missing or invalid verdict returns invalid_guard_verdict; never treat an error or an empty response as safe. Private reasoning is hidden on both input surfaces. If it consumes the entire generation budget without usable output, the request fails with empty_model_output.

Connecting to a managed SIE platform

The examples above target a local server. For a managed SIE gateway, pass the gateway URL as base_url and your API key (sent as a Bearer token):

from sie_sdk import SIEClient

client = SIEClient(
    "https://your-gateway.example.com",
    api_key="YOUR_API_KEY",
)

Generation execution evidence

SIEClient.last_model_revision retains the X-SIE-Model-Revision response header from the latest call in the current thread. On buffered gateway responses, this is the lowercase 64-hex executed bundle/config SHA-256 when worker evidence matches the routing snapshot. It is distinct from a catalog weights revision such as a 40-hex Hugging Face commit.

Gateway SSE responses omit that header: headers are sent before terminal execution evidence is available. Fully consuming stream_generate() leaves last_model_revision as None. A successful terminal GenerateChunk may instead carry execution_identity_sha256 and execution_binding_sha256 as an optional complete pair of lowercase 64-hex SHA-256 digests. Both Python clients preserve those fields. Older or self-hosted deployments may omit both; absence is compatible, but cannot prove which deployment executed. The terminal digests are distinct from the weights revision and config hash.

Object storage and model caches

Install the storage extra to use s3://, gs://, abfs(s)://, or native Alibaba oss:// model/cache paths:

pip install 'sie-sdk[storage]'

Alibaba OSS always uses region-scoped Signature V4. Set SIE_OSS_REGION to the bucket's region (for example, eu-central-1). Set SIE_OSS_USE_INTERNAL_ENDPOINT=true only inside the matching Alibaba Cloud network; the SDK derives the HTTPS endpoint and does not accept endpoint URLs from storage paths.

On ACK, RRSA supplies ALIBABA_CLOUD_ROLE_ARN, ALIBABA_CLOUD_OIDC_PROVIDER_ARN, and ALIBABA_CLOUD_OIDC_TOKEN_FILE. The SDK requires all three together and refreshes short-lived credentials through the Alibaba Credentials client. Local operators can use that client's standard credential sources instead. Never place credentials, queries, or fragments in an oss:// URL.

oss:// is supported for model discovery, cache population, and object copies. It is deliberately not a mutable sie-config epoch store because OSS PutObject cannot provide the required non-empty ETag compare-and-swap contract; use local/PVC, S3, GCS, or Azure storage for that store.

Error handling

Server-reported errors are raised as typed exceptions from sie_sdk.client.errors; all inherit from SIEError, and errors that carry a server response expose .code and .status_code. Invalid client-side arguments (for example, a bad base_url_headers value) raise ordinary Python exceptions such as ValueError.

Several 503 codes are transient and retried automatically (see the next section for the RESOURCE_EXHAUSTED budget):

• PROVISIONING — the cluster is scaling capacity from zero. Retry is governed by wait_for_capacity: retried under provision_timeout_s when True (the default); surfaces immediately as ProvisioningError when False.
• MODEL_LOADING — the worker accepted the request and is cold-loading the target model. Retried until provision_timeout_s; raises ModelLoadingError if the budget is exhausted.
• LORA_LOADING — the requested LoRA adapter is still loading. Retried a bounded number of times; raises LoraLoadingError when the retry budget is exhausted.
• RESOURCE_EXHAUSTED — the server ran out of GPU memory and exhausted its internal recovery. Retried with bounded backoff; raises ResourceExhaustedError when retries run out. Pass max_oom_retries=0 to disable these retries and fail fast.

One generation-specific error is terminal and never retried:

• empty_model_output — the generation finished nominally but produced no visible output text (for example, private reasoning consumed the whole token budget). Tokens were genuinely consumed, so the request is not re-run. Surfaces as ServerError with code == "empty_model_output"; on streaming calls it is raised mid-stream with the gateway request id attached for correlation.

Handling resource exhaustion

The SDK automatically retries requests that the server signals as transient — model still loading, scale-from-zero in progress, or GPU memory pressure (RESOURCE_EXHAUSTED). You don't have to write retry logic for these.

What happens by default

When the server's GPU runs out of memory mid-request, the worker first attempts an internal recovery (clear cache → evict an idle sibling model → recursively halve the batch). If that succeeds you get a normal 200 response — slightly slower than usual.

If recovery is exhausted, the server returns 503 RESOURCE_EXHAUSTED with a Retry-After: 5 header. The SDK then retries with bounded exponential backoff (5s → 10s → 20s, capped at 30s, max 3 attempts). The first retry logs at WARNING so you can see it at default log levels:

WARNING sie_sdk.client.sync: Server resource exhausted, retrying in 5.0s (attempt 1/3, elapsed: 0.4s, timeout: 900.0s)

If all retries are exhausted, the SDK raises sie_sdk.client.errors.ResourceExhaustedError (a subclass of ServerError).

Tuning the behaviour

Parameter	Default	Effect
max_oom_retries=N	3	Cap on auto-retries. Pass 0 to fail fast.
provision_timeout_s=T	900 (15 min)	Total wall-clock budget. OOM retries are clamped to the remaining budget — you'll never sleep past your timeout.

Examples

Default (resilient) — recommended for most callers:

result = client.encode("BAAI/bge-m3", Item(text="Hello"))
# Auto-retries on RESOURCE_EXHAUSTED. May take up to ~35s extra
# if recovery + retries are needed.

Fail-fast (CI tests, latency-critical hot paths):

from sie_sdk.client.errors import ResourceExhaustedError

try:
    result = client.encode(
        "BAAI/bge-m3",
        Item(text="Hello"),
        max_oom_retries=0,  # No retries; surface failure immediately
    )
except ResourceExhaustedError:
    # Server is under memory pressure — fall back to a smaller model,
    # batch later, or surface to the user.
    ...

Tight wall-clock budget:

result = client.encode(
    "BAAI/bge-m3",
    Item(text="Hello"),
    provision_timeout_s=10.0,  # Total budget; OOM retries clamped to it
)

What you'll see in your logs

Server state	Client outcome	Log level
GPU OK	200, normal latency	(none)
OOM, server-side recovery succeeds	200, +1-3s latency	(none)
OOM, SDK retries succeed	200, +5-35s latency	WARNING on 1st retry
OOM, SDK retries exhausted	ResourceExhaustedError	WARNING + traceback

If you see frequent Server resource exhausted, retrying... warnings, your cluster's GPU pool is undersized for the workload. Talk to the operator running SIE — they have observability and tuning knobs (SIE_OOM_RECOVERY__*) that aren't visible from the SDK side.

License

Apache 2.0