S3-Compatible Storage

duck_net provides functions for interacting with any S3-compatible object storage (AWS S3, MinIO, Cloudflare R2, etc.) using AWS Signature v4 authentication.

Functions

Function	Parameters	Returns
s3_get	(endpoint, bucket, key, access_key, secret_key, region)	STRUCT(success BOOLEAN, body VARCHAR, status INTEGER, message VARCHAR)
s3_put	(endpoint, bucket, key, body, access_key, secret_key, region)	STRUCT(success, body, status, message)
s3_list	(endpoint, bucket, prefix, access_key, secret_key, region)	STRUCT(success BOOLEAN, keys VARCHAR[], message VARCHAR)
s3_get_secret	(secret_name, bucket, key)	STRUCT(success, body, status, message)
s3_put_secret	(secret_name, bucket, key, data)	STRUCT(success, body, status, message)
s3_list_secret	(secret_name, bucket, prefix)	STRUCT(success, keys, message)

Reading Objects

-- Read an object from S3
SELECT (s3_get(
    'https://s3.amazonaws.com',
    'my-bucket', 'data/report.json',
    'AKIAIOSFODNN7EXAMPLE', 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
    'us-east-1'
)).body;

Writing Objects

-- Upload an object
SELECT (s3_put(
    'https://s3.amazonaws.com',
    'my-bucket', 'data/output.json',
    '{"results": [1, 2, 3]}',
    'AKIAIOSFODNN7EXAMPLE', 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
    'us-east-1'
)).success;

Listing Objects

-- List objects with a prefix
SELECT (s3_list(
    'https://s3.amazonaws.com',
    'my-bucket', 'data/',
    'AKIAIOSFODNN7EXAMPLE', 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
    'us-east-1'
)).keys;

Using Secrets (Recommended)

-- Store S3 credentials
SELECT duck_net_add_secret('aws', 's3', '{
    "key_id": "AKIAIOSFODNN7EXAMPLE",
    "secret": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "region": "us-east-1",
    "endpoint": "https://s3.amazonaws.com"
}');

-- Use the secret
SELECT (s3_get_secret('aws', 'my-bucket', 'data/report.json')).body;
SELECT (s3_put_secret('aws', 'my-bucket', 'data/output.json', '{"key": "value"}')).success;
SELECT (s3_list_secret('aws', 'my-bucket', 'data/')).keys;

S3-Compatible Services

Works with any S3-compatible endpoint:

-- MinIO
SELECT duck_net_add_secret('minio', 's3', '{
    "key_id": "minioadmin", "secret": "minioadmin",
    "region": "us-east-1", "endpoint": "http://localhost:9000"
}');

-- Cloudflare R2
SELECT duck_net_add_secret('r2', 's3', '{
    "key_id": "...", "secret": "...",
    "region": "auto", "endpoint": "https://ACCOUNT_ID.r2.cloudflarestorage.com"
}');

DuckDB Native Secrets

duck_net also integrates with DuckDB's built-in secrets manager. See DuckDB Native Secrets for details on using CREATE SECRET with S3.

Security Considerations

• AWS Signature v4 ensures credentials are never sent in plaintext.
• Never embed access keys in SQL. Use the secrets manager.
• S3 endpoints are validated against SSRF rules.
• Session tokens are supported via the session_token secret key.