🦆 DuckDB Delta Sharing

The most efficient way to query Delta Lake tables directly from DuckDB. This extension implements the Delta Sharing protocol, allowing you to stream data from remote Delta Sharing servers with native performance.

---

🚀 Key Features

• 🏎️ Ultra-Fast Native Reader: Built directly on DuckDB's MultiFileReader and C++ Parquet scanner. No Python overhead.
• 🕒 Time Travel: Query your data as it existed at any point in history.
• 🔄 Change Data Feed (CDF): Easily track incremental additions, removals, and updates.
• 📉 Advanced Predicate Pushdown: Filters are pushed to the server to minimize data transfer.
• 🔒 Secure by Design: Supports OAuth/Bearer token authentication and industry-standard encryption.
• 📊 Query Telemetry: Optional SQL tracking to help administrators optimize performance.

---

🛠️ Getting Started

1. Installation

Inside your DuckDB session, run:

INSTALL duckdb_delta_sharing;
LOAD duckdb_delta_sharing;
-- Required for network access
INSTALL httpfs;
LOAD httpfs;

2. Basic Configuration

Set your sharing endpoint and bearer token using DuckDB Secrets:

CREATE SECRET (
    TYPE delta_sharing,
    PROVIDER config,
    ENDPOINT 'https://your-delta-sharing-server.com/api/2.0/delta-sharing',
    BEARER_TOKEN 'your_private_token'
);

Alternatively, if you have DELTA_SHARING_ENDPOINT and DELTA_SHARING_BEARER_TOKEN set in your environment:

CREATE SECRET (TYPE delta_sharing, PROVIDER env);

---

📂 Functional Reference

Discovering Content

List available shares, schemas, and tables:

-- List all shares
SELECT * FROM delta_share_list();

-- List schemas in a share
SELECT * FROM delta_share_list('my_share');

-- List tables in a schema
SELECT * FROM delta_share_list('my_share', 'my_schema');

-- List every table in a share, across all schemas
SELECT * FROM delta_share_list_all_tables('my_share');

-- List a table's columns (column_name, column_type, is_nullable, ordinal_position)
SELECT * FROM delta_share_list('my_share', 'my_schema', 'my_table');

Reading Tables

Query a remote Delta table just like a local file:

SELECT * FROM delta_share_read('my_share', 'my_schema', 'my_table') 
WHERE year = 2024 
LIMIT 100;

---

📖 What is Delta Sharing?

Delta Sharing is an open protocol for secure data sharing from a Delta Lake to any client. Instead of copying large datasets, the server provides the client with temporary, short-lived URLs to the underlying Parquet files.

Why use this extension?

Traditional Delta Sharing clients often rely on heavy frameworks like Spark. This extension allows DuckDB—the world's fastest analytical database—to pull those files directly into its memory. It handles:

• Partition Discovery: Automatically mapping folder structures to columns.
• Deletion Vectors: Handling row-level deletes without rewriting files.
• Schema Evolution: Safely mapping Delta types to DuckDB's native types.

---

🕒 Time Travel

Need to see how things looked yesterday? Provide a timestamp to look back in time:

-- Query data as of a specific point in time
SELECT count(*) 
FROM delta_share_read('my_share', 'my_schema', 'my_table', '2024-04-09 12:00:00');

---

🔄 Change Data Feed (CDF)

Track exactly what changed between versions. This is perfect for incremental synchronization or auditing.

-- Get all changes starting from version 10
SELECT * 
FROM delta_share_change_data_feed('my_share', 'my_schema', 'my_table', 10);

-- Query changes within a timestamp range
SELECT _change_type, _commit_version, *
FROM delta_share_change_data_feed('my_share', 'my_schema', 'my_table', '2024-04-01', '2024-04-10');

Columns like _change_type (insert, delete, update), _commit_version, and _commit_timestamp are automatically synthesized for you.

---

⚙️ Configuration Options

Setting / Secret Key	Type	Description	Default
ENDPOINT	VARCHAR	Base URL of the Delta Sharing server (Secret key)
BEARER_TOKEN	VARCHAR	Secret JWT token for authentication (Secret key)
delta_sharing_query_telemetry_enabled	BOOLEAN	Whether to send your SQL to the server for telemetry	false

📊 Query Telemetry

When enabled (false by default), the extension sends a Base64-encoded snippet of your SQL query in the delta-sharing-query-sql HTTP header. This allows server administrators to see which queries are being run and optimize data layout accordingly.

---

🤝 Developed by Prequel

This extension is maintained by Prequel. For bugs or feature requests, please open an issue in the repository.