duckdb-extension-template-zig

March 16, 2024 · View on GitHub

A Zig & Nix toolkit template for building extensions against multiple versions of DuckDB using Zig, C or C++.

Usage

> nix develop -c $SHELL
> duckdb -unsigned
D LOAD 'zig-out/lib/quack.duckdb_extension';
D FROM duckdb_extensions();
┌──────────────────┬─────────┬───────────┬──────────────┬────────────────────────────────────────────────────────────────────────────────────┬───────────────────┐
│  extension_name  │ loaded  │ installed │ install_path │                                    description                                     │      aliases      │
│     varchar      │ boolean │  boolean  │   varchar    │                                      varchar                                       │     varchar[]     │
├──────────────────┼─────────┼───────────┼──────────────┼────────────────────────────────────────────────────────────────────────────────────┼───────────────────┤
│ arrow            │ false   │ false     │              │ A zero-copy data integration between Apache Arrow and DuckDB                       │ []                │
...
│ quack            │ true    │           │              │                                                                                    │ []                │
...
│ visualizer       │ true    │           │              │ Creates an HTML-based visualization of the query plan                              │ []                │
├──────────────────┴─────────┴───────────┴──────────────┴────────────────────────────────────────────────────────────────────────────────────┴───────────────────┤
│ 24 rows                                                                                                                                              6 columns │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
D FROM duckdb_extensions();
> WHERE function_name ILIKE '%quack%';
┌───────────────┬─────────────┬───────────────┬───────────────┬─────────────┬─────────────┬───┬─────────┬──────────────────┬──────────────────┬──────────┬──────────────┬─────────┐
│ database_name │ schema_name │ function_name │ function_type │ description │ return_type │ … │ varargs │ macro_definition │ has_side_effects │ internal │ function_oid │ example │
│    varchar    │   varchar   │    varchar    │    varchar    │   varchar   │   varchar   │   │ varchar │     varchar      │     boolean      │ boolean  │    int64     │ varchar │
├───────────────┼─────────────┼───────────────┼───────────────┼─────────────┼─────────────┼───┼─────────┼──────────────────┼──────────────────┼──────────┼──────────────┼─────────┤
│ system        │ main        │ quack         │ scalar        │             │ VARCHAR     │ … │         │                  │ false            │ true     │         1473 │         │
├───────────────┴─────────────┴───────────────┴───────────────┴─────────────┴─────────────┴───┴─────────┴──────────────────┴──────────────────┴──────────┴──────────────┴─────────┤
│ 1 rows                                                                                                                                                    14 columns (12 shown) │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
D SELECT quack('howdy');
┌────────────────┐
│ quack('howdy') │
│    varchar     │
├────────────────┤
│ Quack howdy 🐥 │
└────────────────┘

How it Works

DuckDB is a fast in-process analytical database written in C++ that can be extended by creating and loading a dynamically linked library using the extension API. Typically extensions are written in C++ using the officially supported extension template.

But you're one of the cool kids and want to write your extension in Zig! Fortunately the Zig build system ships with a Zig, C & C++ compiler.

1. Create a project directory initialized with the multi flake template

The Nix environment generated by the flake.nix template provides a self contained Linux & MacOS development toolchain:

  • Clang (16.0.6)
  • libcxx headers (16.0.6)
  • Zig master (0.12.0-dev.3247+26e895e3d)
  • Multiple duckdb CLI & libduckdb versions linked to the same versions of libc & libcxx as the Zig compiler (v0.10.0, v0.9.2 & main)
> mkdir myextension && cd myextension
> nix flake init -t github:rupurt/duckdb-extension-template-zig#multi
> nix develop -c $SHELL

2. Implement 2 extension loader functions

When a DuckDB extension is loaded via LOAD 'myextension.duckdb_extension'; it requires 2 symbols to be defined (myextension_version & myextension_init). The value returned from *_init must match the version of DuckDB loading the extension.

We create a simple header file to expose these 2 symbols in our built extension.

3. Create a C++ bridge that calls DuckDB::ExtensionUtil

The extension utils helper plugs into DuckDB internals such as:

  • scalar functions
  • table functions
  • custom catalogs
  • much more...

The example in this repository registers a simple scalar function called quack

4. Configure the Zig build system and compile the extension

The Zig build system is configured in build.zig.

  • We'll need to add a shared library exposing the DuckDB extension hooks defined in root.zig.
  • Add the include path for the C header file exposing these hooks.
  • Don't forget the C++ bridge
  • By convention DuckDB extensions use the file suffix .duckdb_extension. Zig writes the dynamic library using the format libmyextension.[so|dylib|dll]. Add a custom install step to use the DuckDB naming convention for the extension filename.

Limitations

Currently this template can only build extensions using versions of duckdb provided by the duckdb-nix flake. The derivation built by the flake includes header files for duckdb third_party dependencies.

I have opened a Github issue to include those libraries in the nixpkgs derivation.

Development

This repository assumes you have Nix installed

> nix develop -c $SHELL
> nix develop .#v0-10-0 -c $SHELL
> nix develop .#v0-9-2 -c $SHELL
> nix develop .#main -c $SHELL

> make

Run the Zig test suite

> make test

Delete artifacts from previous builds

> make clean

Build extension binary with Zig

> make build

Run duckdb cli allowing -unsigned extensions

> make run

License

duckdb-extension-template-zig is released under the MIT license