A DuckDB extension to work with RDF

Read, write and manipulate RDF within DuckDB. This extension has three broad capabilities

• reading/profiling RDF from standard serializations: Turtle, NTriples, NQuads, TriG, and RDF/XML into a standard columnar schema or pivoted based on observed predicates.
• writing using R2RML or YARRRML mappings
• querying either remote sources or using a subset of SPARQL and the R2RML/Yarrrml mappings in reverse

The extension works across all platforms that DuckDB supports however, note that RDF/XML parsing and read_sparql() (which requires OS-level networking via libcurl) are not available in WASM.

Gzip and Zst compression are supported for reads (for example .nt.gz). Zst compression requires the parquet library to be installed and loaded prior to invocation.

Installation

rdf is a DuckDB Community Extension.

To install and use the extension, run these SQL commands in your DuckDB session:

INSTALL rdf FROM community;
LOAD rdf;

That's it! The extension is now ready to use.

Quick start

-- Read multiple files with a glob pattern
SELECT COUNT(*) FROM read_rdf('shards/*.nt');

-- Execute a Sparql query over DuckDB tables using an R2RML mapping
memory D CREATE TABLE emp AS SELECT 7369 AS EMPNO, 'SMITH' AS ENAME, 10 AS DEPTNO;
memory D SELECT * FROM execute_sparql(
             'PREFIX ex: <http://example.com/ns#> SELECT ?e ?name WHERE { ?e ex:name ?name }',
             'test/r2rml/full_r2rml_lower.ttl'
         );
┌───────────────────────────────────────┬─────────┐
│                  v_e                  │ v_name  │
│                varchar                │ varchar │
├───────────────────────────────────────┼─────────┤
│ http://data.example.com/employee/7369 │ SMITH   │
└───────────────────────────────────────┴─────────┘

Sparql support is very experimental but quite interesting as it uses Duck's query optimizer. Please contribute issues with full steps to reproduce if you find cases that should work but don't. Sparql parsing and conversion to SQL is via the sql2rdf library, more information on the subset supported can be found in docs/sparql.md.

Full documentation for all the functions can be found in docs/functions.md.

Building

Managing dependencies

Where possible VCPKG is used for dependencies. However, libraries like serd are not available on VCPKG, so for these, CMake FetchContent is used instead (e.g. you need an Internet connection at build time.)

Enabling VCPKG is very simple: follow the installation instructions or just run the following:

cd <your-working-dir-not-the-plugin-repo>
git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg && git checkout ce613c41372b23b1f51333815feb3edd87ef8a8b
sh ./scripts/bootstrap.sh -disableMetrics
export VCPKG_TOOLCHAIN_PATH=`pwd`/vcpkg/scripts/buildsystems/vcpkg.cmake

Build steps

To build the extension, first clone this repo. Then in the repo base locally run:

git submodule update --init --recursive

To bring submodules up to same as upstream, run

git submodule update --recursive

To get the source for DuckDB and CI-tools. Next run:

make

If you have ninja avilable you can use that for faster builds:

GEN=ninja make

The main binaries that will be built are:

./build/release/duckdb
./build/release/test/unittest
./build/release/extension/rdf/rdf.duckdb_extension

• duckdb is the binary for the duckdb shell with the extension code automatically loaded.
• unittest is the test runner of duckdb. Again, the extension is already linked into the binary.
• rdf.duckdb_extension is the loadable binary as it would be distributed.

Cleanliness

Use make format to format all code to the DuckDB standards, make tidy-check to lint.

Running the extension

To run the extension code, simply start the shell with ./build/release/duckdb.

Running the tests

Test for this extension are SQL tests in ./test/sql. They rely on a samples in the test/rdf directory. These SQL tests can be run using:

make test

Note that the SPARQL tests require an internet connection to be able to reach out to Wikidata query service.

Installing the deployed binaries directly (e.g. not via community)

To install from GitHub actions:

• navigate to the actions for this repo
• click on the latest successful build (or build for a release)
• select the architecture you want from the left hand navigation
• open the Run actions/upload artifact step
• find the artifact URL for the compiled extension
• download, unzip and then install to DudkDB

To install your extension binaries from S3, you will need to do two things. Firstly, DuckDB should be launched with the allow_unsigned_extensions option set to true. How to set this will depend on the client you're using. Some examples:

CLI:

duckdb -unsigned

Python:

con = duckdb.connect(':memory:', config={'allow_unsigned_extensions' : 'true'})

NodeJS:

db = new duckdb.Database(':memory:', {"allow_unsigned_extensions": "true"});

Secondly, you will need to set the repository endpoint in DuckDB to the HTTP url of your bucket + version of the extension you want to install. To do this run the following SQL query in DuckDB:

SET custom_extension_repository='bucket.s3.eu-west-1.amazonaws.com/<your_extension_name>/latest';

Note that the /latest path will allow you to install the latest extension version available for your current version of DuckDB. To specify a specific version, you can pass the version instead.

After running these steps, you can install and load your extension using the regular INSTALL/LOAD commands in DuckDB:

INSTALL rdf
LOAD rdf

This repository is based on https://github.com/duckdb/extension-template, check it out if you want to build and ship your own DuckDB extension.