@db2lake/core

Introduction

db2lake is a small framework for extracting data from databases and loading it into data lakes and warehouses. It provides a tiny, stable core API and a set of driver packages (sources and destinations). Drivers can be scheduled and resumed using cursor information so that only new data is transferred on subsequent runs.

This repository is a monorepo that includes the core package plus multiple source and destination drivers.

Learn more about db2lake in our post on Dev.to: Introducing db2lake: A Lightweight and Powerful ETL Framework for Node.js

Install

Install the core package:

npm install @db2lake/core

Source drivers

Purpose	Driver	Install
MySQL	@db2lake/driver-mysql	npm i @db2lake/driver-mysql
Firestore	@db2lake/driver-firestore	npm i @db2lake/driver-firestore
Postgres	@db2lake/driver-postgres	npm i @db2lake/driver-postgres
Oracle	@db2lake/driver-oracle	npm i @db2lake/driver-oracle

Destination drivers

Purpose	Driver	Install
BigQuery	@db2lake/driver-bigquery	npm i @db2lake/driver-bigquery
Databricks	@db2lake/driver-databricks	npm i @db2lake/driver-databricks
Redshift	@db2lake/driver-redshift	npm i @db2lake/driver-redshift
Snowflake	@db2lake/driver-snowflake	npm i @db2lake/driver-snowflake

Quick install example

Install the core plus the MySQL source and BigQuery destination (example):

npm install @db2lake/core @db2lake/driver-mysql @db2lake/driver-bigquery

Complete TypeScript example

The following example demonstrates a simple pipeline using the MySQL source driver and the BigQuery destination driver. It uses a transformer to adapt the source rows and a lightweight logger passed into the pipeline.

Save as examples/mysql-to-bigquery.ts and run with ts-node or compile with tsc.

import { Pipeline, ITransformer, ILogger } from '@db2lake/core';
import { MySQLSourceDriver } from '@db2lake/driver-mysql';
import { BigQueryDestinationDriver } from '@db2lake/driver-bigquery';

// --- Configure drivers (fill with your credentials) ---
const mysqlConfig = {
    query: 'SELECT * FROM orders WHERE order_id > ? LIMIT 50',
    params: [0],
    cursorField: 'order_id',
    cursorParamsIndex: 0,
    connectionUri: 'mysql://user:password@localhost:3306/shopdb'
};

const bigqueryConfig = {
    bigQueryOptions: {
        keyFilename: './service-account.json',
        projectId: 'my-project-id'
    },
    dataset: 'my_dataset',
    table: 'users',
    batchSize: 1000,
    // Optional: use streaming for real-time inserts
    writeOptions: {
        sourceFormat: 'NEWLINE_DELIMITED_JSON'
    }
};

// --- Transformer: adapt source row shape to destination schema ---
const transformer: ITransformer<any, any> = (rows) => rows.map(r => ({
    id: r.id,
    fullName: `${r.name}`,
    createdAt: r.created_at instanceof Date ? r.created_at.toISOString() : r.created_at
}));

// --- Logger ---
const logger: ILogger = (level, message, data) => {
    const ts = new Date().toISOString();
    console.log(`${ts} [${level.toUpperCase()}] ${message}`);
    if (data) console.debug(data);
};

async function main() {
    const source = new MySQLSourceDriver(mysqlConfig as any);
    const dest = new BigQueryDestinationDriver(bigqueryConfig as any);

    const pipeline = new Pipeline(source as any, dest as any, transformer, logger);

    try {
        await pipeline.run();
        console.log('Pipeline finished', pipeline.getMetrics());
    } catch (err) {
        console.error('Pipeline error', err);
    }
}

main().catch(err => { console.error(err); process.exit(1); });

Contributing

PRs that add drivers or improve the core API are welcome. Try to keep the core API minimal and well-documented so drivers remain simple to implement.

License

MIT