Danish Democracy Data

March 9, 2025 ยท View on GitHub

This repo is a personal hobby project using open source alternatives to create a data warehouse for the data of the Danish parliament. The API used is the open API from Folketinget API as the primary source of data.

Development

Use curl to download the script and execute it with sh:

macOS and Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

windows:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Running this project

Following commands create and activate a virtual environment and run the project.

  • Bash: 
    curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/bgarcevic/danish-democracy-data.git
cd danish-democracy-data
uv run python extract/danish_parliament_data_retriever.py
cd dbt
uv run dbt deps
uv run dbt build
uv run dbt docs generate
uv run dbt docs serve
  • PowerShell: 
    powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
git clone https://github.com/bgarcevic/danish-democracy-data.git
cd danish-democracy-data
uv run python extract\danish_parliament_data_retriever.py
cd dbt
uv run dbt deps
uv run dbt build
uv run dbt docs generate
uv run dbt docs serve
  • Windows CMD: 
    powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
git clone https://github.com/bgarcevic/danish-democracy-data.git
cd danish-democracy-data
uv run python extract\danish_parliament_data_retriever.py
cd dbt
dbt deps
dbt seed
dbt build
dbt docs generate
dbt docs serve

Development Tools

  • Code formatting: ruff
  • Add-in: dbt Power User
  • SQL linting/formatting: sqlfluff

File Locations

DuckDB location

DuckDB file will be in project-root under danish_democracy_data.duckdb

Staging:

Run

  • PowerShell: 
    uv run extract\danish_parliament_pipeline.py

dbt

Install dbt_utils:

dbt deps

Run models:

dbt run

Run tests:

dbt test

sqlfluff

Run SQL linter on dbt models:

sqlfluff lint

export the data to parquet

Change the materialization to 'external' in dbt_projects.yml for the mart layer and run dbt build.

Read more here.

Browsing the data

Some options:

For more information on dbt: