Developer Guide

June 30, 2026 · View on GitHub

Communication

Where to start

Understanding the project

  • Read the Technical Overview page, particularly the architecture section
  • Read the latest white papers at DTI including DTP overview
  • Understand if your organization is going to run DTP as a server, contribute an extension that adds support for your organization's API, or contribute to generic extensions that use a common API supported by your organization's servers.

Understanding the code

  • Read the README files in each top level directory
  • Integrate a new service provider or data model via the Integration Guide

Running the project

  • See the instructions below to run locally

First Run/Setup

From the top level data-transfer-project directory:

  • Install Git: sudo apt-get install git-all
  • Install Gradle: sudo apt-get install gradle
  • Install NVM: curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
  • Install Node: mac installer or nvm install v16.15.1
  • Install Angular: npm install -g @angular/cli
  • Install Docker
  • Install the Angular CLI ./gradlew client-rest:installLocalAngularCli
  • In the client-rest/ directory, run npm install

IntelliJ setup

The following instructions work for IntelliJ IDEA version 2017.2.6.

Import the project

  • Open IntelliJ -> Import Project
  • Navigate to the 'data-transfer-project' directory where you have git cloned the repo
  • Import project from external model (Gradle) -> uncheck 'Create separate module per source set' -> Finish

Enable annotation processing

  • Go to File -> Settings -> Build, Execution, Deployment -> Compiler -> Annotation Processors
  • Check 'Enable annotation processing'
  • Store generated sources relative to: 'Module content root' (not the default)
  • Production sources directory: build/classes/java/

Setup formatting

Setup automatic license header

  • Go to File -> Settings -> Editor -> Copyright -> Copyright Profiles
  • Click the green + button, add a new profile called "Data-Transfer-Project-Authors". Use this as the text:
Copyright $today.year The Data Transfer Project Authors.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
  • Create a new temp file (e.g. Test.java) in Intellij.
  • Inside the file, enter Alt-Insert -> Copyright
  • It should prompt you to select the new Copyright profile
  • The copyright should be added to all new files from now on (note: it might be collapsed so not immediately obvious)

Running locally with Docker

See Running Locally for instructions.

Running tests in Docker without a local JDK

Experimental. This workflow is new and is intended to gradually become the standard way to build and test the project, eventually replacing the requirement for a local JDK. Feedback welcome.

This is unrelated to the demo image built by dockerize above -- it's a separate, additive way to run the Java test suite without installing a JDK locally. The Dockerfile at the repo root pins gradle:8.10.2-jdk11; the source is bind-mounted at run time rather than baked into the image, so a rebuild isn't needed after every code change.

Run the full check task against the current working tree:

docker compose run --rm test

To run a specific Gradle task or test, override the default command:

docker compose run --rm test test --tests SomeTest

The gradle-cache named volume (defined in docker-compose.yml) persists resolved dependencies across runs, so only the first run pays the full resolution cost.

The JDK is pinned to 11 because the Gradle wrapper (6.9.2) can't parse Java 17 bytecode when compiling build.gradle/settings.gradle -- a gradle:*-jdk17 image fails outright for this reason.

Deploying in production

A demo distribution for Google Cloud Platform is available at distributions/demo-google-deployment.

A demo Azure distribution is also in development.

These distributions have :dockerize tasks which generate dockerfiles and build images, similar to our local demo.

Build Problem F.A.Q

AutoValue errors

If you get an error error: duplicate class... final class AutoValue_... it is indicative of your IDE and gradle clashing. To fix it you want to delete the referenced build/classes directory.

Demo server

There is a rest version of the client and demo-server in progress. While it is not complete yet and lacks some of the features in the http based client, it is still runnable and works for all services that implement OAuth2.

The following runs the client-rest api

  • cd client-rest
  • ng serve --ssl --port 3000 --proxy-config proxy.conf.json

The following builds and runs the demo-server (which contains the worker and the api) with the jettyrest transport to be used with the client-rest UI.

  • docker network create dataportability
  • ./gradlew -PtransportType=jettyrest -PapiPort=3000 -PcloudType=local clean check :distributions:demo-server:dockerize
  • docker run --rm -p 8080:8080 -p 5005:5005 -p 3000:3000 --env-file distributions/demo-server/env.secrets --name demoserver --network dataportability datatransferproject/demo

Publishing to Maven Central

See here for more details.