DIVE Python Packages

June 12, 2026 · View on GitHub

There are several important python packages in this application

  • dive_server is a collection of girder plugins for the web server
  • dive_tasks is a collection of girder worker plugins for the celery worker
  • scripts has general command-line utilities
  • dive_utils is shared code between the above packages

Girder 5 stack

The server targets Girder 5 (girder, girder_jobs, girder_worker, and related plugins pinned in pyproject.toml). Docker Compose adds Redis for notifications (GIRDER_NOTIFICATION_REDIS_URL); the web client receives job updates over WebSockets.

Docker Compose runs a required localworker service on the local queue in addition to the standard workers. With docker-compose.override.yml, that service mounts your local server/ code for development.

Upgrading an existing deployment: Upgrading to Girder 5.

Prerequisites

Set up your system as described in the Basic Deployment

Development

In development, the server and client are run in separate processes. In production, the client is built and bundled as static files into the server image.

This python project uses uv for Python dependency management.

# Optional, for intellisense or whatever.  Not required for docker-compose
uv sync

# Include dev tools (tox, tox-uv, MkDocs, …) so `uv run tox` and doc targets work from this directory
uv sync --group dev

Running in development with docker

# Copy .env.default and make any changes
cp .env.default .env

# Option 1) Build the project from source
docker-compose build
# Option 2) Pull pre-build images
docker-compose pull

# Start the project in default mode (GPU-enabled workers)
docker-compose up -d

# Start the project in CPU-only mode
docker-compose --profile cpu up -d

# The web server has hot reload, so code changes will
# immediately trigger a server restart.

# The Celery workers do not have hot reload.
# To test code changes, a restart is needed
docker-compose up girder_worker_default
# or
docker-compose up girder_worker_pipelines
# or
docker-compose up girder_worker_training

Access the server at http://localhost:8010

Worker profiles and feature availability

Docker Compose now separates workers by profile:

  • Default mode: runs girder_worker_default plus girder_worker_pipelines and girder_worker_training.
  • --profile cpu: runs girder_worker_default only.

If GPU workers are not present, DIVE disables pipeline/training actions in the web interface and rejects related API job launch requests.

To work on the Vue client, see development instructions in ../client.

Unit Testing and Static Checks

Automation is done with Tox with tox-uv.

# run only lint checks
uv run tox -e lint

# run only type checks
uv run tox -e type

# run only unit tests
uv run tox -e testunit

# run only a particular test
uv run tox -e testunit -- -k test_image_sort

# run all three tests above
uv run tox

# automatically format all code to comply to linting checks
uv run tox -e format

# run MkDocs dev server (from ./server; default http://127.0.0.1:8090/ — use `tox -e docs -- -a HOST:PORT` to override)
uv run tox -e docs

# one-shot static build into ../site at the repository root
uv run tox -e builddocs

Integration Testing

Get an API key from production Girder. DO NOT use a full-scoped token, use a read only token.

# start the server
docker-compose up -d

# set an API key from production girder
export GIRDER_API_KEY=CHANGEME

# run the tests
uv run tox -e testintegration

# run only a particular test (be mindful of dependencies)
uv run tox -e testintegrationkeyword test_pipelines

After integration tests are complete, visually inspect the results to make sure all jobs completed, new datasets open correctly, etc.

Debug utils and command line tools

# Requires a local uv installation
uv sync

# show options
uv run dive --help

# build the standalone executable into ./dist
uv run tox -e buildcli

Metadata properties

This section explains the metadata properties used to record application state in Girder. These properties can be modified through the Girder UI editor.

Dataset

Image chips that compose a video are stored as girder items in a folder. Videos are stored as a single item in its own folder. The parent folder must have the following metadata.

  • annotate (boolean) marks a folder as a valid DIVE dataset
  • type ('video' | 'image-sequence') dataset type
  • fps (number) annotation framerate, not to be confused with video raw framerate
  • ffprobe_info (JSON) output of ffprobe for raw input video
  • confidenceFilters (JSON) map of filter name to float in [0, 1]
  • imageEnhancements (JSON) values for image enhancements (brightness, contrast, saturation, sharpen)
  • customTypeStyline (JSON) map of class name to GeoJS display attributes.
  • foreign_media_id (string) For "cloned" datasets, this is an objectId pointer to the source media

Video Item

  • codec (string) video codec
  • source_video (boolean) whether the video is a raw user upload or a trancoded video

uv Lockfile Refresh

The girder-large-image plugin dependencies are hosted externally and can occasionally shift beneath pinned resolution. If a build fails because of stale lock metadata, clear uv/pip caches and regenerate the lockfile from the server folder:

uv cache clean && pip cache purge && rm -rf ~/.cache/uv ~/.cache/pip && rm -f uv.lock && uv lock