DOR Services App

July 15, 2026 · View on GitHub

DOR Services App

This Ruby application provides a REST and GraphQL API for DOR Services. There is a OAS 3.0 spec that documents the API in openapi.yml. You can browse the generated documentation at http://sul-dlss.github.io/dor-services-app/

Authentication

To generate an authentication token run RAILS_ENV=production bin/rails generate_token on the prod server. This will use the HMAC secret to sign the token. It will ask you to submit a value for "Account". This should be the name of the calling service, or a username if this is to be used by a specific individual. This value is used for traceability of errors and can be seen in the "Context" section of a Honeybadger error. For example:

{"invoked_by" => "workflow-service"}

GraphQL

DSA exposes a limited GraphQL API at the /graphql endpoint. The API is implemented using graphql-ruby. The purpose of the API is to allow retrieving only the parts of cocina objects that are needed, in particular, to avoid retrieving very large structural metadata.

It is limited in that:

It only supports querying, not mutations.
Only the first level of attributes (description, structural, etc.) are expressed in the GraphQL schema; the contents of each of these attributes are just typed as JSON.

Developer notes:

Most GraphQL code is in app/graphql.
In local development, the GraphiQL browser is available at http://localhost:3000/graphiql.

Database connections

Both the DSA and workflow databases are configured with a maximum number of allowed connections (SHOW max_connections;). The following describes the use of connections by parts of production DSA:

2 per passenger process (1 needed + 1 headroom) x 45 processes x 2 servers = 180
- This is configured by the MAX_DB_CONNECTIONS env variable in puppet.
7 per sidekiq process (concurrency of 6 + 1 headroom) x 8 processes x 5 servers = 280
- This is configured by the MAX_DB_CONNECTIONS env variable in puppet.
2 per export-cocina-head-versions process x 6 processes = 12
- This is configured by setting the MAX_DB_CONNECTIONS env variable in schedule.rb
2 per validate-cocina process x 12 processes = 24
5 per sneakers process x ((1 sneakers worker class x 1 processes per worker class) + 1 supervisor) x 2 servers = 20
CreateEventJob is the worker class. It is configured to have 1 process (instead of default of 4).
headroom = 25 Total: 541

Developer Notes

DOR Services App is a Rails app.

Background Jobs

Dor Services App uses Sidekiq to process background jobs. To spin up Sidekiq, run:

bundle exec sidekiq # use -d option to daemonize/run in the background

See the output of bundle exec sidekiq --help for more information.

Note that the application has a web UI for monitoring Sidekiq activity at /queues.

Running Tests

First, ensure the database container is spun up:

docker compose up # use -d to daemonize/run in background

And if you haven't yet prepared the test database, run:

RAILS_ENV=test bundle exec rails db:test:prepare

To run the tests:

bundle exec rspec

To run rubocop:

bundle exec rubocop

Setup RabbitMQ

You must set up the durable rabbitmq queues that bind to the exchange where workflow messages are published.

RAILS_ENV=production bin/rake rabbitmq:setup

This is going to create queues for this application that bind to some topics.

RabbitMQ queue workers

In a development environment you can start sneakers this way:

WORKERS=CreateEventJob bin/rake sneakers:run

but on the production machines we use systemd to do the same:

sudo /usr/bin/systemctl start sneakers
sudo /usr/bin/systemctl stop sneakers
sudo /usr/bin/systemctl status sneakers

This is started automatically during a deploy via capistrano. Note that sneakers only runs on some worker servers.

Cron check-ins

Some cron jobs (configured via the whenever gem) are integrated with Honeybadger check-ins. These cron jobs will check-in with HB (via a curl request to an HB endpoint) whenever run. If a cron job does not check-in as expected, HB will alert.

Cron check-ins are configured in the following locations:

config/schedule.rb: This specifies which cron jobs check-in and what setting keys to use for the checkin key. See this file for more details.
config/settings.yml: Stubs out a check-in key for each cron job. Since we may not want to have a check-in for all environments, this stub key will be used and produce a null check-in.
config/settings/production.yml in shared_configs: This contains the actual check-in keys.
HB notification page: Check-ins are configured per project in HB. To configure a check-in, the cron schedule will be needed, which can be found with bundle exec whenever. After a check-in is created, the check-in key will be available. (If the URL is https://api.honeybadger.io/v1/check_in/rkIdpB then the check-in key will be rkIdpB).

Reindexing all objects

A full reindex can be invoked with bin/rake indexer:reindex_jobs.

This will enqueue a number of BatchReindexJobs.

A full reindex is scheduled to happen periodically to make sure that all objects have the latest indexing.

Robots

DSA hosts robots that perform DSA actions. This replaces the previous pattern in which a common accessioning robot would invoke a DSA endpoint that would start a DSA job that would perform the action and then update the workflow status.

Robots are in jobs/robots/*. All DSA robots must be added to Workflow Server Rails' QueueService so that the workflow jobs are handled by DSA robots (instead of normal robots).

There also must be a sidekiq process to handle the DSA robot queues. For example:

:labels:
 - robot
:concurrency: 5
:queues:
  - [accessionWF_default_dsa, 2]
  - accessionWF_low_dsa

Workflows

The previous independent Rails-based workflow service has been merged into Dor Services App, and access to it's methods are provided by dor-services-client. The workflow database and robot queues continue to be separate from DSA (meaning that there are separate Postgres and Redis servers for workflow; DSA uses Rail's multi-database support and separate Sidekiq configuration).

The workflows are defined by xml templates which are stored in config/workflows. The templates define a dependency graph. When all prerequisites for a step are complete, the step is marked as "queued" and a corresponding job is pushed into Sidekiq. Some steps are are marked skip-queue="true" which means they are merely logged events and do not kick off a Sidekiq process.

Sidekiq Jobs

When a workflow step is set to done, the service calculates which workflow steps are ready to be worked on and enqueues Sidekiq jobs for them. The queues are named for the workflow and priority. For example:

accessionWF_high
accessionWF_default
accessionWF_low
assemblyWF_high
assemblyWF_default
assemblyWF_low
disseminationWF_high
disseminationWF_default
disseminationWF_low
...

Workflow Variables

If a workflow or workflows for a particular object require data to be persisted and available between steps, workflow variables can be set. These are per object/version pair and thus available to any step in any workflow for a given version of an object once set.

These data are not persisted in Cocina, and are not preserved or available outside of the workflow API, so they should only be used to persist information used during workflow processing.

To use, pass in a "context" parameter as JSON in the body of the request when creating a workflow. The json can contain any number of key/value pairs of context.

This context will then be returned as JSON in each process block of the XML response containing workflow data for use in processing.

This can be used if a user selects an option in Pre-assembly or Argo that needs to be passed through the accessioning pipeline, such as if OCR or captioning is required. The value is set when creating the workflow, and then available to each robot which needs it.

Command-Line Tools

Run reports

Custom reports are stored in dor-services-app in the app/reports directory. Examine some of the reports in that folder to see the pattern of how they are structured and run. Since they need to access to the SDR datastore to run, you can run them on the sdr-infra box, which has read-only access to each environment (qa/stage/prod).

Connect to the sdr-infra box:
```
ssh sdr-infra
```

Start a shell as the deploy user:

# you may or may not need to supply the `-n SUNETID` argument
ksu deploy

Go to the home directory of the deploy user. If you have previously run reports, you may have a directory for your username in here already and can skip this step. If not, you will create it:

cd ~
ls -al # do you see a folder for your sunet?  if so, go to the next step, if not create it below and then clone DSA
mkdir [SUNETID] # make a directory for yourself so you can run reports in isolation from others being run
cd [SUNETID]
git clone https://github.com/sul-dlss/dor-services-app.git # check out the DSA code

Change to your directory (if not already there from the previous step) and checkout the correct branch of DSA:

cd [SUNETID]/dor-services-app
git checkout [BRANCH] # if your report is not merged into main, you can checkout your branch here

Connect to the desired database by setting the environment variables as shown below. This must be done each time you SSH back into the box to run a new report. To run a report against production SDR, use the values below. For stage or QA, replace PROD with STAGE or QA in the HOSTNAME and PASSWORD variables:

STAGE

export DATABASE_NAME="dor_services"
export DATABASE_USERNAME=$DOR_SERVICES_DB_USER
export DATABASE_HOSTNAME=$DOR_SERVICES_DB_STAGE_HOST
export DATABASE_PASSWORD=$DOR_SERVICES_DB_STAGE_PWD
export MAX_DB_CONNECTIONS=2

PROD

 export DATABASE_NAME="dor_services"
 export DATABASE_USERNAME=$DOR_SERVICES_DB_USER
 export DATABASE_HOSTNAME=$DOR_SERVICES_DB_PROD_HOST
 export DATABASE_PASSWORD=$DOR_SERVICES_DB_PROD_PWD
 export MAX_DB_CONNECTIONS=2

Run the report (good idea to do it in a screen or via background process in case you get disconnected):
```
bin/rails r -e production "BadIso8601Dates.report" > BadIso8601Dates.csv
```

When done, you can pull the report to your laptop as needed:

scp sdr-infra:/opt/app/deploy/[SUNETID]/dor-services-app/BadIso8601Dates.csv .

Export Cocina JSON data

You may want to export a full dataset of Cocina JSON perform actions like the following on your local machine:

Run validation. See bin/validata-data (in cocina-models).
Create reports using jq queries. See the /cocina-jq-query skill (also in cocina-models).

To export data, in a screen session on sdr-deploy:

ksu deploy
Export the database variables as above.
cd ~/dor-services-app
Export in parallel: RAILS_ENV=production bin/export-cocina-head-versions -p12 -r --output-dir /sdrmd/cocina

Once this is done, you may download the file to your local computer from /sdrmd/cocina.

Scheduled Cocina JSON data exports

Periodic data exports are available from /dor/cocina_dump in all environments. (See schedule.rb.)

Generate a list of druids from Solr query

$ bin/generate-druid-list 'governed_by_ssim:"druid:rp029yq2361"'

The results are written to druids.txt.

Remove deleted items from a list of druids

$ bin/clean-druid-list -h
Usage: bin/clean-druid-list [options]
    -i, --input FILENAME             File containing list of druids (instead of druids.txt).
    -o, --output FILENAME            File to write list of druids (instead of druids.clean.txt).
    -h, --help                       Displays help.

Solr is used to determine if an item still exists.

Reset accessioning for one or more druids

NOTE: This should only be used in exceptional cases, and with the approval of repository managers.

Usage: bin/reset-accessioning [options]
    -i, --input DRUID_OR_CSV_FILE    Either a single druid or a path to a CSV file with multiple druids
        --[no-]dry_run               Simulate the run (defaults to FALSE)
        --[no-]revert_description    Revert desc metadata from prior closed version (defaults to FALSE)
    -h, --help                       Displays help.

Find druids missing from the Solr index

Run the missing druid rake task:

RAILS_ENV=production bin/rake missing_druids:unindexed_objects

This produces a missing_druids.txt file in the application root.

Missing druids can be indexed with:

RAILS_ENV=production bin/rake missing_druids:index_unindexed_objects

Decommission one or more items

Either provide a single druid, or a CSV with druids:

RAILS_ENV=production bin/rake decommission:item[druid,sunetid,description]
RAILS_ENV=production bin/rake decommission:items[file,sunetid,description]

Release all embargoed objects whose embargoes have passed

RAILS_ENV=production bin/rake dsa:embargo_release

Manage object versions

# Open a new version from an existing version
RAILS_ENV=production bin/rake dsa:open_version[druid,description,from_version]
# Closes a repository object without changing a user version
RAILS_ENV=production bin/rake dsa:close_version[druid]
# Move a user version
RAILS_ENV=production bin/rake dsa:move_user_version[druid,user_version,to_version]

Register items from CSV

RAILS_ENV=production bin/rake seed:register[input_file]

Update a workflow step

RAILS_ENV=production bin/rake workflow:step[druid,workflow,process,status]

Check that objects can be updated

The Migrators::CheckCocinaUpdate migrator can be used to check that all objects that are not open are in an openable state.

RAILS_ENV=production bin/migrate-cocina Migrators::CheckCocinaUpdate

Data migrations / bulk remediations

bin/migrate-cocina provides a framework for data migrations and bulk remediations. It supports optional versioning and publishing of objects after migration. It also can persist by directly committing RepositoryObjectVersions (for any version for a RepositoryObject) using the UpdateObjectService (only for the head version).

Usage: bin/migrate-cocina MIGRATION_CLASS [options]
        --mode [MODE]                Migration mode (dryrun, migrate). Default is dryrun
    -s, --sample SAMPLE              Sample size per type, otherwise all objects.
    -b, --batch BATCH                Batch size for processing. Default is 25000 or 5000 if reading druids from a file.
    -f, --file FILE                  File from which to read specific druids.
    -h, --help                       Displays help.

The process for performing a migration/remediation is:

Implement a Migrator (app/services/migrators/). For the requirements of a Migrator class, see Migrators::Base and its subclasses (Migrators::Exemplar is a simple examples for illustration; the rest were used for production migrations). Migrators should be unit tested.
Perform a dry run: bin/migrate-cocina Migrators::Exemplar --mode dryrun and inspect migrate-cocina.csv for any errors. This is a way to change the cocina and validate the new objects without saving the updated cocina (or publishing or versioning).
Perform migration/remediation: bin/migrate-cocina Migrators::Exemplar --mode migrate and inspect migrate-cocina.csv for any errors.
Optionally, verify by running a dry run again.

Additional notes:

Migrations can be performed against all items or just a list to bin/migrate-cocina.
The migrator specifies the migration strategy (cocina update, commit, commit with versioning, commit with publish), which determines how the object is persisted, whether it is versioned, and whether it is published.
For versioned migrations, a migrator may define self.workflow_context to pass optional workflow context when MigrationRunner closes the opened version and starts accessionWF.
Workflow context defaults to {} if not overridden.
See Migrators::ExemplarWithContext for a minimal example of context usage.
A migrator may define self.migration_tag returning an administrative tag string (e.g. 'Migration : MyMigratorName') to enable tag-based gating. If the tag is already present on an object, migration is skipped (status: SKIPPED). If absent, the object is migrated and the tag is added afterwards (non-dry-run only). This provides idempotent re-runs. Returns nil by default (no gating).
Breaking changes, especially breaking cocina model changes, are going to require additional steps, e.g., stopping SDR processing. The complete process is to be determined.

Reset Process (for QA/Stage)

Steps

Reset the database

Deployment

NOTE: The application is deployed continuously by our on-prem Jenkins service (sul-ci-prod) to the staging environment on every merge to main. See Jenkinsfile for how that is wired up.