Development Guide

This guide explains how to build and maintain the SDRF-Proteomics documentation website.

Repository Structure

proteomics-metadata-standard/
├── scripts/                   # Build and utility scripts
│   ├── build-docs.sh          # Main build script (local & CI/CD)
│   ├── build_template_pages.py # Generate per-template HTML from YAML
│   ├── build_index_templates.py # Generate index.html template section
│   ├── resolve_templates.py   # Template inheritance resolver
│   ├── inject-headers.py      # Navigation header injection
│   ├── transform-links.py     # .adoc→.html link transformation
│   ├── transform-sdrf-tables.py # SDRF example table styling
│   ├── build_sdrf_builder_data.py # SDRF builder JSON data compiler
│   └── add-dev-banner.sh      # Dev version banner
├── sdrf-proteomics/           # Main specification source
│   ├── README.adoc            # Main specification (AsciiDoc)
│   ├── TOOLS.adoc             # Tool support page
│   ├── TEMPLATES.adoc         # Templates guide
│   ├── SAMPLE-GUIDELINES.adoc # Sample metadata guidelines
│   ├── metadata-guidelines/   # Detailed metadata guidelines
│   │   ├── data-analysis-metadata.adoc
│   │   └── sdrf-terms.tsv
│   ├── sdrf-templates/        # YAML template definitions (submodule)
│   └── images/                # Specification images
├── site/                      # Website assets and build scripts
│   ├── css/style.css          # Main stylesheet
│   ├── js/search.js           # Search functionality
│   ├── js/sdrf-builder.js     # Interactive SDRF builder logic
│   ├── index.html             # Homepage
│   ├── quickstart.html        # Quick start guide + interactive builder
│   ├── sdrf-explorer.html     # SDRF dataset explorer
│   ├── sdrf-terms.html        # Terms reference page
│   ├── build-sdrf-index.py    # Builds dataset index
│   └── build-search-index.py  # Builds search index
├── annotated-projects/        # Migration notice and legacy compatibility pointers
├── demo_page/                 # Local preview output (git-ignored)
├── .github/workflows/         # CI/CD configuration
│   └── build-docs.yml         # Website build workflow
├── DEVELOPMENT.md             # This file
└── README.md                  # Project overview

Prerequisites

Install the required tools:

# Install Asciidoctor (Ruby)
gem install asciidoctor
gem install asciidoctor-diagram

# Python 3 is required for build scripts
python3 --version

Building the Website Locally

Using the Build Script (Recommended)

The easiest way to build the documentation is using the provided build script. This script replicates the exact CI/CD build process, ensuring consistency between local and deployed builds.

# Build to default directory (demo_page/)
./scripts/build-docs.sh

# Build with a clean start (removes existing output)
./scripts/build-docs.sh --clean

# Build to a specific directory
./scripts/build-docs.sh docs

# Build dev version (adds development banner)
./scripts/build-docs.sh docs/dev --dev

# Show help
./scripts/build-docs.sh --help

What the Build Script Does

1. Converts AsciiDoc to HTML using Asciidoctor with proper styling options
2. Copies static assets (CSS, JavaScript, images)
3. Copies static HTML pages (index.html, quickstart.html, sdrf-explorer.html, sdrf-editor.html, sdrf-terms.html)
4. Builds SDRF Explorer index using annotated project metadata and links
5. Injects navigation headers into all generated HTML pages
6. Transforms SDRF links to use the SDRF Explorer viewer
7. Transforms SDRF example tables to add color-coded column styling
8. Builds search index for site-wide search functionality
9. Adds dev banner (when using --dev flag)

Manual Build (Advanced)

If you need to build individual files for quick iteration:

# Build the main specification only
asciidoctor \
  -D demo_page \
  -a stylesheet=css/style.css \
  -a linkcss \
  -a toc=left \
  -a toclevels=3 \
  -a sectanchors \
  -a sectlinks \
  -a source-highlighter=highlight.js \
  --backend=html5 \
  -o specification.html \
  sdrf-proteomics/README.adoc

Note: Manual builds will be missing navigation headers, search functionality, and other features. Use the build script for testing the complete site.

View Locally

After building, open the site directly in your browser:

open demo_page/index.html

The site is self-contained and doesn't require a web server.

Key Files to Modify

File	Purpose
sdrf-proteomics/README.adoc	Main specification document
sdrf-proteomics/metadata-guidelines/*.adoc	Detailed metadata guidelines
site/css/style.css	Website styling (including SDRF table colors)
site/index.html	Homepage
site/sdrf-explorer.html	Dataset explorer page
site/sdrf-terms.html	Terms reference page
.github/workflows/build-docs.yml	CI/CD build configuration

SDRF Example Table Styling

SDRF example tables use color-coded columns. Add these CSS classes:

Class	Color	Use For
sample-col	Blue	source name, characteristics[...]
data-col	Green	assay name, comment[...], technology type
factor-col	Orange	factor value[...]

Example HTML in AsciiDoc:

++++
<div class="sdrf-example-table">
<table>
<thead>
<tr>
<th class="sample-col">source name</th>
<th class="sample-col">characteristics [organism]</th>
<th class="data-col">assay name</th>
<th class="data-col">comment [data file]</th>
<th class="factor-col">factor value [disease]</th>
</tr>
</thead>
<tbody>
<tr>
<td class="sample-col">sample_1</td>
<td class="sample-col">homo sapiens</td>
<td class="data-col">run_1</td>
<td class="data-col">file.raw</td>
<td class="factor-col">normal</td>
</tr>
</tbody>
</table>
<div class="sdrf-legend">
<span class="legend-item"><span class="legend-color sample-bg"></span> Sample metadata</span>
<span class="legend-item"><span class="legend-color data-bg"></span> Data file metadata</span>
<span class="legend-item"><span class="legend-color factor-bg"></span> Factor values</span>
</div>
</div>
++++

CI/CD Deployment

The website is automatically built and deployed when pushing to the master branch → https://sdrf.quantms.org/.

The CI/CD workflow (.github/workflows/build-docs.yml) performs the same steps as the local build script, ensuring consistency.

Local vs CI/CD Comparison

Feature	Local (build-docs.sh)	CI/CD
AsciiDoc conversion	✓	✓
Navigation headers	✓	✓
SDRF link transformation	✓	✓
Search index	✓	✓
Dev banner	--dev flag (local only)	—
Output directory	Configurable	docs/

Ensuring Consistency

To verify your local build matches the deployed version:

# Build same as production
./scripts/build-docs.sh --clean

# Build same as dev deployment
./scripts/build-docs.sh --clean --dev

Adding New Annotated Projects

Annotated datasets are now maintained in bigbio/sdrf-annotated-datasets using the layout datasets/{ACCESSION}/{ACCESSION}.sdrf.tsv.

1. Create a folder in datasets/ with the accession in bigbio/sdrf-annotated-datasets
2. Add the SDRF file as {ACCESSION}.sdrf.tsv
3. Validate the file:

   pip install sdrf-pipelines
   parse_sdrf validate-sdrf --sdrf_file datasets/PXD000000/PXD000000.sdrf.tsv
   

4. Open a pull request in bigbio/sdrf-annotated-datasets
5. If this repository needs a refreshed explorer index after migration updates, rebuild it locally:

   python3 site/build-sdrf-index.py
   

References in AsciiDoc

Use bibliography format for proper cross-references:

# In text:
This is supported by research <<ref1>>.

# In References section:
[bibliography]
== References

- [[[ref1,1]]] Author et al. Title. Journal (Year). https://doi.org/xxx[doi:xxx]

Useful Commands

# Validate all SDRF files in the annotated datasets repository
for f in datasets/*/*.sdrf.tsv; do
  parse_sdrf validate-sdrf --sdrf_file "$f"
done

# Find all SDRF files with a specific column
grep -l "characteristics\[disease\]" datasets/*/*.sdrf.tsv

# Count annotated projects
ls -d datasets/PXD* | wc -l

Troubleshooting

Stylesheet Warnings

When building, you may see warnings like:

asciidoctor: WARNING: stylesheet does not exist or cannot be read

These are expected and can be ignored. We use -a linkcss which links the stylesheet at runtime rather than embedding it during build.

Missing Dependencies

If you see errors about missing commands:

• asciidoctor: gem install asciidoctor
• python3: Install Python 3 from https://python.org

Build Output Differences

If the local build looks different from the deployed version:

1. Ensure you're using the build script (./scripts/build-docs.sh), not manual asciidoctor commands
2. Clear the output directory with --clean
3. Check that you have the latest site assets (git pull)

Common Issues

Issue	Solution
No navigation header	Use the build script instead of manual asciidoctor
CSS not loading	Ensure site/css/style.css was copied to output
Dev banner missing	Use --dev flag when building
Search not working	Ensure search-index.json was generated

SDRF Editor Integration

The website includes an embedded SDRF Editor at /sdrf-editor.html. The editor is loaded from the sdrfedit repository via jsDelivr CDN.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│  sdrfedit repo (bigbio/sdrfedit)                                │
│                                                                  │
│  Developer workflow:                                             │
│  1. Make changes to editor source code                           │
│  2. npm run build -- --configuration=production                  │
│  3. git add dist/ && git commit && git push to main              │
│                                                                  │
│  The dist/ folder is committed:                                  │
│  dist/sdrf-editor/browser/                                       │
│  ├── main.js                                                     │
│  ├── polyfills.js                                                │
│  └── styles.css                                                  │
└─────────────────────────────────────────────────────────────────┘
                              ↓
              jsDelivr CDN (automatic, no deployment needed)
              https://cdn.jsdelivr.net/gh/bigbio/sdrfedit@main/...
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│  proteomics-metadata-standard                                    │
│                                                                  │
│  site/sdrf-editor.html loads the editor:                         │
│  <script src="https://cdn.jsdelivr.net/gh/bigbio/sdrfedit@main/ │
│               dist/sdrf-editor/browser/main.js" type="module">   │
│                                                                  │
│  When this repo is rebuilt, it pulls the latest editor version   │
└─────────────────────────────────────────────────────────────────┘

Updating the Editor

When changes are made to the sdrfedit repository:

1. sdrfedit maintainer builds and commits the dist/ folder to main
2. jsDelivr automatically serves the new files (cached ~24 hours)
3. This website gets the updated editor on next rebuild/redeploy

To force an immediate update, the specification website needs to be rebuilt:

• Push to dev branch → triggers rebuild → pulls latest editor
• Push to master branch → triggers rebuild → pulls latest editor

Editor Files

File	Purpose
site/sdrf-editor.html	Editor page (loads Angular app from jsDelivr)
scripts/inject-headers.py	Adds Editor link to navigation

Testing Editor Changes Locally

To test editor changes before publishing:

# In sdrfedit repo
cd ../sdrfedit
npm run build -- --configuration=production

# Serve locally and update sdrf-editor.html to point to local files
# (temporarily change jsDelivr URLs to local paths for testing)

Interactive SDRF Builder

The quickstart page (/quickstart.html) includes an interactive SDRF template builder that helps users create custom SDRF templates by answering questions.

How It Works

1. Build time: scripts/build_sdrf_builder_data.py resolves all YAML templates and sdrf-terms.tsv into sdrf-builder-data.json
2. Runtime: site/js/sdrf-builder.js loads the JSON and renders a branching questionnaire
3. Users select technology, organism, experiment type, review columns, and download a TSV

Builder Files

File	Purpose
scripts/build_sdrf_builder_data.py	Compiles YAML templates + terms into JSON
site/js/sdrf-builder.js	Builder UI logic (vanilla JS)
site/quickstart.html	Page with simplified intro + builder scaffold
site/css/style.css	Builder-specific styles (.builder-*, .option-*)

Updating the Builder

When templates change (new columns, new templates):

1. The build script automatically recompiles sdrf-builder-data.json
2. No JS changes needed unless new template layers or combination rules are added
3. New templates appear automatically in the appropriate step

Contributing

1. Make changes to the appropriate AsciiDoc or HTML files
2. Build locally and test: ./scripts/build-docs.sh --clean
3. View locally: open demo_page/index.html
4. Open a pull request against master
5. After approval and merge, changes deploy to https://sdrf.quantms.org/