Directory structure

November 17, 2025 · View on GitHub

This file describes the development process for Shroud. This includes setting up a development environment, running tests and creating a release.

Directory structure

After cloning the repository, the directories will be as follows. Except for the build directory which is created by the Makefile.

build           # created by Makefile
   sphix        # Generated documentation
   $tempdir
     regression # Output of regression tests
     run        # Output from compiled tests
     venv       # Virtual environment
compiler
   # Directories of code to test compiler features
docs            # RST files for documentation.
shroud          # Python source
regression
  input
     prob.yaml
  reference
     prob
        # Files generated by Shroud
  run
     prob
        # The library wrapped for the problem.
tests
   # unittest python code

Development

All development is driven by the top level Makefile.

Create development environment

A virtual environment is created and the YAML package is installed. Defaults to Python 3.

make virtualenv
make develop

make develop allows changes to be seen without installed the module again.

The tempdir can be verified with make print-tempdir. Similar to temp.linux-x86_64-3.9.

For python2, soon to be dropped, edit the top level Makefile to comment out the line PYTHONEXE := python3. It will then use the python2 on your path.

make virtualenv2
make develop-setup

Note that due to dictionaries not being ordered in Python 2.7, the do-test tests will produce different output in the JSON files.

Documentation

Sphinx is used the create the documentation. The files are in docs directory.

make docs

Creates build/sphinx/html/index.html

make pdf

Creates build/sphinx/latex/shroud.pdf

Testing

Run unittest

make test

The unittests are in the tests directory.

Run a single unittest:

cd tests
../build/$tempdir/venv/bin/python test_declast.py CheckParse.test_inheritance

Running a single Python unit test.

# Use the Python in the virtual environment
setenv WORK  .../build/$tempdir
setenv PYTHONEXE $WORK/venv/bin/python

cd tests
$PYTHONEXE -m unittest test_ast.CheckAst.test_d_generate1

# Run a regression test
setenv PYTHONPATH $WORK/run/struct-class-c/python
cd regression/run/struct-class-c/python
$PYTHONEXE -m unittest test.Struct.test_Arrays1

Parse test

Parse some strings in tests/check_decl.py and compare to reference in tests/check-decl.output. The parse trees are compared.

make test-decl
make test-decl-diff
make test-decl-replace

Regression tests

make do-test

Run the script regression/do-test.py over some regression/input yaml files.

The output is saved in build/$tempdir/regression

A script is added to the output directory to help compare regression files.

cd WORK/build/WORK/build/tempdir/regression/classes ./cmp-shroud

Update fiducials

make do-test-replace [ do-test-args=tutorial ]

Running a single test

make do-test do-test-args=tutorial

Running a test manually after adding lines like "import pdb;pdb.set_trace()". First, run the test via do-test. Extract run line from test.log. It will contain paths and flags used to run do-test.

cd $WORK/build/$tempdir/regression/classes
$WORK/build/$tempdir/venv/bin/shroud ...  $WORK/regression/input/classes.yaml

Test generated code

Compile the files in regression/reference. The Library code which is wrapped is in the regression/run directory.

make test-clean
make test-all -k

Compile the generated code (after do-test-replace) and run some unit tests. These tests are in regression/run and contains a small library which is wrapped by the corresponding yaml file (i.e. regression/input/tutorial.yaml wraps regression/run/tutorial) The make -k flag will keep-going if one of the tests fails.

regression/run/Makfile is included by the top level makefile. regression/run/default.mk has compiler flags.

The default is to use gcc. Other compilers can be specified as:

make compiler=oneapi test-all
make compiler=intel test-all
make compiler=cray test-all

Targets to compile a single test:

fortran-XXX        compile wrappers for XXX.yaml
test-fortran-XXX   run test for XXX.yaml
test-fortran       All Fortran tests
test-cfi           Fortran tests with C-Fortran-Interface

test-c-XXX

py-XXX             compile module for XXX.yaml
test-python-XXX    run Python test for XXX.yaml
test-python        All Python tests

The Fortran tests use fruit, C tests assert, and Python tests use unittest.

test-commit

Runs test-clean, test, test-decl, and do-test. To be used before a commit.

make test-commit

Adding a regression test

  • Add a file regression/input/newtest.yaml.

  • Create directory mkdir regression/reference/newtest.

  • Run the test make do-test do-test-args=newtest

  • Look for errors in $(tempdir)/regression/log.test

  • Add a call to TestDesc in regression/do-test.py to run newtest.

  • mkdir regression/run/newtest

  • Create files Makefile, newtest.cpp and newtest.hpp

  • Add to variable fortran-test-list-std in regression/run/Makefile to run as part of test-all target. Try make -n test-fortran-newtest to see commands which will be run.

Several tests are run with language=c and c++. To accomplish this, the run/pkg files have the .c suffix. The sync-cxx target copies the files to another directory with a .cpp suffix. This directory is added to the VPATH. For the Python tests, there is an explicit run to force the use of CXX. The VPATH finds both the C and C++ files since they have the same stem.

Sample Fortran Makefile to allow TEST_FFLAGS to be in every compile for a specific test.

testdir = xxx
TEST_FFLAGS = -DTEST_C_WRAPPER

C_OBJS = \
    xxx.o \
    wrapxxx.o
F_OBJS = \
    main.o \
    fruit.o \
    wrapfxxx.o

include $(top)/regression/run/defaults.mk

## dependencies

Sample Python makefile

testdir = xxx
pymodule = xxx.so

AS_SHARED := TRUE
include $(top)/regression/run/defaults.mk

PY_OBJS = \
    xxx.o
    pyxxxmodule.o

# dependencies

Debugging

Wrappers

There are a couple of Shroud YAML options that provide useful debugging information.

options:
  debug: True
  debug_index: True

The debug option writes additional information into the generated files. This includes which statement group was used for each argument. This makes it easier to understand how the wrapper is being generated.

The debug_index requires the *debug option to also be True. It dump out more information such as the _function_index and signature fields. _function_index is an integer index used to uniquely identify each funtion to map between the generated code and the internal state dump into the JSON output file. signature is a collection of statement group indexes Shroud uses to help identify if two possible wrappers are the same.

debug is used in the testsuite to help identify when something changes. Using debug_index will produce excessive change when a function is added to a test or a statement group is added to fc-statement.json. The additions can cause indexes to change resulting in excessive changes. debug_index is designed for debugging by the developer, not the user.

memory debugging

Valgrind can be used by setting valgrind=valgrind on the Make command line. Actually, valgrind=path-to-valgrind.

make -k valgrind=valgrind test-all

Recompile code with address sanitizer.

make test-clean ; make -k use_asan=1 test-all

reference counting

Python-3.9.12/Misc/valgrind-python.supp

valgrind --tool=memcheck --suppressions=valgrind-python.supp \
                                          python -E -tt ./my_python_script.py

Build Python with --with-address-sanitizer --with-pydebug

Development

file dependency

main.py
wrapX.py  generate.py
fcfmt.py  metaattrs.py
ast.py
declast.py declstr.py fcmem.py
typemap.py statements.py todict.py
whelpers.py
util.py visitor.py
error.py  # no dependencies
metadata.py

Error with template: 'call SHROUD_copy_array_{cxx_T}({c_var_context}, {f_var}, size({f_var},kind=C_SIZE_T))'

can be debugged by changing util.wformat to remove comment to provide a backtrace.

adding a type

typemap.py

ast.py LibraryNode.create_std_names() add to namespace

declast.py get_canonical_typemap() Convert 'long int' into 'long'

formatting generated code

Always use {nullptr} instead of NULL. Helps C++.

error checking

Write a #error line instead of generating bad code so the compiler will point out the error.

debugging

import yaml
print(yaml.dump(cf_tree, default_flow_style=False))

import pprint
pp = pprint.PrettyPrinter(indent=4)
pp.pprint( dict or tuple )

import pdb; pdb.set_trace()

% yes c | python ... >&! out
Answer 'c' to all pdb prompts.

To start pdb on an exception, uncomment line in main.py

#sys.excepthook = info

pythondevmode

If this environment variable is set to a non-empty string, enable Python Development Mode, introducing additional runtime checks that are too expensive to be enabled by default. New in version 3.7.

setenv PYTHONDEVMODE 1

Release Process

Typically, Shroud releases are done when it makes sense to make new features or other changes available to users.

Shroud follows the semantic versioning scheme for assigning release numbers. Semantic versioning conveys specific meaning about the code and modifications from version to version by the way version numbers are constructed.

The master branch records the official release history of the project. Specifically, whenever, the master branch is changed, it is tagged with a new version number. We use a git ‘lightweight tag’ for this purpose. Such a tag is essentially a pointer to a specific commit on the master branch.

When all pull requests intended to be included in a release have been merged into the develop branch, we create a release candidate branch from the develop branch. The release candidate branch is used to finalize preparations for the release. At this point, the next release cycle begins and work may continue on the develop branch.

When a release candidate branch is ready, it will be merged into the master branch and a release tag will be made. Then, the master branch is merged into the develop branch so that changes made to finalize the release are included there.

Update docs/releases.rst with changes. It is best to continually update this file as user visible changes are made.

Start Release Candidate Branch

Create a release candidate branch off the develop branch to initiate a release. The name of a release branch should contain the associated release version name. Typically, we use a name like v0.14.0-rc

Finalize the Release in the Release Candidate Branch

All changes to Shroud related to finalizing the release documentation, as opposed to source code changes, are done in the release candidate branch. Typical changes include:

  • shroud/metadata.py
  • setup.py Download path
  • pyproject.toml
  • README.md
  • docs/conf.py
  • docs/releases.rst
  • Update reference files with new version. make do-test-replace do-test-args=none

Build a Python Distribution

Build and test python distribution files. Both a tar file and a wheel. Do this before finishing the release to make sure it works.

make install-twine
make sdist
make twine-check

Test it in a virtual environment where SHROUD_DEV is where the git clone is located.

cd $SHROUD_TEST
python3 -m venv venv
source venv/bin/activate or source venv/bin/activate.csh
pip install $SHROUD_DEV/dist/*.tar.gz
$SHROUD_TEST/venv/bin/shroud --version

cd $SHROUD_DEV
make do-test-install install-shroud=$SHROUD_TEST/venv/bin/shroud

Create a Pull Request for the Release

Create a pull request to merge the release candidate branch into master after all release preparation changes have been made. When the release candidate branch is complete, reviewed and approved, it will be merged into master.

Merge Release Candidate

Merge the release candidate branch into the master branch after it has been approved and all CI checks have passed. Do not "squash merge" as it will make the histories of master and develop branches disagree, and we want to preserve the history. After merging, the release candidate branch can be deleted.

Create Shiv file

shiv is a command line utility for building fully self-contained Python zipapps as outlined in PEP 441, but with all their dependencies included! shiv's primary goal is making distributing Python applications fast & easy.

https://pypi.org/project/shiv/

git checkout master
make install-shiv
rm dist-shiv/*
make shiv-file
make do-test-shiv

Draft a GitHub Release

Draft a new Release on GitHub. Describe in https://help.github.com/articles/creating-releases/

  • Enter a Release title. We typically use titles of the following form Shroud-v0.14.0

  • Select master as the target branch to tag a release.

  • Enter the release tag name, such as v0.14.0, and specify to create the tag when the release is published.

  • Summary the information in releases.rst into the release description (omit any sections that are empty).

  • Add shiv file into the binaries box.

  • Publish the release. This will create a tag at the tip of the master branch and add corresponding entry in the Releases section

Merge Master to Develop

Create a pull request to merge master into develop so that changes in the release candidate branch are integrated into subsequent Shroud development.

Upload to PyPi

The Pypi website is used by the pip utility to install Python modules. You should be able to use the dist files from Build a Python distribution since nothing has changed.

You need an account on pypi.org and test.pypi.org.

Pypi uses two factor authentication. You should also create API tokens used in a .pypirc file.

First, updated to https://test.pypi.org.

make testpypi

Verify the test install.

cd ~/tmp
python3 -m venv testpypi
cd testpypi
bin/pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ llnl-shroud
bin/shroud --version

cd $SHROUD_DEV
make do-test-install install-shroud=$SHROUD_TEST/venv/bin/shroud

If the test install works, then do the real install. Note that pypi only allows a file to be uploaded once. You can delete then reupload within 24 hours. Otherwise, a new version number will be required.

make pypi

Then verify the install.

cd ~/tmp
python3 -m venv pypi
cd pypi
bin/pip install llnl-shroud
bin/shroud --version

cd $SHROUD_DEV
make do-test-install install-shroud=$SHROUD_TEST/venv/bin/shroud

Spack

After a commit hash is created, update scripts/spack/packages/py-shroud/package.py.

Find the hash with git rev-list -n 1 v0.14.0

In the spack-packages repository update repos/spack_repo/builtin/packages/py_shroud

Annual Changes

Update copyright in LICENSE. docs/conf.py copyright date.