KEH Policy Methods Library

The Policy Methods Library is a collection of functions to encapsulate the business logic for checking adherence to the GitHub Usage Policy within ONS.

Table of Contents

• KEH Policy Methods Library
  • Table of Contents
  • Prerequisites
  • Makefile
  • Using the Package
    • GitHub App Setup
    • End User Instructions
    • Developer Instructions
  • Package Structure
  • Deployment
  • Documentation
    • GitHub Actions for Documentation
    • Local Development of Documentation
  • Linting and Testing
    • GitHub Actions
    • Running Tests and Linters Locally
      • Primary Language
      • MegaLinter
      • Documentation linting and building

Prerequisites

• Python 3.12 or higher
• Poetry for dependency management
• Node.js and npm for documentation linting (Markdownlint)

Makefile

This project uses a Makefile to simplify common tasks. To see the available commands, run:

make help

Using the Package

GitHub App Setup

Due to the nature of the checks in this package, it is necessary to set up a GitHub App in order to use the package. The App is used to authenticate with the GitHub API and perform the checks on behalf of the user.

Alternatively, you can use a personal access token (PAT) to authenticate with the GitHub API, but this is not recommended as it is less secure and does not provide the same level of granularity in terms of permissions.

Details on this are available in the GitHub Clients documentation in the docs directory.

The GitHub App needs to be created and installed in the relevant GitHub organisation. GitHub have instructions on how to do this:

Creating a GitHub App :link:

This library will require the following GitHub App secrets:

• GITHUB_APP_ID: The ID of the GitHub App.
• GITHUB_APP_PRIVATE_KEY: The private key of the GitHub App, in PEM format.
• GITHUB_ORGANISATION: The GitHub organisation that the App is installed in.

More information on this process is available in the GitHub Auth documentation in the docs directory.

A list of permissions required for the GitHub App can be found in the GitHub App Permissions documentation in the docs directory. Additionally, each check in the policy_methods_library has a list of the permissions required for that check, which can be found in the documentation for that check.

End User Instructions

To use the package, you can install it using your preferred method (e.g. pip, Poetry, etc.) and then import the relevant modules and functions in your code.

# Using pip
pip install git+https://github.com/ONS-Innovation/keh-policy-methods-library@<version>

# or using Poetry
poetry add git+https://github.com/ONS-Innovation/keh-policy-methods-library@<version>

Then, in your Python code, you can import the relevant modules and functions:

from policy_methods_library.github.clients import GitHubRestClient

client = GitHubRestClient(
    owner=github_organisation,
    app_id=app_id,
    private_key=private_key,
)

Detailed examples of the functionality provided by the package can be found in the documentation, which is available in the docs directory and deployed to GitHub Pages.

Developer Instructions

When developing the package, you can use the tests/manual_testing.py file for manual testing of the package during development. This file has a GitHub Client instance already setup which can be used to test the checks in the policy_methods_library.

To use the latest version of the package in the manual_testing.py file, you can install the package into your virtual environment:

pip install .

Note: This should be done from the root directory of the repository, and you should have your virtual environment activated.

Whenever you make changes to the package, you will need to reinstall it for the changes to be reflected in the manual_testing.py file.

Further instructions for developers, including how to add new checks, can be found in the documentation under the "Dev Guides" section.

Package Structure

src/
└── policy_methods_library/
   ├── checks/                # Core business logic for checking policy adherence.
   │   ├── __init__.py
   │   └── ...                # files for each specific check
   │
   ├── github/                # GitHub API integrations.
   │   ├── __init__.py
   │   ├── auth.py            # Functions to handle authentication with the GitHub API.
   │   └── clients.py         # Client for making REST API calls to GitHub.
   │
   ├── utils/                 # Internal helper utilities used by checks.
   │   ├── __init__.py
   │   └── ...                # files for each specific utility
   │
   └── __init__.py            # Init file for the package.

Further documentation can be found within the docs directory.

Deployment

The package can be deployed to GitHub Releases using the GitHub Actions workflow defined in .github/workflows/publish-release.yml.

This workflow is triggered on pushes to tags matching the pattern "v*". The tag must follow the format "v0.0.0" (e.g. "v0.1.0", "v1.0.0", etc.) for the workflow to run successfully.

A production release will be created when a tag is pushed that follows the format "v0.0.0" (e.g. "v0.1.0", "v1.0.0", etc.). A pre-release will be created when a tag is pushed that follows the format "v0.0.0-rc" (e.g. "v0.1.0-rc", "v1.0.0-rc", etc.). This allows for both stable releases and pre-releases to be created and published to GitHub Releases.

Package versioning is derived dynamically from the git tag during the release workflow. To create a new release, create and push a version tag:

git tag v0.1.0
git push origin v0.1.0

This triggers the GitHub Actions workflow to build and publish the package to GitHub Releases using version 0.1.0 derived from tag v0.1.0.

Note: It is important that GitHub Releases are only created via the GitHub Actions workflow, and not manually via the GitHub UI. This is because the workflow ensures that the package is built and published correctly.

Documentation

This repository uses MkDocs for documentation. The documentation source files are located in the docs directory.

GitHub Actions for Documentation

MkDocs gets deployed to GitHub Pages using GitHub Actions. The workflow for this is located at .github/workflows/deploy-docs.yml. Before deployment, another GitHub Action workflow runs to check that the documentation builds correctly and has no linting or formatting issues. This workflow is located at .github/workflows/ci-docs.yml.

Local Development of Documentation

To run the documentation locally:

1. Create a Python virtual environment and activate it.

   python -m venv venv
   source venv/bin/activate
   

2. Install the dependencies for MkDocs.

   make docs-install
   

3. Run the MkDocs development server.

   make docs-serve
   

Linting and Testing

GitHub Actions

This repository has GitHub Actions workflows set up for linting and testing. The workflows are located at:

• .github/workflows/ci-fmt.yml for linting and formatting checks (primary language).
• .github/workflows/ci-test.yml for running automated tests.
• .github/workflows/ci-docs.yml for checking that the documentation builds correctly and has no linting or formatting issues.
• .github/workflows/megalinter.yml for running MegaLinter, which checks for linting and formatting issues across multiple languages and file types (this is a catch-all linter).
• .github/workflows/deploy-docs.yml for deploying documentation to GitHub Pages.

Running Tests and Linters Locally

Primary Language

To run the linters and formatters for the primary language (Python) locally, you can use the following command:

make lint

To apply automatic fixes for any linting or formatting issues found, you can use:

make fmt

To run the tests locally, you can use:

make test

MegaLinter

This repository uses MegaLinter for comprehensive linting across multiple languages and file types. We use this so that all additional assets in the repository (e.g. YAML files, Markdown files, etc.) are also linted and checked for formatting issues, without having to set up specific linters for each file type.

To run MegaLinter locally, you can use the following command:

make megalinter

Documentation linting and building

This repository uses Markdownlint for linting the documentation. To run Markdownlint locally, you can use the following:

make docs-lint

Note: This will install markdownlint-cli globally via npm if it is not already installed.

To apply automatic fixes for any linting issues found by Markdownlint, you can use:

make docs-fix

To test that the documentation builds correctly, you can use the following command:

make docs-build

Note: This depends on MkDocs being set up for the repository. Instructions for setting up MkDocs can be found in the Documentation section of this README.