APM Server Release Checklist

The APM Server follows the Elastic Stack release schedule and versions. A release starts with a Feature Freeze period, during which only bug fixes are allowed to be merged into the specific release branch. We generally follow semver for release versions. For major and minor releases, a new branch is cut from the main branch. For patch releases, only the version on the existing major and minor version branch gets updated. Release workflows can be run manually, and version bump automation is often driven by the centralized release pipeline (typically minor on Feature Freeze day and patch on release day). The Release Manager will ping the team to align the release process.

This documentation is for 9.x releases. If you are releasing a 8.x look here

Patch Release

1. Create a Test Plan.
2. Ensure all relevant backport PRs are merged. We use backport labels on PRs and automation to ensure labels are set.
3. Release notes for patch releases must be manually added at least one day before release.
4. Create a PR targeting the main branch. To add release notes:
   • Add a new section to the existing release notes file (Sample PR) containing the Features and enhancements as well as the Fixes subsections.
   • Add your PR to the documentation release issue in the elastic/dev repo (Sample Issue).
   • The PR should be merged the day before release.
5. On release day, ensure patch version bump automation has run for the release version.
   • For patch releases, this is usually triggered by unified-release-centralized-version-bump, which calls apm-server-version-bump, and then triggers run-patch-release with the new version.
   • This automated patch bump is usually executed on release day.
   • If needed, you can still run run-patch-release manually from main.
   • In "The version", specify the new patch release version, e.g.: if versions should be bumped to 9.3.2, use 9.3.2.
   • This workflow will:
     • Create the update-<VERSION> branch for the provided version.
     • Update version constants across the codebase and create a PR targeting the release branch.

Minor Release

1. Create a Test Plan.
2. Run the run-minor-release workflow (In "Use workflow from", select main branch. Then in "The version", specify the minor release version the release is for). This workflow will:
   • Create a new release branch using the stack version (X.Y).
   • Update the changelog for the release branch and open a PR targeting the release branch titled <major>.<minor>: update docs.
   • Create a PR on main titled <major>.<minor>: update docs, mergify, versions and changelogs.
   • Automated minor version bumps are typically done on Feature Freeze day.

Major Release

1. Create a Test Plan.
2. Run the run-major-release workflow (In "Use workflow from", select main branch. Then in "The version", specify the major release version the release is for). This workflow will:
   • Create a new release branch using the stack version (X.Y).
   • Update the changelog for the release branch and open a PR targeting the release branch titled <major>.<minor>: update docs.
   • Create a PR on main titled <major>.0: update docs, mergify, versions and changelogs.

Before merging them compare commits between latest minor and the new major versions and ensure all relevant PRs have been included in the Changelog. If not, amend it in both PRs. Request and wait a PR review from the team before merging. After it's merged add your PR to the documentation release issue in the elastic/dev repo (Sample Issue).

Update Dependencies

• libbeat:

  • Updates are automatically created for the release branch, multiple times per week.
  • If there is a need for a manual update, you can run make update-beats on the release branch.
  • This might be the case when waiting for an urgent bug fix from beats, or after branching out the release branch.
  • For patch releases, the updates are supposed to only contain bug fixes. Take a quick look at the libbeat changes and raise it with the APM Server team if any larger features or changes are introduced.

• go-elasticsearch:

  • If no branch or tag is available, ping the go-elasticsearch team, go get github.com/elastic/go-elasticsearch/v$major@$major.$minor.

Create a Test Plan

Create a GitHub Issue to track testing of the release branch.

Choose one way to create the initial issue:

• Option 1 (recommended): Run the create-test-plan workflow with the upcoming patch or minor version, then review and adjust the generated issue content.
• Option 2: Create it manually using the test plan issue template.

The issue should include:

• Test all functional changes applied to the new version.
• Any non-functional change or any change already covered by automated tests must not be included.
• Review any PRs updating dependencies, as some functional changes happens through these dependencies.
• Link to PRs in the APM Server repository that need to be tested manually.
  • Apply both the test-plan label and the appropriate version label to the issue - create the version label if it does not already exist.
  • For reference, see the 9.1 Test Plan.
  • For additional examples, you can also view all previous test plans.
• Add other test cases that require manual testing, such as test scenarios on ESS, that are not covered by automated tests or OS compatibility smoke tests for supporting new operating systems.

Between feature freeze and release

• Test the release branch by completing items in the Test Plan:
  • Always use a build candidate (BC) when testing, to ensure we test with the distributed artifacts. The first BC is usually available the day after Feature Freeze.
  • Identify which changes require testing via the created test labels, e.g. for 8.3.0.
  • Grab a PR that hasn't been verified and assign yourself to prevent other devs from re-testing the same change.
  • Test the PR following the Author's how to test this section.
  • Post your testing scenarios on the PR as a comment (for tracking down details in case we run into regressions).
  • Add the test-plan-ok or the test-plan-regression label to the PR. In case of regression, either open a PR with a fix or open an issue with the details.
• Collaborate with the docs team on any release highlights or breaking changes that should be included in the APM Server guide.
• Check that the DRA is built automatically in https://buildkite.com/elastic/apm-server-package.

On release day

• For minor releases: new branches need to be added to conf.yml in the elastic/docs repo. Example. This is handled by the docs release manager.

• For patch releases: if there is an open PR that bumps the version, merge the PR (it may have been created by the GitHub workflow as part of the steps in the "Day after feature freeze" section). If there is no PR, create one.

• A new tag will automatically be created on GitHub.

When compatibility between Agents & Server changes

• Update the agent/server compatibility matrix in the elastic/observability repo.