Immutable Build Strategy

July 16, 2026 · View on GitHub

Scope: The single, authoritative description of the build-once / promote-the-artifact strategy used across the ATAP ecosystem (C# packages, PowerShell modules, and the customer-facing Release Bundle that ships to Chocolatey and WinGet). Audience: Anyone reading the per-area Build/Pack/Push docs; anyone designing a new pipeline; release engineers reasoning about promotion gates. Status: Authoritative for sprint-0007. Supersedes the older build-per-tier ("buildpertier") pattern referenced in earlier sprint-0006 docs.

Companion docs:

1. The principle

Build a release unit exactly once. Promote that exact artifact, with the same SHA-256, through the five tiers. Never rebuild between tiers.

A "release unit" is one of three things:

  1. A C# NuGet package (or family of packages built from the same solution).
  2. A PowerShell module .nupkg.
  3. A Release Bundle — the final installer that contains app code, Flyway migrations, CSV seed files, seed loaders, install/upgrade scripts, and a release manifest. This is the unit that ships to Chocolatey and WinGet.

Each release unit is built once on the lowest tier (Experimental) and then promoted through the ProGet feed chain as quality gates pass. The SHA-256 recorded for the package in the Production feed is the SHA-256 recorded when the package was originally built.

2. Why "build once"

The older build-per-tier pattern rebuilt the package at every tier transition, re-running dotnet pack against the source. That model has three structural problems:

  1. Provenance gap. A package in the QA feed was not the package tested in Integration — they were two distinct compilations from possibly-different commit states. Test evidence did not carry forward.
  2. Reproducibility cost. Each tier had to re-resolve dependencies and re-execute the entire toolchain, multiplying CI time and exposing the build to flakes (NuGet outages, transient build-tool quirks, clock skew).
  3. Hash divergence. Two dotnet pack runs of the same source produce .nupkg files with different inner timestamps and therefore different SHA-256 hashes. Downstream signing, archival, and audit chains break.

Immutable build solves all three: the artifact's hash is fixed at the moment of creation; tier promotion is a metadata change in ProGet; tests at higher tiers run against the existing artifact and their results are attached to that artifact's BuildMaster release record.

3. The pipeline shape (verbal flowchart)

For any release unit, the BuildMaster pipeline does this:

  1. Resolve build context. Branch type (stable, feature/*, sprint/*, release/*), application, package family (C# / PowerShell / ReleaseBundle), whether DB assets are included for this release candidate.
  2. Load build metadata. Semantic version (NBGV), prerelease label, target package IDs, included Flyway migration set, seed data set, compatibility metadata.
  3. Restore dependencies from the appropriate ProGet development feeds (NuGet and PowerShellGet), pinned to specific versions for reproducibility.
  4. Build code.
    • C#: restore, compile, run unit tests, collect symbols and TRX results.
    • PowerShell: run Pester, lint with PSScriptAnalyzer, validate manifest, consolidate sources into the generated .psm1.
  5. Assemble the release artifact. Binaries / module payload + Flyway scripts + seed CSV + loaders + install/update scripts + release manifest (with checksums).
  6. Publish to the Experimental ProGet feed for that artifact family. The artifact never overwrites an existing version (push uses --skip-duplicate so partial runs are idempotent).
  7. Create / update a BuildMaster release record for that exact version. Attach all build artifacts, test results, and the manifest as evidence.
  8. Tier gates run as test stages, not rebuilds:
    • Experimental: smoke tests, packaging validation, installer validation.
    • Development → Integration → QA → Production: progressively broader automated and manual checks. For Release Bundles, includes Flyway migration rehearsal against a snapshot of the previous tier's DB.
  9. On gate pass, promote the same immutable package to the next ProGet feed via ProGet's package-promotion API. BuildMaster never re-runs dotnet pack.
  10. At Release-Branch builds, verify that the full test suite has been run against the tagged release artifact set, or that prior test evidence matches the release tag and manifest checksums, before allowing production publication.
  11. Final step (Release Bundle only): publish the production-tier bundle to Chocolatey and to the hosted installer location referenced by WinGet, only after the Production gate passes.
  12. Record promotion, deployment, and provenance back into BuildMaster release metadata for auditability.

3.1 version.json is the promotion ceiling

The pipeline carries two distinct tier-valued facts. Confusing them is the single most common authoring mistake in OtterScript preambles and promotion helpers, so the vocabulary is fixed here:

ConceptSourceUsed for
CurrentTierBuildMaster stage context ($Tier, -Stage, or the stage environment variable)Which stage is executing, which gate to run, and which feed transition is being attempted
CeilingTierNBGV prerelease label in version.jsonThe highest tier this immutable artifact may reach in the current run

The prerelease label is therefore a promotion ceiling, not the current stage. A Beta package is built once in Experimental, may promote through Development and Integration, and then stops before QA. The full Stage × Ceiling decision matrix, edge cases, and vocabulary map live in the focused companion doc: VersionJsonAsCeiling.md. That file is the single source of truth for the ceiling rule set; the table below is reproduced here only for quick reference while reading this strategy doc.

version.json prerelease labelCeilingTier
Sprint.N or feature labels such as PaymentRefactor.NExperimental
AlphaDevelopment
BetaIntegration
QAQA
noneProduction

The ceiling is enforced fail-closed at the cmdlet boundary. As of 2026-05-18 (BLOCKER-2 resolution), Promote-ProGetPackage requires an explicit -CeilingTier parameter on every call; the only way to bypass it is the audited emergency switch -NoCeilingCheck. Direct-publish cmdlets (Publish-NuGetPackageToProGet, Publish-UniversalPackageToProGet) reject publishes above Experimental unless -CeilingTier permits the destination feed, with -Force as the disaster-recovery bypass. Publish-PSModuleToProGet remains Experimental-only by construction.

4. ProGet feed-per-tier per package family

Each release unit family has its own five ProGet feeds. Promotion is the authoritative tier-state mechanism — branch names do not imply feed names.

Feed familyTierPurpose
nuget-experimentalExperimentalFirst push from any pipeline run.
nuget-developmentDevelopmentPromoted after unit-test gate.
nuget-integrationIntegrationPromoted after integration-test gate (hermetic feed).
nuget-qaQAPromoted after full regression (hermetic feed).
nuget-stableProductionPromoted after manual release-engineer approval.
PowershellGet-experimentalExperimentalMirrors NuGet feed semantics for PowerShell modules.
PowershellGet-developmentDevelopment
PowershellGet-integrationIntegration
PowershellGet-qaQA
PowershellGet-stableProduction
releasebundle-experimentalExperimentalFinal installer bundles (app + DB + installer scripts).
releasebundle-developmentDevelopment
releasebundle-integrationIntegration
releasebundle-qaQA
releasebundle-productionProductionSource feed for Chocolatey / WinGet publication.

Release Bundles are stored as ProGet Universal Packages rather than NuGet packages, because they carry mixed content (DLLs + SQL + CSV + PowerShell) that does not fit the NuGet conventions cleanly. See Release-Bundle-Pipeline.md §3.

5. What promotion is (and is not)

Promotion is a metadata operation in ProGet:

POST /api/promotions/promote
{
  "packageName":  "ATAP.Utilities.Philote",
  "version":     "0.1.0-Alpha.7",
  "fromFeed":    "nuget-development",
  "toFeed":      "nuget-integration",
  "reason":      "BuildMaster pipeline gate INT-PASS for build #4271"
}

ProGet copies the package bytes (or moves a feed-membership pointer, depending on storage configuration) from the source feed to the target. The package's SHA-256 is unchanged. Configure ProGet for vault storage when the stronger "one stored byte-stream across feeds" guarantee is required.

Promotion is not any of these:

  • A new dotnet pack invocation.
  • A new Publish-PSResource invocation.
  • An MSBuild rerun.
  • An NBGV {height} recomputation.
  • A change to the .nuspec, .psd1, or release manifest.

If a higher-tier feed needs a "different" build, that is a new release unit with a new version number — built from a new (or amended) release branch tag, taking the same shape as any other build.

5.1 Idempotence semantics

Promotion is idempotent only for the same package identity and the same content hash. If the target feed already contains (PackageId, Version) with the same SHA-256 as the source feed package, Promote-ProGetPackage returns success and reports ResponseSummary as already promoted (idempotent no-op).

If the target feed already contains (PackageId, Version) with a different SHA-256, the promotion must fail with a clear error. That state means the immutable pipeline has been bypassed or corrupted, not that the promotion was already completed.

5.2 Ceiling enforcement at the promotion call site

Before the ProGet REST call shown in §5 is issued, Promote-ProGetPackage calls Test-PromotionWithinCeiling with the resolved CurrentTier (the stage attempting the promotion's destination feed) and the artifact's CeilingTier (read from the captured run state in _generated/buildmaster/<BuildMasterBuildId>/, see §6.1). The promotion proceeds only when CurrentTier <= CeilingTier in the canonical tier order Experimental < Development < Integration < QA < Production. A ceiling violation raises a structured terminating error carrying PackageName, PackageVersion, FromFeed, ToFeed, CurrentTier, CeilingTier, BuildId, Branch, and CommitSha, so the BuildMaster log records exactly why the promotion was refused. See VersionJsonAsCeiling.md for the full Stage × Ceiling decision matrix and the runbook for handling a refused promotion.

6. Versioning (no special-case for promotion)

NBGV computes the prerelease label and {height} from version.json in the Experimental preamble and the resulting version string is captured under the build-id scoped run-state folder. Later preambles may validate that the current workspace still resolves to the same version, but promotion/test arguments read the captured value, not a flat temp file or BuildMaster build variable. The canonical path is:

_generated/buildmaster/<BuildMasterBuildId>/

<BuildMasterBuildId> is the value returned by $BuildMasterId(build) in BuildMaster. The label declares the ceiling tier:

LabelCeiling tierWhere the artifact starts
SprintExperimentalFirst push lands in *-experimental
AlphaDevelopmentPromoted from Experimental
BetaIntegrationPromoted from Development
QAQAPromoted from Integration
(none)ProductionPromoted from QA

The label is metadata on the artifact; the current tier is the feed and BuildMaster stage it currently lives in. Promotion guards compare CurrentTier to CeilingTier before moving the artifact. The deprecated Get-BuildContext.Tier property aliases CeilingTier; new callers should use the explicit property names.

A consequence: the same artifact lives in the Experimental, Development, Integration, QA, and Production feeds simultaneously while it is being promoted. Each feed's listing is independent; the package's identity is defined by (PackageId, Version).

6.1 Where the captured version lives

SurfaceWhat it holdsSet byRead by
_generated/buildmaster/<BuildMasterBuildId>/_resolved_version.tmp or <ModuleName>.resolved-version.tmpfull SemVer, e.g. 0.1.0-Sprint.42plan preamble script after Get-BuildContextevery later stage of the same BuildMaster build
_generated/buildmaster/<BuildMasterBuildId>/build-context.jsonbuild id, BuildMaster build/execution numbers, branch, source path, current tier, ceiling tier, resolved version, prerelease label, allow/skip decisions, and relevant bundle/module pathsplan preamble scriptoperators, diagnostics, later stage scripts
The artifact's filenamesame SemVer minus the +<hash> build metadatadotnet pack / New-PSModuleNupkg at Experimentalpromotion calls; tier gates
BuildMaster release record metadatafull SemVer + SHA-256Experimental stage's "attach package" stepaudit / forensics

Captured generated state by plan family:

PlanCaptured generated state
C# packagecurrent tier, ceiling tier, resolved version, prerelease label, allow/skip flags
PowerShell modulecurrent tier, ceiling tier, resolved version, prerelease label, module nupkg path, allow/skip flags
Release Bundlecurrent tier, ceiling tier, resolved version, prerelease label, ReleaseBundle context JSON, bundle name, bundle version, bundle path, manifest path, allow/skip flags

The build-id folder is generated run state and is not committed. It prevents stale/cross-run state because two concurrent BuildMaster builds write to different folders. A retry of the same BuildMaster build id may refresh recomputable state, but the helper fails if an existing context captured a different resolved version.

6.2 Why this matters

Two pipeline runs in close succession against the same SHA can produce the same Sprint.{height} but with different +<gitshorthash> build metadata if the underlying tools see different working-tree states. Because +<hash> is part of the SemVer 2.0 identity, two such packages are different artifacts. Capturing the version once at Experimental and reading it everywhere downstream is what makes "promote the artifact" mean a single, identifiable thing. This is the rationale documented in §12 of ReviewedAndArchived/CriticalAnalysisOfImmutableBuildStrategy.md (the +<gitshorthash> problem).

6.3 Resolving "latest in feed X"

Because the same artifact is promoted into multiple feeds (§5), a floating reference resolved against any one feed sees promoted versions alongside versions originally pushed there. The resolution rule:

Under immutable build, "latest in feed X" means "highest version visible through feed X's resolution chain." A floating 0.*-* reference will always pick up the highest version, regardless of whether that version was originally pushed to feed X or promoted into it. This is intentional — once promoted, the artifact has feed-X identity. Consumers who want "the latest version that has not yet been promoted out of feed X" must filter by prerelease label (e.g. 0.*-Sprint*).

6.4 Binary determinism is required, not just version capture

Capturing $ResolvedPackageVersion (§6.1, §6.2) fixes the version identity of an artifact, but it does not by itself guarantee that two pipeline runs of the same source produce the same bytes. The +<gitshorthash> build metadata is stripped from the .nupkg filename, so two runs against the same SHA can produce packages with identical filenames whose inner DLLs differ. ProGet rejects a re-push as a duplicate, but a --skip-duplicate accept would silently diverge the inner DLL SHA-256 from the recorded one.

The Experimental build MUST use ContinuousIntegrationBuild=true (set automatically on BuildMaster agents) to ensure binary determinism. Without this, two pipeline runs of the same source can produce DLLs with different bytes and the same package version, which would silently corrupt the immutability invariant.

See CSharp-Packages-Build-Process.md §10.4 for the mechanics of the ContinuousIntegrationBuild property.

7. What changes in the Build/Pack/Push docs

The per-area docs (C# Build Process, C# Pack-and-Push, PowerShell Build Process, etc.) describe the once-per-release-unit mechanics. They no longer describe re-running pack/push at every tier; that section has been removed or reframed as "tier gates: tests against the existing artifact."

When you read those docs, mentally locate any section that talks about a "build at tier T" — that means "test the artifact at tier T," not "rebuild the artifact." This doc is the authoritative explanation.

8. Branch behavior at sprint / feature boundaries

Pipelines are durable. They are not created or deleted per sprint or per feature. What changes per sprint is metadata: prerelease suffixes, release records in BuildMaster, and the worktree the build agent runs in.

BoundaryAction on pipelinesAction on releases / metadata
Feature startNoneBuildMaster creates a new Release scoped to $FeatureSlug (distinct from the trunk Release). $FeatureSlug is computed from the branch name per E-DEC-01 (PascalCase, ≤16 chars, derived from the feature/ suffix). The first Experimental build produces 0.1.0-<FeatureSlug>.1. Feature artifacts share the trunk feeds; the prerelease suffix provides isolation.
Feature in progress (each sprint)NoneFeature artifacts are promoted through all five tiers under the feature suffix (0.1.0-<FeatureSlug>.NNN) using Promote-ProGetPackage. The QA gate is required before merge to stable — feature artifacts may be promoted to QA, but no Production promotion of feature artifacts is permitted until merge. DB migrations on the feature branch must be additive-only (no ALTER COLUMN, no DROP).
Sprint startNoneCreate a sprint release-train naming context; package versions inherit it.
During sprintNoneEach push triggers an Experimental build via the durable pipeline.
Feature end / merge to stableNoneBefore merge: DB migrations are squashed and re-sequenced to follow trunk's highest existing migration number (no gaps). version.json on trunk is updated so the prerelease label is Sprint (manual step, must precede the pipeline run). After merge, a trunk Experimental build is triggered; the first trunk artifact is Sprint.NNN where NNN resets to trunk's HEAD height. The feature BuildMaster Release is archived. The feature's <FeatureSlug>.NNN artifacts remain in feeds under their suffix but receive no further promotion.
Sprint endNoneCut a release/* branch from stable; build artifacts from the tag.
Release cutNoneBuild the Release Bundle once from the release-branch tag; promote the same artifact through the five tiers.
Full lifecycle details—See Long-Developing-Features.md for the complete feature-branch lifecycle, version-string rules (E-DEC-01), sprint-slice interaction (E-DEC-02), feed targets (E-DEC-03), merge mechanics (E-DEC-04), and DB-compatibility rule (E-DEC-05).

The single exception is the first time a new BuildMaster Application is introduced (e.g., a brand-new component getting its own release-bundle identity). That is a one-off configuration change, not a sprint-cadence operation.

9. Where the old "buildpertier" pattern still appears (and what to do)

If you see any of these in the codebase, it is legacy and should be treated as a documentation bug to fix:

  • An OtterScript stage that calls dotnet pack after the Experimental stage.
  • A "rebuild from source" step in a Development / Integration / QA / Production stage.
  • A doc that describes "the version that lands in QA may differ from the version that landed in Integration."
  • A version.json mutation in a non-Experimental stage.

The fix is always the same: replace the rebuild with a ProGet promotion call (or a no-op, if the artifact is already in the target feed).

11. Dependency Restoration Invariant

A package or module build has two feed decisions.

The target publish feed is the feed that receives the package produced by the build. It is selected from the package's NBGV prerelease label: Sprint publishes to Experimental, Alpha to Development, Beta to Integration, QA to QA, and stable releases to Stable.

The allowed restore feed set controls where the build may obtain direct and transitive dependencies. A build may restore internal packages only from its own tier or from a more stable tier. It may never restore an internal package from a less stable tier.

This rule is enforced by generated tier-specific NuGet configuration for NuGet packages and by tier-specific PSResourceRepository visibility for PowerShell modules.

11.1 Required C# restore policy

The build should generate or select a tier-specific NuGet configuration before restore:

Consumer tierEnabled internal sources for restore
Experimentalnuget-experimental, nuget-development, nuget-integration, nuget-qa, nuget-stable
Developmentnuget-development, nuget-integration, nuget-qa, nuget-stable
Integrationnuget-integration, nuget-qa, nuget-stable
QAnuget-qa, nuget-stable
Stablenuget-stable

nuget.org remains enabled for external packages, but internal package ID patterns should be mapped only to internal ProGet sources. For this repo the internal pattern is currently ATAP.*. For AceCommander or other consumers, the mapping probably also needs their internal prefixes, for example AceCommander.*.

11.2 Required PowerShell repository policy

The PowerShell build should register or enable only the tier-appropriate PSResource repositories during dependency restore/test/package validation:

Consumer tierEnabled PowerShellGet repositories for restore/validation
Sprint / Experimentalpowershellget-experimental, powershellget-development, powershellget-integration, powershellget-qa, powershellget-stable
Alpha / Developmentpowershellget-development, powershellget-integration, powershellget-qa, powershellget-stable
Beta / Integrationpowershellget-integration, powershellget-qa, powershellget-stable
QApowershellget-qa, powershellget-stable
Production / Stablepowershellget-stable

The build should then validate RequiredModules and any other package dependency metadata using that repository view. If dependency checks must be skipped for a special bootstrap scenario, the skip should be explicit, logged, and gated.

11.3 Direct and transitive dependency behavior

If a consumer C has a direct PackageReference to a supplier S, NuGet chooses a version of S according to the requested version range, central package management, lock file, and enabled sources. If S depends on T, NuGet recursively resolves T using the same source configuration and source mapping. Therefore the tier policy must apply to both direct and transitive dependencies. The effective policy is:

Allowed package source set = source mapping(package ID) intersect tier view(consumer tier)
Resolved package = best version satisfying the requested range in the allowed set

If C uses ProjectReference to S in the same checkout, feeds are bypassed for that edge because S is built from source.