AKS Engine on Azure Stack Hub
June 11, 2026 ยท View on GitHub
- Introduction
- Marketplace Prerequisites
- Service Principals and Identity Providers
- CLI flags
- Cluster Definition (aka API Model)
- Azure Stack Hub Instances Registered with Azure's China cloud
- Disconnected Azure Stack Hub Instances
- AKS Engine Versions
- Cloud Provider for Azure
- Volume Provisioners: Container Storage Interface Drivers (Azure Disk CSI Driver)
- Known Issues and Limitations
- Frequently Asked Questions
Introduction
Specific AKS Engine versions can be used to provision self-managed Kubernetes clusters on Azure Stack Hub. AKS Engine's generate, deploy, upgrade, and scale commands can be executed as if you were targeting Azure's public cloud. You are only required to slightly update your cluster definition to provide some extra information about your Azure Stack Hub instance.
The goal of this guide is to explain how to provision Kubernetes clusters to Azure Stack Hub using AKS Engine and to capture the differences between Azure and Azure Stack Hub. Bear in mind as well that not every AKS Engine feature or configuration option is currently supported on Azure Stack Hub. In most cases, these are not available because dependent Azure components are not part of Azure Stack Hub.
Marketplace prerequisites
Because Azure Stack Hub instances do not have infinite storage available, Azure Stack Hub administrators are in charge of managing it by selecting which marketplace items are downloaded from Azure's marketplace. The Azure Stack Hub administrator can follow this guide for a general explanation about how to download marketplace items from Azure.
Before you try to deploy the first Kubernetes cluster, make sure these marketplace items were made available to the target subscription by the Azure Stack Hub administrator.
Custom Script for Linux 2.0virtual machine extension- Required
AKS Base Imagevirtual machine
Service Principals and Identity Providers
Kubernetes uses a service principal identity to talk to Azure Stack Hub APIs to dynamically manage resources such as storage or load balancers. Therefore, you will need to create a service principal before you can provision a Kubernetes cluster using AKS Engine.
This guide explains how to create and manage service principals on Azure Stack Hub for both Azure Active Directory (AAD) and Active Directory Federation Services (ADFS) identity providers. This other guide is a good resource to understand the permissions that the service principal requires to deploy under your subscription.
Once you have created the required service principal, make sure to assign it the contributor role at the target subscription scope.
CLI flags
To indicate to AKS Engine that your target platform is Azure Stack Hub, all commands require CLI flag azure-env to be set to "AzureStackCloud".
If your Azure Stack Hub instance uses ADFS to authenticate identities, then flag identity-system is also required.
aks-engine-azurestack deploy \
--azure-env AzureStackCloud \
--api-model kubernetes.json \
--location local \
--resource-group kube-rg \
--identity-system adfs \
--client-id $SPN_CLIENT_ID \
--client-secret $SPN_CLIENT_SECRET \
--subscription-id $TENANT_SUBSCRIPTION_ID \
--output-directory kube-rg
Cluster Definition (aka API Model)
This section details how to tailor your cluster definitions in order to make them compatible with Azure Stack Hub. You can start off from this template.
Unless otherwise specified down below, standard cluster definition properties should also work with Azure Stack Hub. Please create an issue if you find that we missed a property that should be called out.
location
| Name | Required | Description |
|---|---|---|
| location | yes | The region name of the target Azure Stack Hub. |
kubernetesConfig
kubernetesConfig describes Kubernetes specific configuration.
| Name | Required | Description |
|---|---|---|
| addons | no | A few addons are not supported on Azure Stack Hub. See the complete list down below. |
| kubernetesImageBase | no | For AKS Engine versions lower than v0.48.0, this is a required field. It specifies the default image base URL to be used for all Kubernetes-related containers such as hyperkube, cloud-controller-manager, pause, addon-manager, etc. This property should be set to "mcr.microsoft.com/k8s/azurestack/core/". |
| networkPlugin | yes | Specifies the network plugin implementation for the cluster. Valid values are "kubenet" (default) for k8s software networking implementation and "azure", which provides an Azure native networking experience. |
| networkPolicy | no | Specifies the network policy enforcement tool for the cluster (currently Linux-only). Valid values are: "azure" (experimental) for Azure CNI-compliant network policy (note: Azure CNI-compliant network policy requires explicit "networkPlugin": "azure" configuration as well). |
| useInstanceMetadata | no | Use the Azure cloud provider instance metadata service for appropriate resource discovery operations. This property should be always set to "false". |
customCloudProfile
customCloudProfile contains information specific to the target Azure Stack Hub instance.
| Name | Required | Description |
|---|---|---|
| environment | no | The custom cloud type. This property should be always set to "AzureStackCloud". |
| identitySystem | yes | Specifies the identity provider used by the Azure Stack Hub instance. Valid values are "azure_ad" (default) and "adfs". |
| portalUrl | yes | The tenant portal URL. |
| dependenciesLocation | no | Specifies where to locate the dependencies required to during the provision/upgrade process. Valid values are "public" (default), "china", "german" and "usgovernment". |
masterProfile
masterProfile describes the settings for control plane configuration.
| Name | Required | Description |
|---|---|---|
| vmsize | yes | Specifies a valid Azure Stack Hub VM size. |
| distro | yes | Specifies the control plane's Linux distribution. "aks-ubuntu-18.04" is supported. This is a custom image based on UbuntuServer that come with pre-installed software necessary for Kubernetes deployments. |
agentPoolProfiles
agentPoolProfiles are used to create agents with different capabilities.
| Name | Required | Description |
|---|---|---|
| vmsize | yes | Describes a valid Azure Stack Hub VM size. |
| osType | no | Specifies the agent pool's Operating System. Supported values are "Windows" and "Linux". Defaults to "Linux". |
| distro | yes | Specifies the control plane's Linux distribution. "aks-ubuntu-18.04" is supported. This is a custom image based on UbuntuServer that come with pre-installed software necessary for Kubernetes deployments. |
| availabilityProfile | yes | Only "AvailabilitySet" is currently supported. |
| acceleratedNetworkingEnabled | yes | Use Azure Accelerated Networking feature for Linux agents. This property should be always set to "false". |
linuxProfile provides the linux configuration for each linux node in the cluster
| Name | Required | Description |
|---|---|---|
| enableUnattendedUpgrades | no | Configure each Linux node VM (including control plane node VMs) to run /usr/bin/unattended-upgrade in the background according to a daily schedule. If enabled, the default unattended-upgrades package configuration will be used as provided by the Ubuntu distro version running on the VM. More information here. By default, enableUnattendedUpgrades is set to true. |
| runUnattendedUpgradesOnBootstrap | no | Invoke an unattended-upgrade when each Linux node VM comes online for the first time. In practice this is accomplished by performing an apt-get update, followed by a manual invocation of /usr/bin/unattended-upgrade, to fetch updated apt configuration, and install all package updates provided by the unattended-upgrade facility, respectively. Defaults to "false". |
Azure Stack Hub Instances Registered with Azure's China cloud
If your Azure Stack Hub instance is located in China, then the dependenciesLocation property of your cluster definition should be set to "china". This switch ensures that the provisioning process fetches software dependencies from reachable hosts within China's mainland.
Disconnected Azure Stack Hub Instances
By default, the AKS Engine provisioning process relies on an internet connection to download the software dependencies required to create or upgrade a cluster (Kubernetes images, etcd binaries, network plugins and so on).
If your Azure Stack Hub instance is air-gapped or if network connectivity in your geographical location is not reliable, then the default approach will not work, take a long time or timeout due to transient networking issues.
To overcome these issues, you should set the distro property of your cluster definition to "aks-ubuntu-18.04". This will instruct AKS Engine to deploy VM nodes using a base OS image called AKS Base Image. This custom image, generally based on Ubuntu Server, already contains the required software dependencies in its file system. Hence, internet connectivity won't be required during the provisioning process.
The AKS Base Image marketplace item has to be available in your Azure Stack Hub's Marketplace before it could be used by AKS Engine. Your Azure Stack Hub administrator can follow this guide for a general explanation about how to download marketplace items from Azure.
Each AKS Engine release is validated and tied to a specific version of the AKS Base Image. Therefore, you need to take note of the base image version required by the AKS Engine release that you plan to use, and then download exactly that base image version. New builds of the AKS Base Image are frequently released to ensure that your disconnected cluster can be upgraded to the latest supported version of each component.
Make sure linuxProfile.runUnattendedUpgradesOnBootstrap is set to "false" when you deploy, or upgrade, a cluster to air-gapped Azure Stack Hub clouds.
AKS Engine Versions
* Starting from Kubernetes v1.21, only Cloud Provider for Azure is supported on Azure Stack Hub.
** Starting from Kubernetes v1.24, ONLY the
containerdcontainer runtime is supported. Please refer to the section Upgrading Kubernetes clusters created with docker container runtime for more details.
Cloud Provider for Azure
The Cloud Provider for Azure project (aka cloud-controller-manager, out-of-tree cloud provider or external cloud provider) implements the Kubernetes cloud provider interface for Azure clouds. The out-of-tree implementation is the replacement for the deprecated in-tree implementation.
On Azure Stack Hub, starting from Kubernetes v1.21, AKS Engine-based clusters will exclusively use cloud-controller-manager. Hence, to deploy a Kubernetes v1.21+ cluster, it is required to set orchestratorProfile.kubernetesConfig.useCloudControllerManager to true in the API Model (example). AKS Engine's upgrade process will automatically update the useCloudControllerManager flag and the "--cloud-provider" flags in the api model to ensure external cloud provider is used.
Upgrade considerations: the process of upgrading a Kubernetes cluster from v1.20 (or lower version) to v1.21 (or greater version) will cause downtime to workloads relying on the
kubernetes.io/azure-diskin-tree volume provisioner. Before upgrading to Kubernetes v1.21+, it is highly recommended to perform a full backup of the application data and validate in a pre-production environment that the cluster storage resources (PV and PVC) can be migrated to the a new volume provisioner. Learn how to migrate to the Azure Disk CSI driver here.
Volume Provisioners: Container Storage Interface Drivers (Azure Disk CSI Driver)
The in-tree volume provisioner is only compatible with the in-tree cloud provider. Therefore, a v1.21+ cluster has to include a Container Storage Interface (CSI) Driver if user workloads rely on persistent storage.
AKS Engine will not enable any CSI driver by default on Azure Stack Hub. For workloads that require a CSI driver, it is possible to either:
- For AKS Engine versions 0.75.3 and above: explicitly enable the
azuredisk-csi-driveraddon (Linux and/or Windows cluster)- If you are using Windows HostProcess containers in your workloads, do not use the addon. Instead use
Helmto install theazuredisk-csi-driverchart and set the following value:--set windows.useHostProcessContainers=true. (Windows cluster only)
- If you are using Windows HostProcess containers in your workloads, do not use the addon. Instead use
- For AKS Engine versions 0.70.0 and above: explicitly enable the
azuredisk-csi-driveraddon (Linux cluster only) - For AKS Engine versions 0.70.0 and above: use
Helmto install theazuredisk-csi-driverchart (Linux and/or Windows clusters).
Azure Disk CSI Driver: Version Mapping
The following table shows the supported Azure Disk CSI Driver version for each Kubernetes version.
- If using addon, the Azure Disk CSI Driver version is automatically selected based on the cluster's Kubernetes version.
- If using Helm, the Azure Disk CSI Driver version has to be specified manually based on the cluster's Kubernetes version.
| Kubernetes Version | Azure Disk CSI Driver Version |
|---|---|
| <= 1.25 | 1.10.0 |
| 1.26 | 1.26.5 |
| 1.27 | 1.28.3 |
| 1.28 - 1.30 | 1.29.12 |
| 1.31 | 1.31.12 |
| 1.32 | 1.32.11 |
| 1.33 | 1.33.7 |
| 1.34 | 1.34.3 |
| 1.35 | 1.34.3 |
| 1.34 | 1.34.3 |
| 1.35 | 1.34.3 |
- Exception: For AKS Engine v0.77.0, kubernetes version 1.25 uses Azure Disk CSI Driver version v1.26.5
- Note: Only the versions in the mapping have been tested. While other version combinations may work, they are not officially supported.
Migrate Persistent Storage to the Azure Disk CSI driver
The process of upgrading an AKS Engine-based cluster from v1.20 (or lower version) to v1.21 (or greater version) will cause downtime to workloads relying on the kubernetes.io/azure-disk in-tree volume provisioner as this provisioner is not part of the Cloud Provider for Azure.
If the data persisted in the underlying Azure disks should be preserved, then the following extra steps are required once the cluster upgrade process is completed:
- Install the Azure Disk CSI driver.
- Remove the deprecated in-tree storage classes.
- Recreate the persistent volumes and claims.
1. Install Azure Disk CSI driver manually
The following script uses Helm to install the Azure Disk CSI Driver:
DRIVER_VERSION=v1.26.5 # if using k8s v1.26
DRIVER_VERSION=v1.28.3 # if using k8s v1.27
DRIVER_VERSION=v1.29.12 # if using k8s v1.28 - v1.30
DRIVER_VERSION=v1.31.12 # if using k8s v1.31
DRIVER_VERSION=v1.32.11 # if using k8s v1.32
DRIVER_VERSION=v1.33.7 # if using k8s v1.33
DRIVER_VERSION=v1.34.3 # if using k8s v1.34 - v1.35
helm repo add azuredisk-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/charts
helm install azuredisk-csi-driver azuredisk-csi-driver/azuredisk-csi-driver \
--namespace kube-system \
--set cloud=AzureStackCloud \
--set controller.runOnMaster=true \
--set node.supportZone=false \
--set windows.getNodeInfoFromLabels=true \
--set linux.getNodeInfoFromLabels=true \
# --set windows.useHostProcessContainers=true \ # if using Windows HostProcess containers
--version ${DRIVER_VERSION}
2. Replace Storage Classes
The kube-addon-manager will automatically create the Azure Disk CSI driver storage classes (disk.csi.azure.com) once the in-tree storage classes (kubernetes.io/azure-disk) are manually deleted:
IN_TREE_SC="default managed-premium managed-standard"
# Delete deprecated "kubernetes.io/azure-disk" storage classes
kubectl delete storageclasses ${IN_TREE_SC}
# Wait for addon manager to create the "disk.csi.azure.com" storage class resources
kubectl get --watch storageclasses
3. Recreate Persistent Volumes
Once the Azure Disk CSI Driver is installed and the storage classes replaced, the next step is to recreate the persistent volumes (PV) and persistent volumes claims (PVC) using the Azure Disk CSI driver (or alternative CSI solution).
This is a multi-step process that can be different depending on how these resources were initially deployed. The high level steps are:
- Delete the deployment or statefulset that references the PV + PVC pairs to migrate (backup resource definition if necessary).
- Ensure the PVs'
persistentVolumeReclaimPolicyproperty is set to valueRetain(example). - Delete the PV + PVC pairs to migrate (backup resource definitions if necessary).
- To migrate, update the PVs' resource definition by removing the
azureDiskobject and adding acsiobject with reference to the original AzureDisk (example). - Recreate, in the following order, the PV resource/s, PVC resource/s (if necessary), and finally the deployment or statefulset.
The following migration script is provided as a template.
After running the migration script, if the pod is stuck with error "Unable to attach or mount volumes", make sure Azure Disk CSI Driver was installed and storage classes were recreated.
Azure Disk CSI Driver: Details
As a replacement of the current in-tree volume provisioner, Azure Disk Container Storage Interface (CSI) Driver is available on Azure Stack Hub. Please find details in the following table.
| Azure Disk CSI Driver | |
|---|---|
| Stage on Azure Stack Hub | GA |
| Project Repository | azuredisk-csi-driver |
| CSI Driver Version | v1.0.0+ |
| Access Mode | ReadWriteOnce |
| Windows Agent Node | Support |
| Dynamic Provisioning | Support |
| Considerations | Azure Disk CSI Driver Limitations |
| Slack Support Channel | #provider-azure |
To deploy a CSI driver to an air-gapped cluster, make sure that your
helmchart is referencing container images that are reachable from the cluster nodes.
Azure Disk CSI Driver: Requirements
- Azure Stack build 2011 and later.
- AKS Engine version v0.60.1 and later.
- Kubernetes version 1.18 and later.
- Since the Controller server of CSI Drivers requires 2 replicas, a single node master pool is not recommended.
- Helm 3
Azure Disk CSI Driver: Examples
In this section, please follow the example commands to deploy a StatefulSet application consuming CSI Driver.
# Install CSI Driver
DRIVER_VERSION=v1.26.5 # if using k8s v1.26
DRIVER_VERSION=v1.28.3 # if using k8s v1.27
DRIVER_VERSION=v1.29.12 # if using k8s v1.28 - v1.30
DRIVER_VERSION=v1.31.12 # if using k8s v1.31
DRIVER_VERSION=v1.32.11 # if using k8s v1.32
DRIVER_VERSION=v1.33.7 # if using k8s v1.33
DRIVER_VERSION=v1.34.3 # if using k8s v1.34 - v1.35
helm repo add azuredisk-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/charts
helm install azuredisk-csi-driver azuredisk-csi-driver/azuredisk-csi-driver \
--namespace kube-system \
--set cloud=AzureStackCloud \
--set controller.runOnMaster=true \
--set node.supportZone=false \
--set windows.getNodeInfoFromLabels=true \
--set linux.getNodeInfoFromLabels=true \
# --set windows.useHostProcessContainers=true \ # if using Windows HostProcess containers
--version ${DRIVER_VERSION}
# Deploy Storage Class
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/storageclass-azuredisk-csi-azurestack.yaml
# Deploy example StatefulSet application
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/statefulset.yaml
# Validate volumes and applications
# You should see a sequence of timestamps are persisted in the volume.
kubectl exec statefulset-azuredisk-0 -- tail /mnt/azuredisk/outfile
# Delete example StatefulSet application
kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/statefulset.yaml
# Delete Storage Class
# Before delete the Storage Class, please make sure Pods that consume the Storage Class have been terminated.
kubectl delete -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/storageclass-azuredisk-csi-azurestack.yaml
# Uninstall CSI Driver
helm uninstall azuredisk-csi-driver --namespace kube-system
helm repo remove azuredisk-csi-driver
Known Issues and Limitations
This section lists all known issues you may find when you use the GA version.
Unsupported Addons
AKS Engine includes a number of optional addons that can be deployed as part of the cluster provisioning process.
The list below includes the addons currently unsupported on Azure Stack Hub:
- AAD Pod Identity
- Cluster Autoscaler
- KeyVault Flex Volume
- SMB Flex Volume
- Azure Monitor for containers
Chrony daemon fails to restart
The chrony daemon on a Linux node may fail to restart with error message: "Could not open /dev/ptp_hyperv: No such file or directory".
The workaround for this issue is to manually reboot the affected Linux nodes.
This operation will regenerate the /dev/ptp_hyperv symlink and allow the chrony daemon to restart successfully.
OSProfile exceeds maximum characters length error
Addons enabled in the API Model are Base64 encoded and included in the VMs ARM template. There is a length limit of 87380 characters for the custom data, thus if too many addons are enabled in the API Model, the aks-engine-azurestack operations could fail with the below error:
Custom data in OSProfile must be in Base64 encoding and with a maximum length of 87380 characters
In such cases, try reduce the number of enabled addons or remove all of them in the API Model.
get-versions command
By default, aks-engine-azurestack get-versions shows which Kubernetes versions are supported by each AKS Engine release on Azure's public cloud. Include flag --azure-env to get the list of supported Kubernetes versions on a custom cloud such as an Azure Stack Hub cloud (aks-engine-azurestack get-versions --azure-env AzureStackCloud). Upgrade paths for Azure Stack Hub can also be found here.
Upgrade from private-preview Kubernetes cluster with Windows nodes
There is no official support for private-preview Kubernetes cluster with Windows nodes created with AKS Engine v0.43.1 to upgrade with AKS Engine v0.55.0. Users are encouraged to deploy new Kubernetes cluster with Windows nodes with the latest AKS Engine version.
Upgrading Kubernetes clusters created with the Ubuntu 20.04 distro
Starting with AKS Engine v0.82.1, the Ubuntu 20.04 distro is not longer a supported option as the OS reached its end-of-life. For AKS Engine v0.82.1 or later versions, aks-engine-azurestack upgrade will automatically overwrite the unsupported aks-ubuntu-20.04 distro value with aks-ubuntu-22.04.
Upgrading Kubernetes clusters created with the Ubuntu 18.04 distro
Starting with AKS Engine v0.75.3, the Ubuntu 18.04 distro is not longer a supported option as the OS reached its end-of-life. For AKS Engine v0.75.3 or later versions, aks-engine-azurestack upgrade will automatically overwrite the unsupported aks-ubuntu-18.04 distro value with aks-ubuntu-20.04.
Upgrading Kubernetes clusters created with the Ubuntu 16.04 distro
Starting with AKS Engine v0.63.0, the Ubuntu 16.04 distro is not longer a supported option as the OS reached its end-of-life. For AKS Engine v0.67.0 or later versions, aks-engine upgrade will automatically overwrite the unsupported aks-ubuntu-16.04 distro value with with aks-ubuntu-18.04. For AKS Engine v0.75.3 or later versions, aks-engine-azurestack upgrade will automatically overwrite the unsupported aks-ubuntu-16.04 distro value with aks-ubuntu-20.04.
Upgrading Kubernetes clusters created with docker container runtime
In Kubernetes v1.24, the dockershim component was removed from kubelet. As a result, the docker container runtime is no longer a supported option. See Kubernetes v1.24 release notes for more information. For AKS Engine v0.75.3 or later versions, aks-engine-azurestack upgrade will automatically overwrite the unsupported docker containerRuntime value with containerd.
For AKS Engine release v0.75.3, clusters with windows nodes on Kubernetes v1.23 can use the Windows base image with Docker runtime. Clusters with windows nodes on Kubernetes v1.24 can use the Windows base image with Containerd runtime.
Frequently Asked Questions
Sample extensions are not working
Extensions in AKS Engine provide an easy way to include your own customization at provisioning time.
Because Azure and Azure Stack Hub currently rely on a different version of the Compute Resource Provider API, you may find that some of sample extensions fail to deploy correctly.
This can be resolved by making a small modification to the extension template.json file. Replacing all usages of template parameter apiVersionDeployments by the hard-code value 2017-12-01 (or whatever API version Azure Stack Hub runs at the time you try to deploy) should be all you need.
Once you are done updating the extension template, host the extension directory in your own Github repository or storage account. Finally, at deployment time, make sure that your cluster definition points to the new rootURL.
The cluster nodes do not contain the latest Ubuntu OS security patches
If an aks-ubuntu-18.04 image is created by the AKS Engine team prior to the release of an OS security patch, then the image won't include those security patches until an unattended upgrade is triggered.
When linuxProfile.enableUnattendedUpgrades is set to true, unattended upgrades will be checked and/or installed once a day. To ensure that the nodes are rebooted in a non-disruptive way, you can deploy kured or similar solutions.
To deploy a cluster that includes the latests OS security patches right from the beginning, set linuxProfile.runUnattendedUpgradesOnBootstrap to "true" (see example).
To apply the latest OS security patches to an existing cluster, you can either do it manually or use the aks-engine-azurestack upgrade command. A manual upgrade can be done by executing apt-get update && apt-get upgrade and rebooting the node if necessary. If you use the aks-engine-azurestack upgrade command, set linuxProfile.runUnattendedUpgradesOnBootstrap to "true" in the generated apimodel.json and execute aks-engine-azurestack upgrade (a forced upgrade to the current Kubernetes version also works).
Troubleshooting
This how-to guide has a good high-level explanation of how AKS Engine interacts with the Azure Resource Manager (ARM) and lists a few potential issues that can cause AKS Engine commands to fail.
Please refer to the get-logs command documentation to simplify the logs collection task.