Cluster Definitions

Cluster Defintions for apiVersion "vlabs"

Here are the cluster definitions for apiVersion "vlabs":

apiVersion

Name	Required	Description
apiVersion	yes	The version of the template. For "vlabs" the value is "vlabs"

orchestratorProfile

orchestratorProfile describes the orchestrator settings.

Name	Required	Description
orchestratorType	no	Defaults to "Kubernetes"; a legacy property to accommodate previously supported orchestrators (e.g., DC/OS)
orchestratorRelease	no	Specifies the orchestrator release for the cluster
orchestratorVersion	no	Specifies the orchestrator version for the cluster

Here are the valid values for the orchestrator types:

1. Kubernetes - this represents the Kubernetes orchestrator.

To learn more about supported versions, run the get-versions command:

$ aks-engine-azurestack get-versions

kubernetesConfig

kubernetesConfig describes Kubernetes specific configuration.

Name	Required	Description
addons	no	Configure various Kubernetes addons configuration. See addons configuration below
components	no	Configure core Kubernetes components. See components configuration below
apiServerConfig	no	Configure various runtime configuration for apiserver. See apiServerConfig below
cloudControllerManagerConfig	no	Configure various runtime configuration for cloud-controller-manager. See cloudControllerManagerConfig below
clusterSubnet	no	The IP subnet used for allocating IP addresses for pod network interfaces. The subnet must be in the VNET address space. With Azure CNI enabled, the default value is 10.240.0.0/12. Without Azure CNI, the default value is 10.244.0.0/16.
containerRuntime	no	The container runtime to use as a backend. The default is docker. Also supported is containerd. Windows support for containerd is Experimental - see Windows ContainerD
containerRuntimeConfig	no	A map of key-value pairs to drive configuration of the container runtime. Currently accepts a single key, "dataDir", which configures the root data directory for the container runtime. dataDir must be an absolute path. This is only implemented on Linux. See an example which places docker on the tmp disk of a Linux VM.
controllerManagerConfig	no	Configure various runtime configuration for controller-manager. See controllerManagerConfig below
customWindowsPackageURL	no	Configure custom windows Kubernetes release package URL for deployment on Windows. The format of this file is a zip file with multiple items (binaries, cni, infra container) in it. This setting will be deprecated in a future release of aks-engine-azurestack where the binaries will be pulled in the format of Kubernetes releases that only contain the kubernetes binaries.
WindowsNodeBinariesURL	no	Windows Kubernetes Node binaries can be provided in the format of Kubernetes release (example: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#node-binaries-1). This setting allows overriding the binaries for custom builds.
WindowsContainerdURL	no (for development only)	Experimental - see Windows ContainerD
WindowsSdnPluginURL	no (for development only)	Experimental - see Windows ContainerD
dnsServiceIP	no	IP address for coredns or kube-dns to listen on. If specified must be in the range of serviceCidr
mobyVersion	no (for development only)	Enables an explicit moby version, e.g. 3.0.3. Default is 3.0.5. This kubernetesConfig property is for development only, and applies only to cluster creation: aks-engine-azurestack upgrade will always statically set mobyVersion to the default version at the time of upgrade, to ensure that upgraded clusters have the most recent, validated version of moby.
linuxMobyURL	no (for development only)	Allow the use of custom moby package for Linux nodes.
containerdVersion	no (for development only)	Enables an explicit containerd version, e.g. 1.1.4. Default is 1.1.5. This kubernetesConfig property is for development only, and applies only to cluster creation: aks-engine-azurestack upgrade will always statically set containerdVersion to the default version at the time of upgrade, to ensure that upgraded clusters have the most recent, validated version of containerd. This value is currently ignored for Windows. It is also ignored when linuxContainerdURL is defined.
linuxContainerdURL	no (for development only)	Allow the use of custom containerd package for Linux nodes.
dockerBridgeSubnet	no	The specific IP and subnet used for allocating IP addresses for the docker bridge network created on the kubernetes master and agents. Default value is 172.17.0.1/16. This value is used to configure the docker daemon using the --bip flag
enableAggregatedAPIs	no	Enable Kubernetes Aggregated APIs. enableRbac must be set to true to use aggregated APIs. Aggregated API functionality is required by Service Catalog. (boolean - default is true)
enableDataEncryptionAtRest	no	Enable kubernetes data encryption at rest.This is currently an alpha feature. (boolean - default == false)
enableEncryptionWithExternalKms	no	Enable kubernetes data encryption at rest with external KMS. This is currently a beta feature. (boolean - default == false). If enableEncryptionWithExternalKms is enabled, then you must also configure your cluster with either service principals or user-assigned identity in order to use aks-engine-azurestack upgrade. If you use system-assigned identity, then you will not be able to upgrade your cluster using aks-engine-azurestack upgrade. See useManagedIdentity and userAssignedID below for a more thorough description of user-assigned and system-assigned identity
enablePodSecurityPolicy	no	Deprecated, see the pod-security-policy addon for a description of the AKS Engine-configured PodSecurityPolicy spec that is bootstrapped as a Kubernetes addon
enableRbac	no	Enable Kubernetes RBAC (boolean - default == true) RBAC support is required for Kubernetes 1.15.0 and greater, so enableRbac=false is not an allowed configuration for clusters >= 1.15.0. If you upgrade a cluster to 1.15.0 or greater from a version less than 1.15, and RBAC is disabled, the cluster configuration will be statically modified to enable RBAC as a result of running aks-engine-azurestack upgrade.
etcdDiskSizeGB	no	Size in GB to assign to etcd data volume. Defaults (if no user value provided) are: 256 GB for clusters up to 3 nodes; 512 GB for clusters with between 4 and 10 nodes; 1024 GB for clusters with between 11 and 20 nodes; and 2048 GB for clusters with more than 20 nodes
etcdStorageLimitGB	no	Size in GB to limit etcd data storage to. Defaults to 2, maximum is 8. See here for more information.
etcdEncryptionKey	no	Enryption key to be used if enableDataEncryptionAtRest is enabled. Defaults to a random, generated, key
etcdVersion	no (for development only)	Enables an explicit etcd version, e.g. 3.2.23. Default is 3.3.25. This kubernetesConfig property is for development only, and recommended only for ephemeral clusters. However, you may use aks-engine-azurestack upgrade on a cluster with an API model that includes a user-modified etcdVersion value. If aks-engine-azurestack upgrade determines that the user-modified version is greater than the current AKS Engine default, aks-engine-azurestack upgrade will not replace the newer version with an older version. However, if aks-engine-azurestack upgrade determines that the user-modified version is older than the current AKS Engine default, it will build the newly upgraded master node VMs with the newer, AKS Engine default version of etcd.
gcHighThreshold	no	Sets the --image-gc-high-threshold value on the kublet configuration. Default is 85. See kubelet Garbage Collection
gcLowThreshold	no	Sets the --image-gc-low-threshold value on the kublet configuration. Default is 80. See kubelet Garbage Collection
kubeletConfig	no	Configure various runtime configuration for kubelet. See kubeletConfig below
kubeReservedCgroup	no	The name of a systemd slice to create for containment of both kubelet and the container runtime. When this value is a non-empty string, a file will be dropped at /etc/systemd/system/$KUBE_RESERVED_CGROUP.slice creating a systemd slice. Both kubelet and docker will run in this slice. This should not point to an existing systemd slice. If this value is unspecified or specified as the empty string, kubelet and the container runtime will run in the system slice by default.
kubernetesImageBase	no	Specifies the default image base URL (everything preceding the actual image filename) to be used for all kubernetes-related containers such as hyperkube, cloud-controller-manager, kube-addon-manager, etc. e.g., registry.k8s.io/
loadBalancerSku	no	Sku of Load Balancer and Public IP. Candidate values are: basic and standard. If not set, it will be default to "standard". NOTE: Because VMs behind standard SKU load balancer will not be able to access the internet without an outbound rule configured with at least one frontend IP, AKS Engine creates a Load Balancer with an outbound rule and with agent nodes added to the backend pool during cluster creation, as described in the Outbound NAT for internal Standard Load Balancer scenarios doc
loadBalancerOutboundIPs	no	Number of outbound IP addresses (e.g., 3) to use in Standard LoadBalancer configuration. If not set, AKS Engine will configure a single outbound IP address. You may want more than one outbound IP address if you are running a large cluster that is processing lots of connections. See here for more documentation about how adding more outbound IP addresses can increase the number of SNAT ports available for use by the Standard Load Balancer in your cluster. Note: this value is only configurable at cluster creation time, it can not be changed using aks-engine-azurestack upgrade.
networkPlugin	no	Specifies the network plugin implementation for the cluster. Valid values are: "azure" (default), which provides an Azure native networking experience "kubenet" for k8s software networking implementation. "cilium" for using the default Cilium CNI IPAM (requires the "cilium" networkPolicy as well) "antrea" for using the Antrea network plugin (requires the "antrea" networkPolicy as well)
networkPolicy	no	Specifies the network policy enforcement tool for the cluster (currently Linux-only). Valid values are: "calico" for Calico network policy. "cilium" for cilium network policy (uses the "cilium" networkPlugin exclusively). "antrea" for Antrea network policy (uses the "antrea" networkPlugin exclusively). "azure" (experimental) for Azure CNI-compliant network policy (note: Azure CNI-compliant network policy requires explicit "networkPlugin": "azure" configuration as well). See network policy examples for more information.
privateCluster	no	Build a cluster without public addresses assigned. See privateClusters below.
schedulerConfig	no	Configure various runtime configuration for scheduler. See schedulerConfig below
serviceCidr	no	IP range for Service IPs, Default is "10.0.0.0/16". This range is never routed outside of a node so does not need to lie within clusterSubnet or the VNET
useInstanceMetadata	no	Use the Azure cloudprovider instance metadata service for appropriate resource discovery operations. Default is true
useManagedIdentity	no	Defaults to true. Includes and uses MSI identities for all interactions with the Azure Resource Manager (ARM) API. Instead of using a static service principal written to /etc/kubernetes/azure.json, Kubernetes will use a dynamic, time-limited token fetched from the MSI extension running on master and agent nodes. When creating your cluster using either aks-engine-azurestack deploy or az group deployment create (or equivalent), your ARM deployment will require a service principal that can create role assignment resources in the resource group. If you use this configuration by itself (without including a userAssignedID value, see below), that is a system-assigned identity configuration, derived from the VM itself that Kubernetes runtime is installed onto.
userAssignedID	no	When useManagedIdentity is set to true, including a userAssignedID value indicates that user-assigned identity will be the type of managed identity used for cluster nodes, and appropriate pods. If the string value of "userAssignedID" is a fully qualified resource ID (e.g., "/subscriptions/7a8f2518-7462-11ea-bc55-0242ac130003/resourceGroups/my-resource-group/providers/Microsoft.ManagedIdentity/userAssignedIdentities/my-user-assigned-identity"), then the cluster will re-use that pre-existing user assigned managed identity resource; if the string value of "userAssignedID" is a simple string (e.g., "my-new-user-assigned-identity"), then a new user assigned managed identity resource will be created in the cluster resource group, with a name that matches that string value.
azureCNIURLLinux	no	Deploy a private build of Azure CNI on Linux nodes. This should be a full path to the .tar.gz
azureCNIURLWindows	no	Deploy a private build of Azure CNI on Windows nodes. This should be a full path to the .tar.gz
maximumLoadBalancerRuleCount	no	Maximum allowed LoadBalancer Rule Count is the limit enforced by Azure Load balancer. Default is 250
kubeProxyMode	no	kube-proxy --proxy-mode value, either "iptables" or "ipvs". Default is "iptables". See https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/ for further reference.
outboundRuleIdleTimeoutInMinutes	no	Specifies a value for IdleTimeoutInMinutes to control the outbound flow idle timeout of the agent standard loadbalancer. This value is set greater than the default Linux idle timeout (15.4 min): https://pracucci.com/linux-tcp-rto-min-max-and-tcp-retries2.html
cloudProviderBackoff	no	Use the Azure cloudprovider exponential backoff implementation when encountering retry-able errors from the Azure API. Defaults to true for Kubernetes v1.14.0 and greater.
cloudProviderBackoffMode	no	Which version of the Azure cloudprovider backoff implementation to use: the options are "v1" or "v2" (Kubernetes v1.14.0 or greater only). "v2" is a more recent backoff implementation which better honors Azure API HTTP headers to align backoff timings with the Azure API. Defaults to "v2" for Kubernetes v1.14.0 and greater, and "v1" for earlier versions of Kubernetes.
cloudProviderBackoffRetries	no	How many backoff retries before terminally failing the original Azure API operation. Defaults to 6.
cloudProviderBackoffJitter	no	Only available in the "v1" cloudProviderBackoffMode: how much random variation to inject in retry timings, to better distribute stacked retries. Defaults to 1 when using cloudProviderBackoffMode "v1", which effectively disables jitter.
cloudProviderBackoffDuration	no	The base duration, in seconds, in between retry attempts. Defaults to 5.
cloudProviderBackoffExponent	no	Only available in the "v1" cloudProviderBackoffMode: the factor to multiply cloudProviderBackoffDuration by for each retry iteration. Defaults to 1.5 when using cloudProviderBackoffMode "v1".
cloudProviderRateLimit	no	Use the Azure cloudprovider rate limiter to reduce the rate of calls to Azure APIs. Defaults to true.
cloudProviderRateLimitBucket	no	The size of the overflow read request queue when cloudProviderRateLimit is enabled. Defaults to the calculation "100 * number of agent pools in cluster configuration.
cloudProviderRateLimitQPS	no	QPS for Azure cloudprovider read request rate limiter enforcement. Defaults to a minimum factor of 0.1 (i.e., 10%) of the cloudProviderRateLimitBucket (e.g., given a cloudProviderRateLimitBucket of 100, cloudProviderRateLimitQPS defaults to 10), or 3, whichever is greater.
cloudProviderRateLimitBucketWrite	no	The size of the overflow write request queue when cloudProviderRateLimit is enabled. Follows the same defaults calculation as cloudProviderRateLimitBucket.
cloudProviderRateLimitQPSWrite	no	QPS for Azure cloudprovider write request rate limiter enforcement. Follows the same defaults calculation as cloudProviderRateLimitQPS.
cloudProviderDisableOutboundSNAT	no	For clusters w/ Standard LB only: enforces the disabling of outbound NAT for that load balancing rule. See here for more details. Defaults to false.
microsoftAptRepositoryURL	no	You may configure certain Microsoft-curated apt packages to be sourced from a custom repository so long as it acts as a mirror to the data at "https://packages.microsoft.com/" (the default value).
enableMultipleStandardLoadBalancers	no	Using multiple standard load balancers per cluster. The loadBalancerSku must be standard. Default to false.
tags	no	Specify the tags which will be applied to all of the resources managed by the cloud provider, with the format a=b,c=d.

addons

addons is an interface to define user-configurable Kubernetes componentry. It is a child property of kubernetesConfig. Below is a list of currently available addons:

Name of addon	Enabled by default?	How many pods	Description
tiller	false	1	Delivers the Helm server-side component: tiller. See https://github.com/kubernetes/helm for more info
kubernetes-dashboard	false	1	Deprecated. We recommend installing dashboard manually, see: https://github.com/kubernetes/dashboard for more info.
rescheduler	false	1	Deprecated, no longer available after aks-engine-azurestack v0.60.0.
cluster-autoscaler	false	1	Delivers the Kubernetes cluster autoscaler component. See https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/azure for more info; only supported for VMSS clusters on the first agent pool.
nvidia-device-plugin	Enabled by default if you have at least one N-series VM node pool. You may explicitly disable this addon if you want to manage your own GPU driver implementation, for example using the nvidia gpu-operator solution.	1	Delivers the Kubernetes NVIDIA device plugin component. See https://github.com/NVIDIA/k8s-device-plugin for more info.
container-monitoring	false	1	Deprecated
smb-flexvolume	false	as many as linux agent nodes	Access SMB server by using CIFS/SMB protocol (Note: CIFS/SMB FlexVolume driver is in maintenance mode.)
aad-pod-identity	false	1 + 1 on each linux agent nodes	Assign Azure Active Directory Identities to Kubernetes applications. (Note: this addon is no longer maintained. We recommend using the official helm chart to install and maintain aad-pod-identity on your aks-engine-azurestack cluster.)
scheduled-maintenance	false	1 + 1 on each linux agent nodes	Cordon and drain node during planned/unplanned azure maintenance
azuredisk-csi-driver	true if using a Kubernetes cluster (v1.13+) with useCloudControllerManager enabled	1 + 1 on each linux agent nodes	Allows Kubernetes to use Azure Disk volume
azurefile-csi-driver	true if using a Kubernetes cluster (v1.13+) with useCloudControllerManager enabled	1 + 1 on each linux agent nodes	Allows Kubernetes to use Azure File volume
azure-policy	false	2	Open Policy Agent Gatekeeper with Azure Policy integration
node-problem-detector	false	as many as linux agent nodes	Reports problems on Kubernetes nodes to kube-apiserver
kube-dns	false; if set to true, coredns must be set to false, i.e., only one cluster DNS addon may be used on a given cluster	1	Cluster DNS services
coredns	true; if set to false, kube-dns must be set to true, i.e., you need at least one (and only one) of kube-dns or coredns enabled. For more configuration info, see coredns configuration below	1	Cluster DNS services
kube-proxy	true	1	a network proxy that runs on each node in your cluster
pod-security-policy	required for Kubernetes v1.15+; defaults to false for Kubernetes 1.14 and earlier	0	a cluster-level resource that controls security-sensitive aspects of the pod specification
audit-policy	true	0	defines rules about what events should be recorded and what data they should include
azure-cloud-provider	true	0	Delivers required ClusterRole, ClusterRoleBinding, and StorageClass resources required for running the Azure cloudprovider runtime. May not be disabled.
aad	true if adminGroupID is specified in the aadProfile configuration	0	ClusterRoleBinding specification that adds an admin group matching the adminGroupID
calico	true if networkPolicy is "calico";	6	A NetworkPolicy implementation by the Calico project (currently supports v3.8)
cilium	true if networkPolicy is "cilium"; currently validated against Kubernetes v1.13, v1.14, and v1.15	0	A NetworkPolicy CRD implementation by the Cilium project (currently supports v1.4)
csi-secrets-store	false	as many as linux agent nodes	Integrates secrets stores (Azure keyvault) via a Container Storage Interface (CSI) volume. (Note: this addon is no longer maintained. We recommend using the official helm chart to install and maintain secrets-store-csi on your aks-engine-azurestack cluster.)
azure-arc-onboarding	false	7	Attaches the cluster to Azure Arc enabled Kubernetes.

To give a bit more info on the addons property: We've tried to expose the basic bits of data that allow useful configuration of these cluster features. Here are some example usage patterns that will unpack what addons provide:

To enable an addon (using "tiller" as an example):

"kubernetesConfig": {
    "addons": [
        {
            "name": "tiller",
            "enabled" : true
        }
    ]
}

As you can see above, addons is an array child property of kubernetesConfig. Each addon that you want to add custom configuration to would be represented as an object item in the array. For example, to disable the "azuredisk-csi-driver" addon:

"kubernetesConfig": {
    "addons": [
        {
            "name": "azuredisk-csi-driver",
            "enabled" : false
        }
    ]
}

More usefully, let's add some custom configuration to an addon. The example below customizes the "cluster-autoscaler" addon:

"kubernetesConfig": {
    "addons": [
        {
            "name": "cluster-autoscaler",
            "enabled": true,
            "containers": [
              {
                "name": "cluster-autoscaler",
                "image": "myDockerHubUser/cluster-autoscaler:v1.20.1-alpha",
                "cpuRequests": "500m",
                "memoryRequests": "800Mi",
                "cpuLimits": "500m",
                "memoryLimits": "800Mi"
              }
            ],
            "config": {
              "max-nodes": "5",
              "min-nodes": "1",
              "scan-interval": "10s"
            }
        }
    ]
}

The above example includes specific resource limit values across the following dimensions:

• cpuRequests
• memoryRequests
• cpuLimits
• memoryLimits

See https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ for more on Kubernetes resource limits.

Additionally above, we specified a custom docker image to use, let's say we want to build a cluster and test an alpha version of cluster-autoscaler in it. Important note! customizing the image is not sticky across upgrade/scale, to ensure that AKS Engine always delivers a version-curated, known-working addon when moving a cluster to a new version. Considering all that, providing a custom image reference for an addon configuration should be considered for testing/development, but not for a production cluster. If you'd like to entirely customize one of the addons available, including across scale/upgrade operations, you may include in an addon's spec a base64-encoded string of a Kubernetes yaml manifest. E.g.,

"kubernetesConfig": {
    "addons": [
        {
            "name": "kube-proxy",
            "enabled": true,
            "data": "IGFwaVZlcnNpb246IHYxCmtpbmQ6IFNlcnZpY2UKbWV0YWRhdGE6CiAgbmFtZToga3ViZS1kbnMKICBuYW1lc3BhY2U6IGt1YmUtc3lzdGVtCiAgbGFiZWxzOgogICAgazhzLWFwcDoga3ViZS1kbnMKICAgIGt1YmVybmV0ZXMuaW8vY2x1c3Rlci1zZXJ2aWNlOiAidHJ1ZSIKICAgIGFkZG9ubWFuYWdlci5rdWJlcm5ldGVzLmlvL21vZGU6IFJlY29uY2lsZQogICAga3ViZXJuZXRlcy5pby9uYW1lOiAiS3ViZUROUyIKc3BlYzoKICBzZWxlY3RvcjoKICAgIGs4cy1hcHA6IGt1YmUtZG5zCiAgY2x1c3RlcklQOiA8a3ViZUROU1NlcnZpY2VJUD4KICBwb3J0czoKICAtIG5hbWU6IGRucwogICAgcG9ydDogNTMKICAgIHByb3RvY29sOiBVRFAKICAtIG5hbWU6IGRucy10Y3AKICAgIHBvcnQ6IDUzCiAgICBwcm90b2NvbDogVENQCi0tLQphcGlWZXJzaW9uOiB2MQpraW5kOiBTZXJ2aWNlQWNjb3VudAptZXRhZGF0YToKICBuYW1lOiBrdWJlLWRucwogIG5hbWVzcGFjZToga3ViZS1zeXN0ZW0KICBsYWJlbHM6CiAgICBrdWJlcm5ldGVzLmlvL2NsdXN0ZXItc2VydmljZTogInRydWUiCiAgICBhZGRvbm1hbmFnZXIua3ViZXJuZXRlcy5pby9tb2RlOiBSZWNvbmNpbGUKLS0tCmFwaVZlcnNpb246IHYxCmtpbmQ6IENvbmZpZ01hcAptZXRhZGF0YToKICBuYW1lOiBrdWJlLWRucwogIG5hbWVzcGFjZToga3ViZS1zeXN0ZW0KICBsYWJlbHM6CiAgICBhZGRvbm1hbmFnZXIua3ViZXJuZXRlcy5pby9tb2RlOiBFbnN1cmVFeGlzdHMKLS0tCmFwaVZlcnNpb246IGV4dGVuc2lvbnMvdjFiZXRhMQpraW5kOiBEZXBsb3ltZW50Cm1ldGFkYXRhOgogIG5hbWU6IGt1YmUtZG5zCiAgbmFtZXNwYWNlOiBrdWJlLXN5c3RlbQogIGxhYmVsczoKICAgIGs4cy1hcHA6IGt1YmUtZG5zCiAgICBrdWJlcm5ldGVzLmlvL2NsdXN0ZXItc2VydmljZTogInRydWUiCiAgICBhZGRvbm1hbmFnZXIua3ViZXJuZXRlcy5pby9tb2RlOiBSZWNvbmNpbGUKc3BlYzoKICAjIHJlcGxpY2FzOiBub3Qgc3BlY2lmaWVkIGhlcmU6CiAgIyAxLiBJbiBvcmRlciB0byBtYWtlIEFkZG9uIE1hbmFnZXIgZG8gbm90IHJlY29uY2lsZSB0aGlzIHJlcGxpY2FzIHBhcmFtZXRlci4KICAjIDIuIERlZmF1bHQgaXMgMS4KICAjIDMuIFdpbGwgYmUgdHVuZWQgaW4gcmVhbCB0aW1lIGlmIEROUyBob3Jpem9udGFsIGF1dG8tc2NhbGluZyBpcyB0dXJuZWQgb24uCiAgc3RyYXRlZ3k6CiAgICByb2xsaW5nVXBkYXRlOgogICAgICBtYXhTdXJnZTogMTAlCiAgICAgIG1heFVuYXZhaWxhYmxlOiAwCiAgc2VsZWN0b3I6CiAgICBtYXRjaExhYmVsczoKICAgICAgazhzLWFwcDoga3ViZS1kbnMKICB0ZW1wbGF0ZToKICAgIG1ldGFkYXRhOgogICAgICBsYWJlbHM6CiAgICAgICAgazhzLWFwcDoga3ViZS1kbnMKICAgICAgYW5ub3RhdGlvbnM6CiAgICAgICAgc2NoZWR1bGVyLmFscGhhLmt1YmVybmV0ZXMuaW8vY3JpdGljYWwtcG9kOiAnJwogICAgICAgIHNlY2NvbXAuc2VjdXJpdHkuYWxwaGEua3ViZXJuZXRlcy5pby9wb2Q6ICdkb2NrZXIvZGVmYXVsdCcKICAgIHNwZWM6CiAgICAgIHByaW9yaXR5Q2xhc3NOYW1lOiBzeXN0ZW0tY2x1c3Rlci1jcml0aWNhbAogICAgICB0b2xlcmF0aW9uczoKICAgICAgLSBrZXk6ICJDcml0aWNhbEFkZG9uc09ubHkiCiAgICAgICAgb3BlcmF0b3I6ICJFeGlzdHMiCiAgICAgIHZvbHVtZXM6CiAgICAgIC0gbmFtZToga3ViZS1kbnMtY29uZmlnCiAgICAgICAgY29uZmlnTWFwOgogICAgICAgICAgbmFtZToga3ViZS1kbnMKICAgICAgICAgIG9wdGlvbmFsOiB0cnVlCiAgICAgIGNvbnRhaW5lcnM6CiAgICAgIC0gbmFtZToga3ViZWRucwogICAgICAgIGltYWdlOiA8a3ViZXJuZXRlc0t1YmVETlNTcGVjPgogICAgICAgIHJlc291cmNlczoKICAgICAgICAgICMgVE9ETzogU2V0IG1lbW9yeSBsaW1pdHMgd2hlbiB3ZSd2ZSBwcm9maWxlZCB0aGUgY29udGFpbmVyIGZvciBsYXJnZQogICAgICAgICAgIyBjbHVzdGVycywgdGhlbiBzZXQgcmVxdWVzdCA9IGxpbWl0IHRvIGtlZXAgdGhpcyBjb250YWluZXIgaW4KICAgICAgICAgICMgZ3VhcmFudGVlZCBjbGFzcy4gQ3VycmVudGx5LCB0aGlzIGNvbnRhaW5lciBmYWxscyBpbnRvIHRoZQogICAgICAgICAgIyAiYnVyc3RhYmxlIiBjYXRlZ29yeSBzbyB0aGUga3ViZWxldCBkb2Vzbid0IGJhY2tvZmYgZnJvbSByZXN0YXJ0aW5nIGl0LgogICAgICAgICAgbGltaXRzOgogICAgICAgICAgICBtZW1vcnk6IDE3ME1pCiAgICAgICAgICByZXF1ZXN0czoKICAgICAgICAgICAgY3B1OiAxMDBtCiAgICAgICAgICAgIG1lbW9yeTogNzBNaQogICAgICAgIGxpdmVuZXNzUHJvYmU6CiAgICAgICAgICBodHRwR2V0OgogICAgICAgICAgICBwYXRoOiAvaGVhbHRoY2hlY2sva3ViZWRucwogICAgICAgICAgICBwb3J0OiAxMDA1NAogICAgICAgICAgICBzY2hlbWU6IEhUVFAKICAgICAgICAgIGluaXRpYWxEZWxheVNlY29uZHM6IDYwCiAgICAgICAgICB0aW1lb3V0U2Vjb25kczogNQogICAgICAgICAgc3VjY2Vzc1RocmVzaG9sZDogMQogICAgICAgICAgZmFpbHVyZVRocmVzaG9sZDogNQogICAgICAgIHJlYWRpbmVzc1Byb2JlOgogICAgICAgICAgaHR0cEdldDoKICAgICAgICAgICAgcGF0aDogL3JlYWRpbmVzcwogICAgICAgICAgICBwb3J0OiA4MDgxCiAgICAgICAgICAgIHNjaGVtZTogSFRUUAogICAgICAgICAgIyB3ZSBwb2xsIG9uIHBvZCBzdGFydHVwIGZvciB0aGUgS3ViZXJuZXRlcyBtYXN0ZXIgc2VydmljZSBhbmQKICAgICAgICAgICMgb25seSBzZXR1cCB0aGUgL3JlYWRpbmVzcyBIVFRQIHNlcnZlciBvbmNlIHRoYXQncyBhdmFpbGFibGUuCiAgICAgICAgICBpbml0aWFsRGVsYXlTZWNvbmRzOiAzCiAgICAgICAgICB0aW1lb3V0U2Vjb25kczogNQogICAgICAgIGFyZ3M6CiAgICAgICAgLSAtLWRvbWFpbj08a3ViZXJuZXRlc0t1YmVsZXRDbHVzdGVyRG9tYWluPi4KICAgICAgICAtIC0tZG5zLXBvcnQ9MTAwNTMKICAgICAgICAtIC0tY29uZmlnLWRpcj0va3ViZS1kbnMtY29uZmlnCiAgICAgICAgLSAtLXY9MgogICAgICAgIGVudjoKICAgICAgICAtIG5hbWU6IFBST01FVEhFVVNfUE9SVAogICAgICAgICAgdmFsdWU6ICIxMDA1NSIKICAgICAgICBwb3J0czoKICAgICAgICAtIGNvbnRhaW5lclBvcnQ6IDEwMDUzCiAgICAgICAgICBuYW1lOiBkbnMtbG9jYWwKICAgICAgICAgIHByb3RvY29sOiBVRFAKICAgICAgICAtIGNvbnRhaW5lclBvcnQ6IDEwMDUzCiAgICAgICAgICBuYW1lOiBkbnMtdGNwLWxvY2FsCiAgICAgICAgICBwcm90b2NvbDogVENQCiAgICAgICAgLSBjb250YWluZXJQb3J0OiAxMDA1NQogICAgICAgICAgbmFtZTogbWV0cmljcwogICAgICAgICAgcHJvdG9jb2w6IFRDUAogICAgICAgIHZvbHVtZU1vdW50czoKICAgICAgICAtIG5hbWU6IGt1YmUtZG5zLWNvbmZpZwogICAgICAgICAgbW91bnRQYXRoOiAva3ViZS1kbnMtY29uZmlnCiAgICAgIC0gbmFtZTogZG5zbWFzcQogICAgICAgIGltYWdlOiA8a3ViZXJuZXRlc0ROU01hc3FTcGVjPgogICAgICAgIGxpdmVuZXNzUHJvYmU6CiAgICAgICAgICBodHRwR2V0OgogICAgICAgICAgICBwYXRoOiAvaGVhbHRoY2hlY2svZG5zbWFzcQogICAgICAgICAgICBwb3J0OiAxMDA1NAogICAgICAgICAgICBzY2hlbWU6IEhUVFAKICAgICAgICAgIGluaXRpYWxEZWxheVNlY29uZHM6IDYwCiAgICAgICAgICB0aW1lb3V0U2Vjb25kczogNQogICAgICAgICAgc3VjY2Vzc1RocmVzaG9sZDogMQogICAgICAgICAgZmFpbHVyZVRocmVzaG9sZDogNQogICAgICAgIGFyZ3M6CiAgICAgICAgLSAtdj0yCiAgICAgICAgLSAtbG9ndG9zdGRlcnIKICAgICAgICAtIC1jb25maWdEaXI9L2V0Yy9rOHMvZG5zL2Ruc21hc3EtbmFubnkKICAgICAgICAtIC1yZXN0YXJ0RG5zbWFzcT10cnVlCiAgICAgICAgLSAtLQogICAgICAgIC0gLWsKICAgICAgICAtIC0tY2FjaGUtc2l6ZT0xMDAwCiAgICAgICAgLSAtLW5vLW5lZ2NhY2hlCiAgICAgICAgLSAtLWxvZy1mYWNpbGl0eT0tCiAgICAgICAgLSAtLXNlcnZlcj0vY2x1c3Rlci5sb2NhbC8xMjcuMC4wLjEjMTAwNTMKICAgICAgICAtIC0tc2VydmVyPS9pbi1hZGRyLmFycGEvMTI3LjAuMC4xIzEwMDUzCiAgICAgICAgLSAtLXNlcnZlcj0vaXA2LmFycGEvMTI3LjAuMC4xIzEwMDUzCiAgICAgICAgcG9ydHM6CiAgICAgICAgLSBjb250YWluZXJQb3J0OiA1MwogICAgICAgICAgbmFtZTogZG5zCiAgICAgICAgICBwcm90b2NvbDogVURQCiAgICAgICAgLSBjb250YWluZXJQb3J0OiA1MwogICAgICAgICAgbmFtZTogZG5zLXRjcAogICAgICAgICAgcHJvdG9jb2w6IFRDUAogICAgICAgICMgc2VlOiBodHRwczovL2dpdGh1Yi5jb20va3ViZXJuZXRlcy9rdWJlcm5ldGVzL2lzc3Vlcy8yOTA1NSBmb3IgZGV0YWlscwogICAgICAgIHJlc291cmNlczoKICAgICAgICAgIHJlcXVlc3RzOgogICAgICAgICAgICBjcHU6IDE1MG0KICAgICAgICAgICAgbWVtb3J5OiAyME1pCiAgICAgICAgdm9sdW1lTW91bnRzOgogICAgICAgIC0gbmFtZToga3ViZS1kbnMtY29uZmlnCiAgICAgICAgICBtb3VudFBhdGg6IC9ldGMvazhzL2Rucy9kbnNtYXNxLW5hbm55CiAgICAgIC0gbmFtZTogc2lkZWNhcgogICAgICAgIGltYWdlOiA8a3ViZXJuZXRlc0ROU1NpZGVjYXJTcGVjPgogICAgICAgIGxpdmVuZXNzUHJvYmU6CiAgICAgICAgICBodHRwR2V0OgogICAgICAgICAgICBwYXRoOiAvbWV0cmljcwogICAgICAgICAgICBwb3J0OiAxMDA1NAogICAgICAgICAgICBzY2hlbWU6IEhUVFAKICAgICAgICAgIGluaXRpYWxEZWxheVNlY29uZHM6IDYwCiAgICAgICAgICB0aW1lb3V0U2Vjb25kczogNQogICAgICAgICAgc3VjY2Vzc1RocmVzaG9sZDogMQogICAgICAgICAgZmFpbHVyZVRocmVzaG9sZDogNQogICAgICAgIGFyZ3M6CiAgICAgICAgLSAtLXY9MgogICAgICAgIC0gLS1sb2d0b3N0ZGVycgogICAgICAgIC0gLS1wcm9iZT1rdWJlZG5zLDEyNy4wLjAuMToxMDA1MyxrdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLjxrdWJlcm5ldGVzS3ViZWxldENsdXN0ZXJEb21haW4+LDUsU1JWCiAgICAgICAgLSAtLXByb2JlPWRuc21hc3EsMTI3LjAuMC4xOjUzLGt1YmVybmV0ZXMuZGVmYXVsdC5zdmMuPGt1YmVybmV0ZXNLdWJlbGV0Q2x1c3RlckRvbWFpbj4sNSxTUlYKICAgICAgICBwb3J0czoKICAgICAgICAtIGNvbnRhaW5lclBvcnQ6IDEwMDU0CiAgICAgICAgICBuYW1lOiBtZXRyaWNzCiAgICAgICAgICBwcm90b2NvbDogVENQCiAgICAgICAgcmVzb3VyY2VzOgogICAgICAgICAgcmVxdWVzdHM6CiAgICAgICAgICAgIG1lbW9yeTogMjBNaQogICAgICAgICAgICBjcHU6IDEwbQogICAgICBkbnNQb2xpY3k6IERlZmF1bHQgICMgRG9uJ3QgdXNlIGNsdXN0ZXIgRE5TLgogICAgICBzZXJ2aWNlQWNjb3VudE5hbWU6IGt1YmUtZG5zCiAgICAgIG5vZGVTZWxlY3RvcjoKICAgICAgICBiZXRhLmt1YmVybmV0ZXMuaW8vb3M6IGxpbnV4"
        }
    ]
}

The reason for the unsightly base64-encoded input type is to optimize delivery payload, and to squash a human-maintainable yaml file representation into something that can be tightly pasted into a JSON string value without the arguably more unsightly carriage returns / whitespace that would be delivered with a literal copy/paste of a Kubernetes manifest.

coredns

The coredns addon includes integration with the cluster-proportional-autoscaler project to automatically scale out coredns pod replicas according to node, or core count. More information at the official docs here. The AKS Engine default configuration tunes the autoscaler thresholds to "32" nodes, and "512" cores (whichever threshold is crossed first engages scaling behaviors), with a minimum replica count of "1". The scale thresholds are higher than those seen in example docs due to observed (not catastrophic) increases in per-DNS resolution response times. In other words, for smaller clusters that aren't coredns pod-constrained, a single coredns pod is more responsive. These configurations are entirely user-configurable: you may tune them according to the operational DNS characteristics of your environment. E.g.:

"kubernetesConfig": {
    "addons": [
        ...
        {
          "name": "coredns",
          "enabled": true,
          "config": {
            "cores-per-replica": "512",
            "min-replicas": "3",
            "nodes-per-replica": "32"
          }
        },
        ...
    ]
}

calico

The calico addon includes configurable verbosity via the logSeverityScreen configuration property. The default is "info". To override that default, (e.g., to "error"), see this example:

...
"kubernetesConfig": {
    "addons": [
        ...
        {
          "name": "calico-daemonset",
          "enabled": true,
          "config": {
            "logSeverityScreen": "error"
          }
        }
        ...
    ]
}
...

Available options for logSeverityScreen are documented here.

nvidia-device-plugin

The nvidia-device-plugin addon installs a nvidia driver listener via a daemonset for nodes backed by the docker container runtime. The daemonset will install itself as a pod container on each node that has GPU support (i.e., nodes backed by N-series VMs). For example a pod running on a GPU-enabled node:

$ kubectl logs nvidia-device-plugin-w8vnt -n kube-system
2021/04/08 18:38:05 Loading NVML
2021/04/08 18:38:05 Starting FS watcher.
2021/04/08 18:38:05 Starting OS watcher.
2021/04/08 18:38:05 Retreiving plugins.
2021/04/08 18:38:05 Starting GRPC server for 'nvidia.com/gpu'
2021/04/08 18:38:05 Starting to serve 'nvidia.com/gpu' on /var/lib/kubelet/device-plugins/nvidia.sock
2021/04/08 18:38:05 Registered device plugin for 'nvidia.com/gpu' with Kubelet

This addon will be enabled automatically if the cluster is configured for docker, and has at least one N-series VM-configured node pool.

If you're using one or more N-series node pools and using containerd, then this addon will not be installed, as containerd-backed clusters require the user to bring his or her own GPU drivers solution. The AKS Engine project recommends the official nvidia gpu-operator solution for containerd GPU solutions. See here:

• https://developer.nvidia.com/blog/announcing-containerd-support-for-the-nvidia-gpu-operator/

The gpu-operator solution is regularly tested against N-series node pool clusters backed by containerd.

If you're using docker and would like to provide your own GPU drivers solution, you may disable the nvidia-device-plugin addon manually:

...
"kubernetesConfig": {
    "addons": [
        ...
        {
          "name": "nvidia-device-plugin",
          "enabled": false
        }
        ...
    ]
}
...

Available options for logSeverityScreen are documented here.

components

components is an interface to allow for user-configurable core Kubernetes component implementations. Normally, you won't need to modify this configuration, as AKS Engine will use the best, known-working component implementations validated against Azure for all supported versions of Kubernetes. To support the rapid development of Azure + Kubernetes (e.g., Azure cloudprovider), this configuration vector may be useful for validating a custom build or configuration of the various core components on a running Azure Kubernetes cluster. Again, as with addons, this configurable vector is designed for cluster creation only. Using aks-engine-azurestack upgrade on a cluster will override the original, user-configured settings during the upgrade operation, rendering an upgraded cluster with the AKS Engine defaults for kube-controller-manager, cloud-controller-manager, kube-apiserver, kube-scheduler, and kube-addon-manager.

components is a child property of kubernetesConfig. Below is a list of components you may provide.:

Name of addon	Enabled by default?	How many pods	Description
kube-controller-manager	true	1 per master node	The Kubernetes controller manager is a daemon that embeds the core control loops shipped with Kubernetes. Official docs here.
cloud-controller-manager	false	1 per master node	The Cloud controller manager is a daemon that embeds the cloud specific control loops shipped with Kubernetes. Official docs here.
kube-apiserver	true	1 per master node	The Kubernetes API server validates and configures data for the api objects which include pods, services, replicationcontrollers, and others. The API Server services REST operations and provides the frontend to the cluster's shared state through which all other components interact. Official docs here.
kube-scheduler	true	1 per master node	The Kubernetes scheduler is a policy-rich, topology-aware, workload-specific function that significantly impacts availability, performance, and capacity. The scheduler needs to take into account individual and collective resource requirements, quality of service requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference, deadlines, and so on. Official docs here.
kube-addon-manager	true	1 per master node	Addon manager provides a standard way to deliver additional Kubernetes componentry. The addons supported by AKS Engine (documented above) are installed into the cluster using Addon manager. Official docs here.
cluster-init	false	n/a	This is an interface to deliver Kubernetes resource configuration that may be tightly coupled to an ARM deployment. For example, if you use AKS Engine in an automated pipeline and you can scale up a cluster quickly by loading a Kubernetes specification immediately after bootstrapping the control plane, give the base64-encoded YAML representation of that spec to cluster-init. That data is decoded at cluster creation time to the path /opt/azure/containers/cluster-init.yaml on the first master VM, and then loaded into the cluster via kubectl apply -f. Seekubectl apply for more documentation on how to specify the source YAML. Note: the cluster-init spec is ignored during aks-engine-azurestack upgrade and aks-engine-azurestack scaleoperations.

To give a bit more info on the components property: Currently, there are two configuration vectors available for use. Firstly, you have the option to pass in a custom image reference for the container that the AKS Engine-provided specs implement (all components are implemented as single-container Pod resources); in addition you may provide an alternate command string to execute inside the container. E.g.:

"kubernetesConfig": {
    "components": [
        {
            "name": "kube-controller-manager",
            "enabled" : true,
            "containers": [
                {
                    "name": "kube-controller-manager",
                    "image": "myDockerHubUser/custom-controller-manager-build:v1.17.2-dirty",
                }
            ],
            "config": {
                "command": "kube-controller-manager-custom"
            }
        }
    ]
}

In addition, a component may be disabled by setting "enabled": false. This only makes practical sense for cloud-controller-manager, which is an optional Kubernetes control plane implementation that runs the Azure-specific control plane runtime in a separate pod from kube-controller-manager. For all other components, if they are disabled, then AKS Engine will not by itself create a functional Kubernetes cluster. This configuration is made available for development purposes only, and is not recommended for users building functional Kubernetes clusters using AKS Engine.

Note: kube-addon-manager does not support a command string, as that configuration isn't appropriate for Addon Manager, which only defines a container image as configurable input.

For each Kubernetes component (with the exception of kube-addon-manager), the args to the command are provided via the equivalent "<component>Config" property. E.g.:

• kube-controller-manager
  • Command arguments map to controllerManagerConfig key=val pairs
• cloud-controller-manager
  • Command arguments map to cloudControllerManagerConfig key=val pairs
• kube-apiserver
  • Command arguments map to apiServerConfig key=val pairs
• kube-scheduler
  • Command arguments map to schedulerConfig key=val pairs

See https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ for more on Kubernetes resource limits.

As with addons, you may include an entirely custom component spec as a base64-encoded string of a Kubernetes yaml manifest. E.g.,

"kubernetesConfig": {
    "components": [
        {
            "name": "kube-controller-manager",
            "enabled": true,
            "data": "YXBpVmVyc2lvbjogdjEKa2luZDogUG9kCm1ldGFkYXRhOgogIG5hbWU6IGt1YmUtY29udHJvbGxl
ci1tYW5hZ2VyCiAgbmFtZXNwYWNlOiBrdWJlLXN5c3RlbQogIGxhYmVsczoKICAgIHRpZXI6IGNv
bnRyb2wtcGxhbmUKICAgIGNvbXBvbmVudDoga3ViZS1jb250cm9sbGVyLW1hbmFnZXIKc3BlYzoK
ICBwcmlvcml0eUNsYXNzTmFtZTogc3lzdGVtLW5vZGUtY3JpdGljYWwKICBob3N0TmV0d29yazog
dHJ1ZQogIGNvbnRhaW5lcnM6CiAgICAtIG5hbWU6IGt1YmUtY29udHJvbGxlci1tYW5hZ2VyCiAg
ICAgIGltYWdlOiBrOHMuZ2NyLmlvL2h5cGVya3ViZS1hbWQ2NDp2MS4xNi42CiAgICAgIGltYWdl
UHVsbFBvbGljeTogSWZOb3RQcmVzZW50CiAgICAgIGNvbW1hbmQ6IFsiL2h5cGVya3ViZSIsICJr
dWJlLWNvbnRyb2xsZXItbWFuYWdlciJdCiAgICAgIGFyZ3M6IFsiLS1hbGxvY2F0ZS1ub2RlLWNp
ZHJzPWZhbHNlIiwgIi0tY2xvdWQtY29uZmlnPS9ldGMva3ViZXJuZXRlcy9henVyZS5qc29uIiwg
Ii0tY2xvdWQtcHJvdmlkZXI9YXp1cmUiLCAiLS1jbHVzdGVyLWNpZHI9MTAuMjQwLjAuMC8xMiIs
ICItLWNsdXN0ZXItbmFtZT1rdWJlcm5ldGVzLXdlc3R1czItMzMyODEiLCAiLS1jbHVzdGVyLXNp
Z25pbmctY2VydC1maWxlPS9ldGMva3ViZXJuZXRlcy9jZXJ0cy9jYS5jcnQiLCAiLS1jbHVzdGVy
LXNpZ25pbmcta2V5LWZpbGU9L2V0Yy9rdWJlcm5ldGVzL2NlcnRzL2NhLmtleSIsICItLWNvbmZp
Z3VyZS1jbG91ZC1yb3V0ZXM9ZmFsc2UiLCAiLS1jb250cm9sbGVycz0qLGJvb3RzdHJhcHNpZ25l
cix0b2tlbmNsZWFuZXIiLCAiLS1mZWF0dXJlLWdhdGVzPUxvY2FsU3RvcmFnZUNhcGFjaXR5SXNv
bGF0aW9uPXRydWUsU2VydmljZU5vZGVFeGNsdXNpb249dHJ1ZSIsICItLWhvcml6b250YWwtcG9k
LWF1dG9zY2FsZXItY3B1LWluaXRpYWxpemF0aW9uLXBlcmlvZD0zMHMiLCAiLS1ob3Jpem9udGFs
LXBvZC1hdXRvc2NhbGVyLWRvd25zY2FsZS1zdGFiaWxpemF0aW9uPTMwcyIsICItLWt1YmVjb25m
aWc9L3Zhci9saWIva3ViZWxldC9rdWJlY29uZmlnIiwgIi0tbGVhZGVyLWVsZWN0PXRydWUiLCAi
LS1ub2RlLW1vbml0b3ItZ3JhY2UtcGVyaW9kPTQwcyIsICItLXBvZC1ldmljdGlvbi10aW1lb3V0
PTVtMHMiLCAiLS1wcm9maWxpbmc9ZmFsc2UiLCAiLS1yb290LWNhLWZpbGU9L2V0Yy9rdWJlcm5l
dGVzL2NlcnRzL2NhLmNydCIsICItLXJvdXRlLXJlY29uY2lsaWF0aW9uLXBlcmlvZD0xMHMiLCAi
LS1zZXJ2aWNlLWFjY291bnQtcHJpdmF0ZS1rZXktZmlsZT0vZXRjL2t1YmVybmV0ZXMvY2VydHMv
YXBpc2VydmVyLmtleSIsICItLXRlcm1pbmF0ZWQtcG9kLWdjLXRocmVzaG9sZD01MDAwIiwgIi0t
dXNlLXNlcnZpY2UtYWNjb3VudC1jcmVkZW50aWFscz10cnVlIiwgIi0tdj0yIl0KICAgICAgdm9s
dW1lTW91bnRzOgogICAgICAgIC0gbmFtZTogZXRjLWt1YmVybmV0ZXMKICAgICAgICAgIG1vdW50
UGF0aDogL2V0Yy9rdWJlcm5ldGVzCiAgICAgICAgLSBuYW1lOiB2YXItbGliLWt1YmVsZXQKICAg
ICAgICAgIG1vdW50UGF0aDogL3Zhci9saWIva3ViZWxldAogICAgICAgIC0gbmFtZTogbXNpCiAg
ICAgICAgICBtb3VudFBhdGg6IC92YXIvbGliL3dhYWdlbnQvTWFuYWdlZElkZW50aXR5LVNldHRp
bmdzCiAgICAgICAgICByZWFkT25seTogdHJ1ZQogIHZvbHVtZXM6CiAgICAtIG5hbWU6IGV0Yy1r
dWJlcm5ldGVzCiAgICAgIGhvc3RQYXRoOgogICAgICAgIHBhdGg6IC9ldGMva3ViZXJuZXRlcwog
ICAgLSBuYW1lOiB2YXItbGliLWt1YmVsZXQKICAgICAgaG9zdFBhdGg6CiAgICAgICAgcGF0aDog
L3Zhci9saWIva3ViZWxldAogICAgLSBuYW1lOiBtc2kKICAgICAgaG9zdFBhdGg6CiAgICAgICAg
cGF0aDogL3Zhci9saWIvd2FhZ2VudC9NYW5hZ2VkSWRlbnRpdHktU2V0dGluZ3MK"
        }
    ]
}

The above is the pattern we use to pass in a cluster-init spec for loading at cluster bootstrap time. E.g.:

"kubernetesConfig": {
    "components": [
        {
            "name": "cluster-init",
            "enabled": true,
            "data": "YXBpVmVyc2lvbjogdjEKa2luZDogUG9kCm1ldGFkYXRhOgogIG5hbWU6IGFrcy1lbmdpbmUtcG9kLWluaXQKc3BlYzoKICBjb250YWluZXJzOgogIC0gbmFtZTogYWtzLWVuZ2luZS1wb2QtaW5pdAogICAgaW1hZ2U6IGJ1c3lib3g6MS4zMS4xCiAgICBhcmdzOgogICAgLSAvYmluL3NoCiAgICAtIC1jCiAgICAtIHdoaWxlIHRydWU7IGRvIHNsZWVwIDYwMDsgZG9uZQogIG5vZGVTZWxlY3RvcjoKICAgIGJldGEua3ViZXJuZXRlcy5pby9vczogbGludXgKLS0tCmFwaVZlcnNpb246IGJhdGNoL3YxCmtpbmQ6IEpvYgptZXRhZGF0YToKICBuYW1lOiBha3MtZW5naW5lLWpvYi1pbml0CnNwZWM6CiAgdGVtcGxhdGU6CiAgICBzcGVjOgogICAgICBjb250YWluZXJzOgogICAgICAtIGltYWdlOiBidXN5Ym94OjEuMzEuMQogICAgICAgIG5hbWU6IGJ1c3lib3gtYWdlbnQKICAgICAgICBjb21tYW5kOiBbJ3NoJywgJy1jJywgJ1sgJChlY2hvICJIZWxsbywgV29ybGQhIiB8IHNoYTI1NnN1bSB8IGN1dCAtZCIgIiAtZjEpID0gImM5OGMyNGI2NzdlZmY0NDg2MGFmZWE2ZjQ5M2JiYWVjNWJiMWM0Y2JiMjA5YzZmYzJiYmI0N2Y2NmZmMmFkMzEiIF0nXQogICAgICByZXN0YXJ0UG9saWN5OiBOZXZlcgogICAgICBub2RlU2VsZWN0b3I6CiAgICAgICAgYmV0YS5rdWJlcm5ldGVzLmlvL29zOiBsaW51eAogIGJhY2tvZmZMaW1pdDogMAo="
        }
    ]
}

kubeletConfig

kubeletConfig declares runtime configuration for the kubelet running on all master and agent nodes. It is a generic key/value object, and a child property of kubernetesConfig. The kubeletConfig configuration under kubernetesConfig will be inherited by a similar kubeletConfig configuration under the masterProfile configuration object, and by each agentPoolProfile in the agentPoolProfiles array. Specific master and per-pool kubelet configurations should be applied there. An example custom kubelet config:

"kubernetesConfig": {
    "kubeletConfig": {
        "--eviction-hard": "memory.available<250Mi,nodefs.available<20%,nodefs.inodesFree<10%"
    }
}

See here for a reference of supported kubelet options.

Below is a list of kubelet options that AKS Engine will configure by default:

kubelet option	default value
"--cadvisor-port"	"0"
"--cloud-config"	"/etc/kubernetes/azure.json" Removed in Kubernetes 1.34+ (use external cloud controllers)
"--cloud-provider"	"azure" for K8s < 1.34, "external" for K8s >= 1.34 In-tree azure provider removed in Kubernetes 1.34+ (must use external cloud controllers)
"--cluster-domain"	"cluster.local"
"--event-qps"	"0"
"--pod-infra-container-image"	"pause-amd64:version" Removed in Kubernetes 1.35+ (pause container configured via runtime/config file)
"--network-plugin"	"cni"
"--max-pods"	"30", or "110" if using kubenet --network-plugin (i.e., "networkPlugin": "kubenet")
"--eviction-hard"	"memory.available<750Mi,nodefs.available<10%,nodefs.inodesFree<5%"
"--node-status-update-frequency"	"10s"
"--image-gc-high-threshold"	"85"
"--image-gc-low-threshold"	"80"
"--non-masquerade-cidr"	"0.0.0.0/0" (unless ip-masq-agent is disabled, in which case this is set to the value of kubernetesConfig.ClusterSubnet ) - Removed in v1.24
"--pod-max-pids"	"-1" (need to activate the feature in --feature-gates=SupportPodPidsLimit=true)
"--image-pull-progress-deadline"	"30m"
"--feature-gates"	No default (can be a comma-separated list). On agent nodes Accelerators=true will be applied in the --feature-gates option for k8s versions before 1.11.0
"--enforce-node-allocatable"	"pods"
"--streaming-connection-idle-timeout"	"4h"
"--rotate-certificates"	"true" (this default is set for clusters >= 1.11.9 )
"--tls-cipher-suites"	"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256"
"--authentication-token-webhook"	"true" (this default is set for clusters >= 1.16.0 )
"--read-only-port"	"0" (this default is set for clusters >= 1.16.0 )
"--register-with-taints"	"node-role.kubernetes.io/control-plane=true:NoSchedule" (masterProfile only; Note: you may add your own master-specific taints in the kubeletConfig under masterProfile, which will augment the built-in "node-role.kubernetes.io/control-plane=true:NoSchedule" taint, which will always be present.)
"--runtime-request-timeout"	"15m" if in a containerd configuration, otherwise this configuration is not passed to kubelet runtime
"--container-runtime-endpoint"	"unix:///run/containerd/containerd.sock" if in a containerd configuration, otherwise this configuration is not passed to kubelet runtime

Below is a list of kubelet options that are not currently user-configurable, either because a higher order configuration vector is available that enforces kubelet configuration, or because a static configuration is required to build a functional cluster:

kubelet option	default value
"--address"	"0.0.0.0"
"--allow-privileged"	"true"
"--anonymous-auth"	"false"
"--authorization-mode"	"webhook"
"--client-ca-file"	"/etc/kubernetes/certs/ca.crt"
"--cluster-dns"	"10.0.0.10" (if ipv4, or "fd00::10" if using ipv6)
"--pod-manifest-path"	"/etc/kubernetes/manifests"
"--node-labels"	(based on Azure node metadata)
"--cgroups-per-qos"	"true"
"--kubeconfig"	"/var/lib/kubelet/kubeconfig"
"--tls-cert-file"	"/etc/kubernetes/certs/kubeletserver.crt"
"--tls-private-key-file"	"/etc/kubernetes/certs/kubeletserver.key"
"--v"	"2"
"--volume-plugin-dir"	"/etc/kubernetes/volumeplugins"
"--image-credential-provider-config"	"/var/lib/kubelet/credential-provider-config.yaml"
"--image-credential-provider-bin-dir"	"/var/lib/kubelet/credential-provider"

controllerManagerConfig

controllerManagerConfig declares runtime configuration for the kube-controller-manager daemon running on all master nodes. Like kubeletConfig it is a generic key/value object, and a child property of kubernetesConfig. An example custom controller-manager config:

"kubernetesConfig": {
    "controllerManagerConfig": {
        "--node-monitor-grace-period": "40s",
        "--pod-eviction-timeout": "5m0s",
        "--route-reconciliation-period": "10s"
        "--terminated-pod-gc-threshold": "5000"
    }
}

See here for a reference of supported controller-manager options.

Below is a list of controller-manager options that AKS Engine will configure by default:

controller-manager option	default value
"--node-monitor-grace-period"	"40s"
"--pod-eviction-timeout"	"5m0s"
"--route-reconciliation-period"	"10s"
"--terminated-pod-gc-threshold"	"5000"
"--feature-gates"	No default (can be a comma-separated list)

Below is a list of controller-manager options that are not currently user-configurable, either because a higher order configuration vector is available that enforces controller-manager configuration, or because a static configuration is required to build a functional cluster:

controller-manager option	default value
"--kubeconfig"	"/var/lib/kubelet/kubeconfig"
"--allocate-node-cidrs"	"false"
"--cluster-cidr"	uses clusterSubnet value
"--cluster-name"	auto-generated using API model properties
"--root-ca-file"	"/etc/kubernetes/certs/ca.crt"
"--cluster-signing-cert-file"	"/etc/kubernetes/certs/ca.crt"
"--cluster-signing-key-file"	"/etc/kubernetes/certs/ca.key"
"--service-account-private-key-file"	"/etc/kubernetes/certs/apiserver.key"
"--leader-elect"	"true"
"--v"	"2"
"--profiling"	"false"
"--use-service-account-credentials"	"false" ("true" if kubernetesConfig.enableRbac is true)

cloudControllerManagerConfig

cloudControllerManagerConfig declares runtime configuration for the cloud-controller-manager daemon running on all master nodes in a Cloud Controller Manager configuration. Like kubeletConfig it is a generic key/value object, and a child property of kubernetesConfig. An example custom cloud-controller-manager config:

"kubernetesConfig": {
    "cloudControllerManagerConfig": {
        "--route-reconciliation-period": "1m"
    }
}

See here for a reference of supported controller-manager options.

Below is a list of cloud-controller-manager options that AKS Engine will configure by default:

controller-manager option	default value
"--route-reconciliation-period"	"10s"

Below is a list of cloud-controller-manager options that are not currently user-configurable, either because a higher order configuration vector is available that enforces controller-manager configuration, or because a static configuration is required to build a functional cluster:

controller-manager option	default value
"--kubeconfig"	"/var/lib/kubelet/kubeconfig"
"--allocate-node-cidrs"	"false"
"--cluster-cidr"	uses clusterSubnet value
"--cluster-name"	auto-generated using API model properties
"--cloud-provider"	"azure" for K8s < 1.34, "external" for K8s >= 1.34 Removed in Kubernetes 1.34+
"--cloud-config"	"/etc/kubernetes/azure.json" Removed in Kubernetes 1.34+
"--leader-elect"	"true"
"--v"	"2"

apiServerConfig

apiServerConfig declares runtime configuration for the kube-apiserver daemon running on all master nodes. Like kubeletConfig and controllerManagerConfig it is a generic key/value object, and a child property of kubernetesConfig. An example custom apiserver config:

"kubernetesConfig": {
    "apiServerConfig": {
        "--request-timeout": "30s"
    }
}

Or perhaps you want to customize/override the set of admission-control flags passed to the API Server by default, you can omit the options you don't want and specify only the ones you need as follows:

"orchestratorProfile": {
      "orchestratorRelease": "1.8",
      "kubernetesConfig": {
        "apiServerConfig": {
          "--admission-control":  "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,AlwaysPullImages"
        }
      }
    }

See here for a reference of supported apiserver options.

Below is a list of apiserver options that AKS Engine will configure by default:

apiserver option	default value
"--admission-control-config-file"	"/etc/kubernetes/apiserver-admission-control.yaml" (default content overridable through CustomFiles)
"--anonymous-auth"	"false"
"--enable-admission-plugins"	"ExtendedResourceToleration" ("PodSecurityPolicy" if the "pod-security-policy" addon is enabled)
"--authorization-mode"	"Node", "RBAC" (the latter if enabledRbac is true)
"--audit-log-maxage"	"30"
"--audit-log-maxbackup"	"10"
"--audit-log-maxsize"	"100"
"--feature-gates"	No default (can be a comma-separated list)
"--oidc-username-claim"	"oid" (if has AADProfile)
"--oidc-groups-claim"	"groups" (if has AADProfile)
"--oidc-client-id"	calculated value that represents OID client ID (if has AADProfile)
"--oidc-issuer-url"	calculated value that represents OID issuer URL (if has AADProfile)
"--service-account-issuer"	"https://kubernetes.default.svc.cluster.local" (Kubernetes v1.20.0 and greater only)
"--service-account-signing-key-file"	"/etc/kubernetes/certs/apiserver.key" (Kubernetes v1.20.0 and greater only)
"--v"	"2"

Below is a list of apiserver options that are not currently user-configurable, either because a higher order configuration vector is available that enforces apiserver configuration, or because a static configuration is required to build a functional cluster:

apiserver option	default value
"--bind-address"	"0.0.0.0"
"--advertise-address"	calculated value that represents listening URI for API server
"--allow-privileged"	"true"
"--audit-log-path"	"/var/log/apiserver/audit.log"
"--insecure-port"	"0"
"--secure-port"	"443"
"--service-account-lookup"	"true"
"--etcd-cafile"	"/etc/kubernetes/certs/ca.crt"
"--etcd-certfile"	"/etc/kubernetes/certs/etcdclient.crt"
"--etcd-keyfile"	"/etc/kubernetes/certs/etcdclient.key"
"--etcd-servers"	calculated value that represents etcd servers
"--profiling"	"false"
"--repair-malformed-updates"	"false" (deprecated in v1.14)
"--tls-cert-file"	"/etc/kubernetes/certs/apiserver.crt"
"--tls-private-key-file"	"/etc/kubernetes/certs/apiserver.key"
"--client-ca-file"	"/etc/kubernetes/certs/ca.crt"
"--service-account-key-file"	"/etc/kubernetes/certs/apiserver.key"
"--kubelet-client-certificate"	"/etc/kubernetes/certs/client.crt"
"--kubelet-client-key"	"/etc/kubernetes/certs/client.key"
"--service-cluster-ip-range"	see serviceCIDR
"--storage-backend"	calculated value that represents etcd version
"--encryption-provider-config"	"/etc/kubernetes/encryption-config.yaml" (if enableDataEncryptionAtRest is true)
"--encryption-provider-config"	"/etc/kubernetes/encryption-config.yaml" (if enableEncryptionWithExternalKms is true)
"--requestheader-client-ca-file"	"/etc/kubernetes/certs/proxy-ca.crt" (if enableAggregatedAPIs is true)
"--proxy-client-cert-file"	"/etc/kubernetes/certs/proxy.crt" (if enableAggregatedAPIs is true)
"--proxy-client-key-file"	"/etc/kubernetes/certs/proxy.key" (if enableAggregatedAPIs is true)
"--requestheader-allowed-names"	"" (if enableAggregatedAPIs is true)
"--requestheader-extra-headers-prefix"	"X-Remote-Extra-" (if enableAggregatedAPIs is true)
"--requestheader-group-headers"	"X-Remote-Group" (if enableAggregatedAPIs is true)
"--requestheader-username-headers"	"X-Remote-User" (if enableAggregatedAPIs is true)
"--cloud-provider"	"azure" (unless useCloudControllerManager is true) Removed in Kubernetes 1.34+
"--cloud-config"	"/etc/kubernetes/azure.json" (unless useCloudControllerManager is true) Removed in Kubernetes 1.34+

schedulerConfig

schedulerConfig declares runtime configuration for the kube-scheduler daemon running on all master nodes. Like kubeletConfig, controllerManagerConfig, and apiServerConfig it is a generic key/value object, and a child property of kubernetesConfig. An example custom apiserver config:

"kubernetesConfig": {
    "schedulerConfig": {
        "--v": "2"
    }
}

See here for a reference of supported kube-scheduler options.

Below is a list of scheduler options that AKS Engine will configure by default:

kube-scheduler option	default value
"--v"	"2"
"--feature-gates"	No default (can be a comma-separated list)

Below is a list of kube-scheduler options that are not currently user-configurable, either because a higher order configuration vector is available that enforces kube-scheduler configuration, or because a static configuration is required to build a functional cluster:

kube-scheduler option	default value
"--kubeconfig"	"/var/lib/kubelet/kubeconfig"
"--leader-elect"	"true"
"--profiling"	"false"

We consider kubeletConfig, controllerManagerConfig, apiServerConfig, and schedulerConfig to be generic conveniences that add power/flexibility to cluster deployments. Their usage comes with no operational guarantees! They are manual tuning features that enable low-level configuration of a kubernetes cluster.

Custom YAML for Kubernetes component manifests

Custom YAML specifications can be configured for kube-scheduler, kube-controller-manager, cloud-controller-manager and kube-apiserver in addition to the addons described above. You will need to pass in a base64-encoded string of the kubernetes manifest YAML file to KubernetesComponentConfig["data"] . For example, to pass a custom kube-scheduler config, do the following:

"kubernetesConfig": {
    "schedulerConfig": {
            "data" : "<base64-encoded string of your k8s manifest YAML>"
        }
}

NOTE: Custom YAML for addons is an experimental feature. Since Addons.Data allows you to provide your own scripts, you are responsible for any undesirable consequences of their errors or failures. Use at your own risk.

sysctldConfig

The sysctldConfig configuration interface allows generic Linux kernel runtime configuration that will be delivered to sysctl. Use at your own risk! It is a generic key/value object, and a child property of both masterProfile and node pool configurations under agentPoolProfiles, for tuning the Linux kernel parameters on master and/or node pool VMs, respectively. An example custom sysctl config that tunes the control plane VMs:

"masterProfile": {
    ...
    "sysctldConfig": {
        "net.ipv4.tcp_keepalive_time": "120",
        "net.ipv4.tcp_keepalive_intvl": "75",
        "net.ipv4.tcp_keepalive_probes": "9"
    }
    ...
}

And, here's that same config override example on a node pool named "my-custom-node-pool":

"agentPoolProfiles": [
  {
    "name": "my-custom-node-pool",
    ...
    "sysctldConfig": {
        "net.ipv4.tcp_keepalive_time": "120",
        "net.ipv4.tcp_keepalive_intvl": "75",
        "net.ipv4.tcp_keepalive_probes": "9"
    }
    ...
  }
]

Kubernetes kernel configuration varies by distro, so please validate that the kernel parameter and value works for the Linux flavor you are using in your cluster.

Below is a list of sysctl configuration that AKS Engine will configure by default for both Ubuntu 16.04-LTS and 18.04-LTS, for both master and node pool VMs:

kernel parameter	default value
"net.ipv4.tcp_retries2"	"8"
"net.core.somaxconn"	"16384"
"net.ipv4.tcp_max_syn_backlog"	"16384"
"net.core.message_cost"	"40"
"net.core.message_burst"	"80"
"net.ipv4.neigh.default.gc_thresh1"	"4096"
"net.ipv4.neigh.default.gc_thresh2"	"8192"
"net.ipv4.neigh.default.gc_thresh3"	"16384"

privateCluster

privateCluster defines a cluster without public addresses assigned. It is a child property of kubernetesConfig.

Name	Required	Description
enabled	no	Enable Private Cluster (boolean - default == false)
jumpboxProfile	no	Configure and auto-provision a jumpbox to access your private cluster. jumpboxProfile is ignored if enabled is false. See jumpboxProfile below
enableHostsConfigAgent	no	Enable config agent to config /etc/hosts file for AKS private clusters

jumpboxProfile

jumpboxProfile describes the settings for a jumpbox deployed via aks-engine-azurestack to access a private cluster. It is a child property of privateCluster.

Name	Required	Description
name	yes	This is the unique name for the jumpbox VM. Some resources deployed with the jumpbox are derived from this name
vmSize	yes	Describes a valid Azure VM Sizes
publicKey	yes	The public SSH key used for authenticating access to the jumpbox. Here are instructions for [generating a public/private key pair][ssh]
osDiskSizeGB	no	Describes the OS Disk Size in GB. Defaults to 30
storageProfile	no	Specifies the storage profile to use. Valid values are ManagedDisks or StorageAccount. Defaults to ManagedDisks
username	no	Describes the admin username to be used on the jumpbox. Defaults to azureuser

masterProfile

masterProfile describes the settings for master configuration.

Name	Required	Description
count	yes	Masters have count value of 1, 3, or 5 masters
dnsPrefix	yes	The dns prefix for the master FQDN. The master FQDN is used for SSH or commandline access. This must be a unique name. (bring your own VNET examples)
subjectAltNames	no	An array of fully qualified domain names using which a user can reach API server. These domains are added as Subject Alternative Names to the generated API server certificate. NOTE: These domains will not be automatically provisioned.
firstConsecutiveStaticIP	only required when vnetSubnetId specified and when MasterProfile is not VirtualMachineScaleSets	The IP address of the first master. IP Addresses will be assigned consecutively to additional master nodes. When MasterProfile is using VirtualMachineScaleSets, this value will be determined by an offset from the first IP in the vnetCidr. For example, if vnetCidr is 10.239.0.0/16, then firstConsecutiveStaticIP will be 10.239.0.4
vmsize	yes	Describes a valid Azure VM Sizes. These are restricted to machines with at least 2 cores and 100GB of disk space
storageProfile	no	Specifies the storage profile to use. Valid values are ManagedDisks or StorageAccount. Defaults to ManagedDisks
osDiskSizeGB	no	Describes the OS Disk Size in GB
osDiskCachingType	no	Which disk caching type configuration to use for the OS disk on master VMs. Allowed configurations are "ReadWrite", "ReadOnly", or "None". Default is "ReadWrite".
vnetSubnetId	only required when using custom VNET	Specifies the Id of an alternate VNET subnet. The subnet id must specify a valid VNET ID owned by the same subscription. (bring your own VNET examples). When MasterProfile is set to VirtualMachineScaleSets, this value should be the subnetId of the master subnet. When MasterProfile is set to AvailabilitySet, this value should be the subnetId shared by both master and agent nodes.
extensions	no	This is an array of extensions. This indicates that the extension be run on a single master. The name in the extensions array must exactly match the extension name in the extensionProfiles
vnetCidr	no	Specifies the VNET cidr when using a custom VNET (bring your own VNET examples). This VNET cidr should include both the master and the agent subnets.
imageReference.name	no	The name of the Linux OS image. Needs to be used in conjunction with resourceGroup, below. (For information on setting this for Windows nodes see WindowsProfile)
imageReference.resourceGroup	no	Resource group that contains the Linux OS image. Needs to be used in conjunction with name, above
imageReference.subscriptionId	no	ID of subscription containing the Linux OS image. Applies only to Shared Image Galleries. All of name, resourceGroup, subscription, gallery, image name, and version must be specified for this scenario.
imageReference.gallery	no	Name of Shared Image Gallery containing the Linux OS image. Applies only to Shared Image Galleries. All of name, resourceGroup, subscription, gallery, image name, and version must be specified for this scenario.
imageReference.version	no	Version containing the Linux OS image. Applies only to Shared Image Galleries. All of name, resourceGroup, subscription, gallery, image name, and version must be specified for this scenario.
distro	no	Specifies the masters' Linux distribution. Currently supported values are: ubuntu-18.04, ubuntu-18.04-gen2 (Ubuntu 18.04-LTS running on a Generation 2 VM), and aks-ubuntu-18.04. For Azure Public Cloud, Azure US Government Cloud, and Azure China Cloud, defaults to aks-ubuntu-18.04. For other Sovereign Clouds, the default is ubuntu-18.04. aks-ubuntu-18.04 is a custom image based on ubuntu-18.04 that comes with pre-installed software necessary for Kubernetes deployments. Note: the ubuntu and aks-ubuntu-16.04 distro values may be used if you have a reason to use the EOL (End of Life) Ubuntu 16.04-LTS OS; we do not recommend using Ubuntu 16.04-LTS, as the OS will no longer be receiving security and other critical patches as of April 30, 2021.
customFiles	no	The custom files to be provisioned to the master nodes. Defined as an array of JSON objects with each defined as "source":"absolute-local-path", "dest":"absolute-path-on-masternodes".See examples
availabilityProfile	no	Supported values are AvailabilitySet (default) and VirtualMachineScaleSets (not recommended: you may not upgrade a cluster built with a VMSS control plane). If you do use a VirtualMachineScaleSets configuration for your control plane: all node pools require a "availabilityProfile": "VirtualMachineScaleSets" configuration; and, to SSH into a control plane node, you need to use ssh -p 50001 instead of port 22.
agentVnetSubnetId	only required when using custom VNET and when MasterProfile is using VirtualMachineScaleSets	Specifies the Id of an alternate VNET subnet for all the agent pool nodes. The subnet id must specify a valid VNET ID owned by the same subscription. (bring your own VNET examples). When MasterProfile is using VirtualMachineScaleSets, this value should be the subnetId of the subnet for all agent pool nodes.
availabilityZones	no	To protect your cluster from datacenter-level failures, you can enable the Availability Zones feature for your master VMs. Check out Availability Zones README for more details.
cosmosEtcd	no	True: uses cosmos etcd endpoint instead of installing etcd on masters
auditDEnabled	no	Enable auditd enforcement at the OS layer for each node VM. This configuration is only valid on an agent pool with an Ubuntu-backed distro, e.g., the default "aks-ubuntu-18.04" distro, or the "ubuntu-18.04" distro. Defaults to false
ultraSSDEnabled	no	Enable UltraSSD feature for each node VM. More details about Ultra disk.
customVMTags	no	Specifies a list of custom tags to be added to the master VMs or Scale Sets. Each tag is a key/value pair (ie: "myTagKey": "myTagValue").
sysctldConfig	no	Configure Linux kernel parameters via /etc/sysctl.d/. See sysctldConfig below
proximityPlacementGroupID	no	Specifies the resource id of the Proximity Placement Group (PPG) to be used for master VMs. Please find more details about PPG in this Azure blog. Note that the PPG should be created in advance. The following Azure CLI documentation explains how to create a PPG.
kubeletConfig	no	Configure various runtime configuration for kubelet running on master nodes. See kubeletConfig above

agentPoolProfiles

A cluster can have 0 to 12 agent pool profiles. Agent Pool Profiles are used for creating agents with different capabilities such as VMSizes, VMSS or Availability Set, Public/Private access, user-defined OS Images, attached storage disks, attached managed disks, or Windows.

Name	Required	Description
availabilityProfile	no	Supported values are VirtualMachineScaleSets (default, except for Kubernetes clusters before version 1.10) and AvailabilitySet.
count	yes	Describes the node count
availabilityZones	no	To protect your cluster from datacenter-level failures, you can enable the Availability Zones feature for your node pool. Check out Availability Zones README for more details.
singlePlacementGroup	no	Supported values are true (default) and false. A value of true: A VMSS with a single placement group and has a range of 0-100 VMs. A value of false: A VMSS with multiple placement groups and has a range of 0-1,000 VMs. For more information, check out virtual machine scale sets placement groups. This configuration is only valid on an agent pool with an "availabilityProfile" value of "VirtualMachineScaleSets"
scaleSetPriority	no	Supported values are Regular (default), Low (DEPRECATED) and Spot. This configuration is only valid on an agent pool with an "availabilityProfile" value of "VirtualMachineScaleSets". Enables the usage of Low-priority VMs on Scale Sets or Azure Spot VMs.
scaleSetEvictionPolicy	no	Supported values are Delete (default) and Deallocate. This configuration is only valid on an agent pool with an "availabilityProfile" value of "VirtualMachineScaleSets" and a "scaleSetPriority" value of "Low" or "Spot".
spotMaxPrice	no	Supported values are -1 (default) or any decimal value greater than zero. Only valid on an agent pool with an "availabilityProfile" value of "VirtualMachineScaleSets" and scaleSetPriority value of "Spot". Specifies the maximum price you are willing to pay for a Azure Spot VM/VMSS. This price is in US Dollars. This price will be compared with the current Azure Spot price for the VM size. Also, the prices are compared at the time of create/update of Azure Spot VM/VMSS and the operation will only succeed if the maxPrice is greater than the current Azure Spot price. The maxPrice will also be used for evicting a Azure Spot VM/VMSS if the current Azure Spot price goes beyond the maxPrice after creation of VM/VMSS. You can set the maxPrice to -1 to indicate that the Azure Spot VM/VMSS should not be evicted for price reasons. Also, the default max price is -1 if it is not provided by you
diskSizesGB	no	Describes an array of up to 4 attached disk sizes. Valid disk size values are between 1 and 1024
dataDiskCachingType	no	Which disk caching type configuration to use for the data disks on master VMs. Allowed configurations are "ReadWrite", "ReadOnly", or "None". Default is "ReadOnly".
dnsPrefix	Required if agents are to be exposed publically with a load balancer	The dns prefix that forms the FQDN to access the loadbalancer for this agent pool. This must be a unique name among all agent pools. Not supported for Kubernetes clusters
name	yes	This is the unique name for the agent pool profile. The resources of the agent pool profile are derived from this name
ports	only required if needed for exposing services publically	Describes an array of ports need for exposing publically. A tcp probe is configured for each port and only opens to an agent node if the agent node is listening on that port. A maximum of 150 ports may be specified. Not supported for Kubernetes clusters
storageProfile	no	Specifies the storage profile to use. Valid values are ManagedDisks, StorageAccount, or Ephemeral. Defaults to ManagedDisks. Ephemeral is an experimental feature - please read more on the feature status page
vmsize	yes	Describes a valid Azure VM Sizes. These are restricted to machines with at least 2 cores
osDiskSizeGB	no	Describes the OS Disk Size in GB
osDiskCachingType	no	Which disk caching type configuration to use for the OS disk on master VMs. Allowed configurations are "ReadWrite", "ReadOnly", or "None". Default is "ReadWrite".
vnetSubnetId	no	Specifies the Id of an alternate VNET subnet. The subnet id must specify a valid VNET ID owned by the same subscription. (bring your own VNET examples)
imageReference.name	no	The name of a a Linux OS image. Needs to be used in conjunction with resourceGroup, below
imageReference.resourceGroup	no	Resource group that contains the Linux OS image. Needs to be used in conjunction with name, above
osType	no	Specifies the agent pool's Operating System. Supported values are Windows and Linux. Defaults to Linux
distro	no	Specifies the agent pool's Linux distribution. Currently supported values are: ubuntu-18.04, aks-ubuntu-18.04, ubuntu-18.04-gen2 (Ubuntu 18.04-LTS running on a Generation 2 VM), and flatcar (Flatcar support is currently experimental - Example of Flatcar Master with Flatcar Agents). For Azure Public Cloud, Azure US Government Cloud, and Azure China Cloud, defaults to aks-ubuntu-18.04. For Sovereign Clouds, the default is ubuntu-18.04. aks-ubuntu-18.04 is a custom image based on ubuntu-18.04 that comes with pre-installed software necessary for Kubernetes deployments. Note: the ubuntu and aks-ubuntu-16.04 distro values may be used if you have a reason to use the EOL (End of Life) Ubuntu 16.04-LTS OS; we do not recommend using Ubuntu 16.04-LTS, as the OS will no longer be receiving security and other critical patches as of April 30, 2021.
acceleratedNetworkingEnabled	no	Use Azure Accelerated Networking feature for Linux agents (You must select a VM SKU that supports Accelerated Networking). Defaults to true if the VM SKU selected supports Accelerated Networking
acceleratedNetworkingEnabledWindows	no	Use Azure Accelerated Networking feature for Windows agents (You must select a VM SKU that supports Accelerated Networking). Defaults to false. Setting it to true requires the base Windows vm images contains the right drivers to support accelerated networking.
vmssOverProvisioningEnabled	no	Use Overprovisioning with VMSS. This configuration is only valid on an agent pool with an "availabilityProfile" value of "VirtualMachineScaleSets". Defaults to false
enableVMSSNodePublicIP	no	Enable creation of public IP on VMSS nodes (must use "availabilityProfile": "VirtualMachineScaleSets"). This configuration is only valid on a cluster running a "Basic" LoadBalancer SKU ("loadBalancerSku": "Basic", see: kubernetesConfig). Defaults to false
LoadBalancerBackendAddressPoolIDs	no	Enables automatic placement of the agent pool nodes into existing load balancer's backend address pools. Each element value of this string array is the corresponding load balancer backend address pool's Azure Resource Manager(ARM) resource ID. By default this property is not included in the API model, which is equivalent to an empty string array.
auditDEnabled	no	Enable auditd enforcement at the OS layer for each node VM. This configuration is only valid on an agent pool with an Ubuntu-backed distro, e.g., the default "aks-ubuntu-18.04" distro, or the "ubuntu-18.04" distro. Defaults to false
customVMTags	no	Specifies a list of custom tags to be added to the agent VMs or Scale Sets. Each tag is a key/value pair (ie: "myTagKey": "myTagValue").
diskEncryptionSetID	no	Specifies ResourceId of the disk encryption set to use for enabling encryption at rest (ie: "/subscriptions/{subs-id}/resourceGroups/{rg-name}/providers/Microsoft.Compute/diskEncryptionSets/{diskEncryptionSet-name}"). More details about Server side encryption of Azure managed disks.
ultraSSDEnabled	no	Enable UltraSSD feature for each node VM. More details about Ultra disk.
extensions	no	Specifies list of extensions to enable for the agent profile. More details about agentPoolProfiles extensions
preProvisionExtension	no	Specifies an extension to be run before the cluster is brought up. More details about agentPoolProfiles extensions
sysctldConfig	no	Configure Linux kernel parameters via /etc/sysctl.d/. See sysctldConfig below
proximityPlacementGroupID	no	Specifies the resource id of the Proximity Placement Group (PPG) to be used for this agentpool. Please find more details about PPG in this Azure blog. Note that the PPG should be created in advance. The following Azure CLI documentation explains how to create a PPG.
kubeletConfig	no	Configure various runtime configuration for kubelet running on this node pool. See kubeletConfig above

linuxProfile

linuxProfile provides the linux configuration for each linux node in the cluster

Name	Required	Description
adminUsername	yes	Describes the username to be used on all linux clusters
ssh.publicKeys[].keyData	yes	The public SSH key used for authenticating access to all Linux nodes in the cluster
secrets	no	Specifies an array of key vaults to pull secrets from and what secrets to pull from each
enableUnattendedUpgrades	no (will be required in a future release)	Configure each Linux node VM (including control plane node VMs) to run /usr/bin/unattended-upgrade in the background according to a daily schedule. If enabled, the default unattended-upgrades package configuration will be used as provided by the Ubuntu distro version running on the VM. More information here. By default, enableUnattendedUpgrades is set to true. We encourage you to declare an explicit configuration for enableUnattendedUpgrades and a warning will be logged if you do not. In a future release of aks-engine-azurestack, we may default this to false, requiring the user to self-maintain OS package and security updates.
runUnattendedUpgradesOnBootstrap	no	Invoke an unattended-upgrade when each Linux node VM comes online for the first time. In practice this is accomplished by performing an apt-get update, followed by a manual invocation of /usr/bin/unattended-upgrade, to fetch updated apt configuration, and install all package updates provided by the unattended-upgrade facility, respectively. Defaults to true for public Azure clouds, and to false for Azure Stack Hub and other non-public, custom cloud environments.
customSearchDomain.name	no	describes the search domain to be used on all linux clusters
customSearchDomain.realmUser	no	describes the realm user with permissions to update dns registries on Windows Server DNS
customSearchDomain.realmPassword	no	describes the realm user password to update dns registries on Windows Server DNS
customNodesDNS.dnsServer	no	describes the IP address of the DNS Server
eth0MTU	no	Allows custom MTU configuration on all Linux node eth0 interfaces. At present, only 1500 and 3900 (JSON type int) are allowed. You may not use this configuration if your cluster is configured for kubenet.

Here are instructions for [generating a public/private key pair][ssh] for ssh.publicKeys.keyData.

Notes on SSH public keys

At least one SSH key is required, but multiple are supported when deploying Kubernetes.

Here's a minimal example using just one key:

    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "ssh-rsa AAAA...w=="
          }
        ]
      }
    },

And an example using two keys.

    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "ssh-rsa AAAA...w=="
          },
          {
            "keyData": "ssh-rsa AAAA...w=="
          }
        ]
      }
    },

secrets

secrets details which certificates to install on the masters and nodes in the cluster.

A cluster can have a list of key vaults to install certs from.

On linux boxes the certs are saved on under the directory "/var/lib/waagent/". 2 files are saved per certificate:

1. {thumbprint}.crt : this is the full cert chain saved in PEM format
2. {thumbprint}.prv : this is the private key saved in PEM format

Name	Required	Description
sourceVault.id	yes	The azure resource manager id of the key vault to pull secrets from
vaultCertificates.certificateUrl	yes	Keyvault URL to this cert including the version

format for sourceVault.id, can be obtained in cli, or found in the portal: /subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.KeyVault/vaults/{keyvaultname}

format for vaultCertificates.certificateUrl, can be obtained in cli, or found in the portal: https://{keyvaultname}.vault.azure.net:443/secrets/{secretName}/{version}

windowsProfile

windowsProfile provides configuration specific to Windows nodes in the cluster

Name	Required	Description
adminUsername	yes	Username for the Windows adminstrator account created on each Windows node
adminPassword	yes	Password for the Windows adminstrator account created on each Windows node
csiProxyURL	no	Path to a package containing csi proxy binaries for Windows.
enableAutomaticUpdates	no	If set to true Windows Update will be configured to automatically apply updates on Windows nodes. Default: false
enableCSIProxy	no	If set to true the csi-proxy specified by windowsProfile.csiProxyURL will get installed during node provisioning. See Windows Csi Proxy for more details.
provisioningScriptsPackageURL	no	Path to a package containing provisioning scripts found in /staging/provisioning/windows. Default values will be maintained by aks-engine-azurestack team and contain script content signed by Microsoft.
windowsPauseImageURL	no	Path to a Windows pause image. Default values will be maintained by aks-engine-azurestack team. e.g. "mcr.microsoft.com/oss/kubernetes/pause:3.8"
alwaysPullWindowsPauseImage	no	If set to true, it will always pull the Windows pause image from the repo specified. Default: false.
windowsPublisher	no	Publisher used to find Windows VM to deploy from marketplace. Default: microsoft-aks
windowsOffer	no	Offer used to find Windows VM to deploy from marketplace. Default: aks-windows
windowsSku	no	SKU usedto find Windows VM to deploy from marketplace. Default: 2019-datacenter-core-smalldisk
imageVersion	no	Specific image version to deploy from marketplace. Default: 17763.737.190923. This default is incremented to include the latest Windows patches after being validated by the AKS Engine team.
windowsImageSourceURL	no	Path to an existing Azure storage blob with a sysprepped VHD. This is used to test pre-release or customized VHD files that you have uploaded to Azure. If provided, the above 4 parameters are ignored.
imageReference.name	no	Name of an Image.
imageReference.resourceGroup	no	Resource group that contains the Image.
imageReference.subscriptionId	no	ID of subscription containing a Shared Image Gallery.
imageReference.gallery	no	Name of a Shared Image Gallery.
imageReference.version	no	Version of an Image from a Shared Image Gallery.
sshEnabled	no	If set to true, OpenSSH will be installed on windows nodes to allow for ssh remoting. Only for Windows version 1809/2019 or later. The same SSH authorized public key(s) will be added from linuxProfile.ssh.publicKeys. Default: true
enableAHUB	no	If set to true, Windows nodepools was licensed as Windows Server on-premises. For more information, see Azure Hybrid Use Benefit for Windows Server
windowsSecureTLSEnabled	no	If set to true, it will always enable secure TLS protocols on Windows nodes. Default: false.

Windows Images

You can configure the image used for all Windows nodes one of the following ways:

Defaults

The AKS Engine team produces images that are optimized for and validated against aks-engine-azurestack-created Kubernetes clusters during the regular development and release process. The latest version of these images at the time of a new release of the aks-engine-azurestack binary are used as the default images for Windows nodes.

These images are published to the Azure Marketplace under the microsoft-aks publisher and aks-windows offer. Release notes for these images can be found under releases/vhd-notes/aks-windows.

Marketplace Images

Aks-engine also supports running 'vanilla' Windows Server images published by Microsoft. These can be used by advanced users if a release other than Winders Server 2019 is needed.

If you want to choose a specific Windows image, but automatically use the latest - set windowsPublisher, windowsOffer, and windowsSku. If you need a specific version, then add imageVersion too.

You can find all available images with az vm image list --all --publisher MicrosoftWindowsServer --offer WindowsServer --output table, and the contents of these images are described in the knowledge base article Windows Server release on Azure Marketplace update history.

If you want to use a specific image then windowsPublisher, windowsOffer, windowsSku, and imageVersion must all be set:

"windowsProfile": {
            "adminUsername": "...",
            "adminPassword": "...",
            "windowsPublisher": "MicrosoftWindowsServer",
            "windowsOffer": "WindowsServer",
            "windowsSku": "2019-Datacenter-Core-with-Containers-smalldisk",
            "imageVersion": "2019.0.20181107"
     },

Custom Images

Listed in order of precedence based on what is specified in the API model:

VHD

To use an image uploaded to an Azure storage account (or any other accessible location) specify windowsImageSourceURL.

Note: URLs containing SAS tokens are not allowed by Azure!

"windowsProfile": {
            "adminUsername": "...",
            "adminPassword": "...",
            "windowsImageSourceURL": "https://images.blob.core.windows.net/vhds/custom_windows_image.vhd",
     },

Shared Image Gallery

To use an Image from a Shared Image Gallery specify imageReference.name, imageReference.resourceGroup, imageReference.subscriptionId, imageReference.galllery, and imageReference.version.

"windowsProfile": {
            "adminUsername": "...",
            "adminPassword": "...",
            "imageReference": {
              "name": "custom-image",
              "resourceGroup": "windows-images",
              "subscriptionId": "00000000-0000-0000-0000-000000000000",
              "gallery": "image-gallery",
              "version": "0.1.0"
            }
     },

Azure Image

To use a pre-existing Azure Image specify imageReference.name and imageReference.resourceGroup.

"windowsProfile": {
            "adminUsername": "...",
            "adminPassword": "...",
            "imageReference": {
              "name": "custom-image",
              "resourceGroup": "windows-images"
            }
     },

servicePrincipalProfile

servicePrincipalProfile describes an Azure Service credentials to be used by the cluster for self-configuration. See service principal for more details on creation.

Name	Required	Description
clientId	yes, for Kubernetes clusters	describes the Azure client id. It is recommended to use a separate client ID per cluster
secret	yes, for Kubernetes clusters	describes the Azure client secret. It is recommended to use a separate client secret per client id
objectId	optional, for Kubernetes clusters	describes the Azure service principal object id. It is required if enableEncryptionWithExternalKms is true
keyvaultSecretRef.vaultId	no, for Kubernetes clusters	describes the vault id of the keyvault to retrieve the service principal secret from. See below for format.
keyvaultSecretRef.secretName	no, for Kubernetes clusters	describes the name of the service principal secret in keyvault
keyvaultSecretRef.version	no, for Kubernetes clusters	describes the version of the secret to use

format for keyvaultSecretRef.vaultId, can be obtained in cli, or found in the portal: /subscriptions/<SUB_ID>/resourceGroups/<RG_NAME>/providers/Microsoft.KeyVault/vaults/<KV_NAME>. See keyvault params for an example.