Deploy a Kubernetes Cluster

October 16, 2018 ยท View on GitHub

Install Prerequisites

All the commands in this guide require both the Azure CLI and acs-engine. Follow the installation instructions to download acs-engine before continuing or compile from source.

For installation instructions see the Azure CLI GitHub repository for the latest release.

Overview

acs-engine reads a cluster definition which describes the size, shape, and configuration of your cluster. This guide takes the default configuration of one master and two Linux agents. If you would like to change the configuration, edit examples/kubernetes.json before continuing.

The acs-engine deploy command automates creation of a Service Principal, Resource Group and SSH key for your cluster. If operators need more control or are interested in the individual steps see the "Long Way" section below.

NOTE: ACS Engine creates a cluster; it doesn't create an Azure Container Service resource. So clusters that you create using the acs-engine command (or ARM templates generated by the acs-engine command) won't show up as ACS resources, for example when you run az acs list. Think of acs-engine as the, er, engine which ACS uses to create clusters: you can use the same engine yourself, but ACS won't know about the results.

After the cluster is deployed the upgrade and scale commands can be used to make updates to your cluster.

Gather Information

  • The subscription in which you would like to provision the cluster. This is a uuid which can be found with az account list -o table.
  • Proper access rights within the subscription. Especially the right to create and assign service principals to applications ( see ACS Engine the Long Way, Step #2)
  • A dnsPrefix which forms part of the the hostname for your cluster (e.g. staging, prodwest, blueberry). The DNS prefix must be unique so pick a random name.
  • A location to provision the cluster e.g. westus2.
$ az account list -o table
Name                                             CloudName    SubscriptionId                        State    IsDefault
-----------------------------------------------  -----------  ------------------------------------  -------  -----------
Contoso Subscription                             AzureCloud   51ac25de-afdg-9201-d923-8d8e8e8e8e8e  Enabled  True

Deploy

For this example, the subscription id is 51ac25de-afdg-9201-d923-8d8e8e8e8e8e, the DNS prefix is contoso-apple, and location is westus2.

Run acs-engine deploy with the appropriate arguments:

$ acs-engine deploy --subscription-id 51ac25de-afdg-9201-d923-8d8e8e8e8e8e \
    --dns-prefix contoso-apple --location westus2 \
    --api-model examples/kubernetes.json

WARN[0005] apimodel: missing masterProfile.dnsPrefix will use "contoso-apple"
WARN[0005] --resource-group was not specified. Using the DNS prefix from the apimodel as the resource group name: contoso-apple
WARN[0008] apimodel: ServicePrincipalProfile was empty, creating application...
WARN[0017] created application with applicationID (7e2d433f-d039-48b8-87dc-83fa4dfa38d4) and servicePrincipalObjectID (db6167e1-aeed-407a-b218-086589759442).
WARN[0017] apimodel: ServicePrincipalProfile was empty, assigning role to application...
INFO[0034] Starting ARM Deployment (contoso-apple-1423145182). This will take some time...
INFO[0393] Finished ARM Deployment (contoso-apple-1423145182).

acs-engine will output Azure Resource Manager (ARM) templates, SSH keys, and a kubeconfig file in _output/contoso-apple-59769a59 directory:

  • _output/contoso-apple-59769a59/azureuser_rsa
  • _output/contoso-apple-59769a59/kubeconfig/kubeconfig.westus2.json

acs-engine generates kubeconfig files for each possible region. Access the new cluster by using the kubeconfig generated for the cluster's location. This example used westus2, so the kubeconfig is _output/<clustername>/kubeconfig/kubeconfig.westus2.json:

$ KUBECONFIG=_output/contoso-apple-59769a59/kubeconfig/kubeconfig.westus2.json kubectl cluster-info
Kubernetes master is running at https://contoso-apple-59769a59.westus2.cloudapp.azure.com
Heapster is running at https://contoso-apple-59769a59.westus2.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/heapster
KubeDNS is running at https://contoso-apple-59769a59.westus2.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://contoso-apple-59769a59.westus2.cloudapp.azure.com/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Administrative note: By default, the directory where acs-engine stores cluster configuration (_output/contoso-apple above) won't be overwritten as a result of subsequent attempts to deploy a cluster using the same --dns-prefix) To re-use the same resource group name repeatedly, include the --force-overwrite command line option with your acs-engine deploy command. On a related note, include an --auto-suffix option to append a randomly generated suffix to the dns-prefix to form the resource group name, for example if your workflow requires a common prefix across multiple cluster deployments. Using the --auto-suffix pattern appends a compressed timestamp to ensure a unique cluster name (and thus ensure that each deployment's configuration artifacts will be stored locally under a discrete _output/<resource-group-name>/ directory).

Note: If the cluster is using an existing VNET please see the Custom VNET feature documentation for additional steps that must be completed after cluster provisioning.

The deploy command lets you override any values under the properties tag (even in arrays) from the cluster definition file without having to update the file. You can use the --set flag to do that. For example:

acs-engine deploy --resource-group "your-resource-group" \
  --location "westeurope" \
  --subscription-id "your-subscription-id" \
  --api-model "./apimodel.json" \
  --set masterProfile.dnsPrefix="your-dns-prefix-override" \
  --set agentPoolProfiles[0].name="your-agentpool-0-name-override" \
  --set agentPoolProfiles[0].count=1 \
  --set linuxProfile.ssh.publicKeys[0].keyData="ssh-rsa PUBLICKEY azureuser@linuxvm" \
  --set servicePrincipalProfile.clientId="spn-client-id" \
  --set servicePrincipalProfile.secret="spn-client-secret"

ACS Engine the Long Way

Step 1: Generate an SSH Key

In addition to using Kubernetes APIs to interact with the clusters, cluster operators may access the master and agent machines using SSH.

If you don't have an SSH key cluster operators may generate a new one.

Step 2: Create a Service Principal

Kubernetes clusters have integrated support for various cloud providers as core functionality. On Azure, acs-engine uses a Service Principal to interact with Azure Resource Manager (ARM). Follow the instructions to create a new service principal and grant it the necessary IAM role to create Azure resources.

Step 3: Edit your Cluster Definition

ACS Engine consumes a cluster definition which outlines the desired shape, size, and configuration of Kubernetes. There are a number of features that can be enabled through the cluster definition: check the examples directory for a number of... examples.

Edit the simple Kubernetes cluster definition and fill out the required values:

  • dnsPrefix: must be a region-unique name and will form part of the hostname (e.g. myprod1, staging, leapingllama) - be unique!
  • keyData: must contain the public portion of an SSH key - this will be associated with the adminUsername value found in the same section of the cluster definition (e.g. 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABA....')
  • clientId: this is the service principal's appId uuid or name from step 2
  • secret: this is the service principal's password or randomly-generated password from step 2

Optional: attach to an existing virtual network (VNET). Details here

Note: you can then use the --set option of the generate command to override values from the cluster definition file directly in the command line (cf. Step 4)

Step 4: Generate the Templates

The generate command takes a cluster definition and outputs a number of templates which describe your Kubernetes cluster. By default, generate will create a new directory named after your cluster nested in the _output directory. If my dnsPrefix was larry my cluster templates would be found in _output/larry-.

Run acs-engine generate examples/kubernetes.json

The generate command lets you override values from the cluster definition file without having to update the file. You can use the --set flag to do that:

acs-engine generate --set linuxProfile.adminUsername=myNewUsername,masterProfile.count=3 clusterdefinition.json

The --set flag only supports JSON properties under properties. You can also work with array, like the following:

acs-engine generate --set agentPoolProfiles[0].count=5,agentPoolProfiles[1].name=myPoolName clusterdefinition.json

Step 5: Submit your Templates to Azure Resource Manager (ARM)

Deploy the output azuredeploy.json and azuredeploy.parameters.json

  • To enable the optional network policy enforcement using calico, you have to set the parameter during this step according to this guide
  • To enable the optional network policy enforcement using cilium, you have to set the parameter during this step according to this guide

Note: If the cluster is using an existing VNET please see the Custom VNET feature documentation for additional steps that must be completed after cluster provisioning.

Checking VM tags

First we get list of Master and Agent VMs in the cluster

az vm list -g <resource group of cluster> -o table
Name                      ResourceGroup                    Location
------------------------  -------------------------------  -------------
k8s-agentpool1-22116803-1       XXXXXXXXXXXX                  southeastasia
k8s-master-22116803-0           XXXXXXXXXXXX                  southeastasia

Once we have the VM Names, we can check tags associated with any of the VMs using the command below

az vm show -g <resource group of cluster> -n <name of Master or agent VM> --query tags

Sample JSON out of this command is shown below. This command can also be used to check the acs-engine version which was used to create the cluster

{
  "acsengineVersion": "v0.15.0",
  "creationSource": "acsengine-k8s-master-22116803-0",
  "orchestrator": "Kubernetes:1.9.5",
  "poolName": "master",
  "resourceNameSuffix": "22116803"
}