Securing a cluster

August 6, 2019 · View on GitHub

Objectives: > Understanding the cluster attack surface > Concepts that can be applied to configure and bootstrap authentication in azure > Minimizing the blast radius by applying least priviliges inside and outside the cluster > Securing and maintaining host vms > Monitoring and securing security events and logs

Securing a cluster
- Securing endpoints for api server and cluster nodes
- Ensuring authentication and authorization
- Setting up & keeping least privileged access for common tasks
- Create administrative boundaries(namespaces) between resources as sample
- Securing communication paths between namespaces (and nodes)
- Continous Monitoring and Auditing of security relevant events
- Running benchmarks and tests to validate cluster setup
- Regular maintenance, security and cleanup tasks
- Configuration best practices

Master Endpoint Security

The best thing to do is to ensure that RBAC with Azure AD Integration is configured. The API Server has three security levels. The first level is Authentication, the second level is Authorization, and the third level is Admission Controllers. By integrating Authentication and Authorization to Azure AD it means that only users from and Organization's Azure AD, and been granted the appopriate permissions, can access the API Server. If not, they are denied. The last layer around Admission Controllers is about applying policy to ensure that the permitted requests that do come through are adhering to the policy of the Organization. Admission Controllers are discussed more in the securing workloads section.

Securing Host Access

By default, SSH access to the Control Planes is not permitted as AKS is a managed service. In terms of the workers, SSH access from the Public Internet is blocked by default via Network Security Groups (NSGs). Even if someone was able to get inside of the VNET, they would need to have the SSH Keys to be able to access the servers.

The recommended practice is to generate your own SSH Keys ahead of Cluster creation and secure those keys in something like Azure Key Vault (AZK). This was the keys are not floating around or stored on a server for someone to use.

RBAC Setup for users and service accounts

The activation of RBAC in the AKS api server is ensured by default - it is strongly recommended to deploy only clusters with activated RBAC - ideally the administrators and developers should also be using identities from azure ad. By default the privileges on workloads and users should be kept to a minimum. That means that the usage of the cluster-admin role should be minimized for the creation of namespaces, resource quotas and cluster-wide operations - and not for daily operations. The deployment and upgrade of applications should be performed by application specific users/ service accounts as specified in

securing_a_cluster/role_deployment_manager.yaml

If multiple independant applications are operated within the same cluster via helm, then it might make sense to create a dedicated service account for tiller that only has deployment permissions in the desired application namespace. See here for further details: https://github.com/helm/helm/blob/master/docs/rbac.md#example-deploy-tiller-in-a-namespace-restricted-to-deploying-resources-only-in-that-namespace In addition the creation of a dedicated role looking at logs might be useful as specified in

securing_a_cluster/role_log_reader.yaml

Please consider if the usage of the dashboard is required for your enterprise. By default the dashboard service account will not have enough permissions to display all objects or change them - which is considered a good practice. It is not recommended to upgrade the role permissions to cluster-admin.

Container Image security

One could argue that Container Image Management is one of the most critical parts of using Containers. It all starts with DevOps and the ability to automate the deployment of Applications/Services into AKS. If you trust where the images are coming from and put policy restrictions in place so that only trusted images are deployed into the AKS Cluster, that is half the battle.

If we can only deploy from a Trusted Registry then that eliminates the threat of deploying a malicious container from an untrusted registry. So how do we trust the registry? It all starts with a solid DevOps pipline and continuous security in the form of Container Image scanning at Build time as well as Runtime. If we scan Container Images at Build Time and only upload trusted images to the Registry, that means we are only pulling/deploying trusted images to our AKS Cluster. But what if a Container Image is comprimised at runtime, somehow someone has got access to a Container via some exploit? This is where Runtime Image scanning comes into play and let's you know that a Container might have a vulnerability. The policy could be simply to Notify someone, or it can be automated to recycle the Container which would immediately terminate the comprimised Container as the process would be killed and restarted.

Dev Namespace Example

The following is an example of how to configure a Namespace, in this case we will call it dev. The goal is to apply LimitRanges, Quotas and RBACk to the Namespace to ensure it does not become a noisy neighbour and only consume the resources it has intended to along with providing security isolation so that it can only interact with the resources it is intended too.

kubectl apply -f securing_a_cluster/create-namespaces.yaml
kubectl apply -f securing_a_cluster/namespace-limitranges.yaml
kubectl apply -f securing-a_cluster/namespace-quotas.yaml

Upgrading and Maintaing Hosts

As security patches are applied to all nodes in a cluster on a nightly basis you still need to ensure that a potentially required restart is done accordingly. The options are described in the aks faq https://docs.microsoft.com/en-us/azure/aks/faq#are-security-updates-applied-to-aks-agent-nodes. You are left with the responsibility to ensure that these reboots are done in time and that enough spare capacity is available in your cluster while the restart is done to ensure that your workloads can continue to operate.

Security Benchmarks

The purpose of running security benchmarks is to ensure that the proper policy, procedures and controls are in place when an AKS Cluster is created. By running a simple check we can ensure things are ready to go and in place from a security perspective.

Kubebench is a great example of one of the tools, click here for more details.

Activating Add-ons

The recommended practice in this area is to enable monitoring so that logs are captured and can be used for troubleshooting down the road. Given there is no SSH access to the Control Plane and SSH access to the worker nodes is limited, having a solid Monitoring and Loggins strategy in place is key to Troubleshooting. A centralized loggin store is a good place to start.

As for the Kubernetes Dashboard, the recommendation is to not install it. If a solid DevOps pipline is in place along with a Loggin strategy, there is not need to provide access to the Kubernetes Dashboard. Misconfiguration of the Dashboard can be a big security risk, allowing users to have admin level access to do and see everything in the AKS Cluster.

Encrypted Service to Service Communication

The question that usually gets raised, what about traffic inside of the cluster? The question back is what vulnerabilities/risks are we trying to mitigate? If we trust services that have got through the front door, do we need to do anything more? This is uncomfortable for a lot of Organizations as they are used to all service to service communication going through a WAF and monitoring traffic. When using Distributed Systems it does not make sense to have each request go back out to the WAF and come back in as the WAF is not aware of where Services are located, which is also one of the main reasons we use Ingress Controllers as they are Service Discovery aware.

So how do we deal with this trust problem? Typically the risk can be mitigated with the use of mutual TLS (mTLS) between services so that services can only interact with other trusted services. This is where the concept of Service Meshes come in. The details of Service Meshes are a topic for another time.

Private API Server and Service Endpoints

Many Organizations are concerned that the Kubernetes API Server is Publicly exposed and are asking for a Private Endpoint. Private Endpoints are not an option today, but are on the roadmap. The ability to lock down access to the Public Endpoint to only certain VNETs (Service Endpoints) is also not available today, but on the roadmap. It is however already possible to put connected azure resources like databases, storage or keyvault into a dedicated vnet and secure the access to your data via service endpoints.

A bit of a duplicate of the first API Server section.

So what can I do in the interim? The best thing to do is to ensure that RBAC with Azure AD Integration is configured. The API Server has three security levels. The first level is Authentication, the second level is Authorization, and the third level is Admission Controllers. By integrating Authentication and Authorization to Azure AD it means that only users from and Organization's Azure AD, and been granted the appopriate permissions, can access the API Server. If not, they are denied. The last layer around Admission Controllers is about applying policy to ensure that the permitted requests that do come through are adhering to the policy of the Organization. Admission Controllers are discussed more in the securing workloads section.