This section provides information about the inventory, features, and steps for installing a Kubernetes solution on the environment.

• Prerequisites
  • Prerequisites for Deployment Node
  • Prerequisites for Cluster Nodes
    • Minimal Hardware Requirements
    • Recommended Hardware Requirements
    • Disk Partitioning Recommendation
    • ETCD Recommendation
    • SSH key Recommendation
    • Private Certificate Authority
• Inventory Preparation
  • Deployment Schemes
    • Non-HA Deployment Schemes
      • All-in-one Scheme
      • One Control Plane and Many Workers Schema
    • HA Deployment Schemes
      • Mini-HA Scheme
      • Full-HA Scheme
  • Taints and Toleration
  • Configuration
    • globals
    • node_defaults
    • nodes
    • cluster_name
    • control_plain
    • public_cluster_ip
    • registry
    • gateway_nodes
    • vrrp_ips
    • services
      • kubeadm
        • Kubernetes version
        • Service Account Issuer
      • kubeadm_kubelet
      • kubeadm_kube-proxy
      • kubeadm_patches
      • kernel_security
        • selinux
        • apparmor
      • packages
        • package_manager
        • management
        • associations
      • thirdparties
      • CRI
      • modprobe
      • sysctl
      • audit
        • Kubernetes Policy
        • Daemon
      • ntp
        • chrony
        • timesyncd
      • resolv.conf
      • etc_hosts
      • coredns
      • loadbalancer
    • patches
    • RBAC pss
      • Configuring Default Profiles
      • Configuring Exemptions
      • Application Prerequisites
    • RBAC Accounts
      • RBAC account_defaults
    • RBAC Authenticated Issuer Discovery
    • Plugins
      • Predefined Plugins
        • calico
        • nginx-ingress-controller
        • envoy-gateway
        • kubernetes-dashboard
        • local-path-provisioner
        • openstack-cinder-csi
        • csi-snapshot-controller
      • Plugins Features
        • plugin_defaults
        • Plugins Reinstallation
        • Plugins Installation Order
        • Node Selector
        • Tolerations
        • Resources Requests and Limits
      • Custom Plugins Installation Procedures
        • template
        • config
        • expect pods
        • expect deployments/daemonsets/replicasets/statefulsets
        • python
        • thirdparty
        • shell
        • ansible
        • helm
  • Advanced Features
    • List Merge Strategy
      • Merge Strategy Positioning
      • List Merge Allowed Sections
    • Dynamic Variables
      • Compilation Stages
      • Limitations
      • Jinja2 Expressions Escaping
    • Environment Variables
  • Installation without Internet Resources
• Installation Procedure
  • Installation Tasks Description
  • Installation of Kubernetes using CLI
    • Custom Inventory File Location
• Installation Features
  • Tasks List Redefinition
  • Logging
  • Dump Files
  • Configurations Backup
  • Ansible Inventory
    • Contents
      • [all]
      • [cluster:children]
      • [balancer], [control-plane], [worker]
      • [cluster:vars]
  • Cumulative Points
• Supported Versions

Prerequisites

The technical requirements for all types of host VMs for Kubemarine installation are specified in this section.

Prerequisites for Deployment Node

Ensure the following requirements are met:

Minimal Hardware

• 1 CPU
• 512MB RAM

Operating System

• Linux
• MacOS
• Windows (See Restrictions)

Preinstalled Software

• python 3.9 (or higher version)
• pip3
• Helm 3 (optional, only if Helm plugins required to be installed)

For more information about the different types of Kubemarine installation, refer to README.

System Clock

System clock should be synchronized the same way as for Cluster nodes system clock.

Windows Deployer Restrictions

There are the following restrictions when deploying from Windows:

• ansible plugin procedures are not supported.
• All Kubemarine input text files must be in utf-8 encoding.

Prerequisites for Cluster Nodes

For cluster machines, ensure the following requirements are met:

Host type

• VM
• Bare-Metal

Host arch

• x86-64

Operating System

• The following distributives and versions are supported:

  • CentOS Stream 9
  • RHEL 8.4, 8.6, 8.7, 8.8, 8.9, 8.10, 9.2, 9.3, 9.4
  • Oracle Linux 8.4, 9.2
  • RockyLinux 8.6, 8.7, 8.8, 9.2, 9.3, 9.4, 9.5, 9.6, 10.1
  • Ubuntu 22.04.1, 24.04.1, 26.04

Note: Ubuntu 20.04 is not supported for Kubernetes 1.35 and higher

The actual information about the supported versions can be found in compatibility_map.distributives section of globals.yaml configuration.

Networking

• Opened TCP-ports:
  • Internal communication:
    • 22: SSH
    • 53: CoreDNS, if access is needed from services bound to the host network.
    • 80, 20080, 21080: HTTP
    • 179: Calico BGP
    • 443, 20443, 21443: HTTPS
    • 5443: Calico API server, if enabled
    • 5473: Calico networking with Typha enabled
    • 6443: Kubernetes API server
    • 8443: Ingress NGINX validating webhook
    • 2379-2380: ETCD server & client API
    • 9091: Calico metrics port
    • 9093: Calico Typha metrics port
    • 10250: Kubelet API
    • 30000-32767: NodePort Services
    • Other ports if communication happens between services with any participant bound to the host network.
  • External communication:
    • 22: SSH, if you use external nodes' IP addresses to deploy the cluster.
    • 80
    • 443
    • 6443: Kubernetes API server, if necessary to access externally. For example, using the helm plugin.
• Opened UDP-ports:
  • Internal communication:
    • 53: CoreDNS, if access is needed from services bound to the host network.
    • 4789: Calico VxLAN encapsulation, if that type of encapsulation is enabled
• Enabled additional protocols:
  • VRRP: protocol is using by keepalived to manage vIP between nodes with role balancer. IP protocol number - 112
  • IPIP: protocol is using for communication between kubernetes nodes of several clusters in Active-Active scheme only. Make shure IPIP protocol is added to all Security Groups in OpenStack and allowed on all intermediate network devices. IP protocol number - 4.
• Internal network bandwidth not less than 1GBi/s.
• Dedicated internal address, IPv4, and IPv6 are supported as well, for each VM.
• Any network security policies are disabled or whitelisted. This is especially important for OpenStack environments.
  • Traffic is allowed for Opened TCP-ports between the nodes inside the cluster's internal subnet.
  • TCP & UDP traffic is allowed for pod subnet between the nodes inside the cluster. Search for address at services.kubeadm.networking.podSubnet. By default, 10.128.0.0/14 for IPv4 or fd02::/48 for IPv6.
  • TCP & UDP traffic is allowed for service subnet between the nodes inside the cluster. Search for address at services.kubeadm.networking.serviceSubnet. By default, 172.30.0.0/16 for IPv4 or fd03::/112 for IPv6.

Warning: Kubemarine works only with firewalld as an IP firewall, and switches it off during the installation. If you have other solution, remove or switch off the IP firewall before the installation.

• In case of NetworkManager usage at the control-plane and/or worker nodes, create the following configuration file at /etc/NetworkManager/conf.d/calico.conf to prevent NetworkManager from interfering with the interfaces being created by Calico:

[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:vxlan-v6.calico

Preinstalled Software

• Installation of the following packages is highly recommended; however, Kubernetes can work without them, but may show warnings:
  • ethtool
  • ebtables (included in the iptables-nft package which is available on systems like RHEL 9+)
  • socat

Warning: You have to specify packages names in "RPM format" if it is possible for you OS, For example, specify conntrack-tools instead of conntrack.

Note: For an automated installation, you can use Packages during installation.

Preinstalled or RPM repository provided in cluster.yaml with the following RPMs from Supported versions table

Note:

• You can install a version other than the recommended version, but it is not supported and can cause unpredictable consequences.
• haproxy is provided via standard repositories on supported RHEL/Rocky/OL 8/9.

Warning: RHEL 8 does not have Python preinstalled. For check_iaas to work correctly, it is required to install Python on the nodes. Execute the following step before the installation procedure.

• Install python 3.9 from the standard RHEL repository:

# dnf install python3.9 -y

Unattended-upgrades

unattended-upgrades is a native to Debian/Ubuntu automatic update mechanism. By default, it is included in the system and is configured to updating packages only from security-repository.

Warning: Do not use unattended-upgrade to automatically update packages besides security updates.

Note: Kubemarine disables unattended-upgrades for some packages. See associations for more information;

Note: Since this mechanism is launched once a day at a random time, it may coincide with the work of other package managers (apt, aptitude, dpkg, and so on), resulting in an error in these package managers. In this case, the simplest solution is to wait for the completion of the unattended-upgrades process and restart apt, aptitude, or dpkg.

Preconfigured

• SSHD running on each VM via port 22.
• User with sudo and no-require-tty parameter in sudoers file.
• SSH key is configured on each node. The key should be available for the connection with a username from the previous statement.

Recommended

• Logrotate policy for /var/log/messages is configured according to the planned load (it is recommended to use limited size and daily rotation)

For more information, refer to the Official Kubernetes Requirements Documentation at https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin.

Minimal Hardware Requirements

The minimum hardware requirements for cluster machines are as follows:

Balancer

• 1 CPU
• 1GB RAM
• 10GB HDD

Control-plane

• 2 CPU
• 2GB RAM
• 40GB HDD

Worker

• 4 CPU
• 4GB RAM
• 80GB HDD

Recommended Hardware Requirements

The recommended hardware requirements are as follows:

Balancer

• 2 CPU
• 1GB RAM
• 10GB HDD

Control-plane

• 4 CPU
• 4GB RAM

Worker

• 8 CPU
• 16GB RAM
• 120GB HDD

Disk Partitioning Recommendation

Kubernetes clusters use the following important folders:

/var/lib/etcd - It is used for the etcd database storage at the control-plane nodes. Etcd is very sensitive to disk performance so it is recommended to put /var/lib/etcd to a separate fast disk, for example, SSD. The size of this disk depends on the etcd database size, but not less than 4 GB. For more information about etcd disks, refer to the ETCD Recommendation section.

/var/lib/containerd - It is a working directory of containerd, and is used for active container runtimes and storage of local images. For control-plane nodes, it should be at least 20 GB, whereas, for worker nodes, it should be 50 GB or more, depending on the application requirements.

/var/lib/kubelet - It is a working directory for kubelet. It includes kubelet's configuration files, pods runtime data, environment variables, kube secrets, emptyDirs and data volumes not backed by persistent storage PVs. Its size varies depending on the running applications.

/var/log - It is used for logs from all Linux subsystems (logs of pods are located there too). The recommended size is 10 to 30 GB or more, depending on the logrotation policy. Also, the logrotation should be configured properly to avoid a disk overflow.

Disk Pressure

To detect DiskPressure events for nodes, Kubernetes controls the nodefs and imagefs file system partitions. The nodefs (or rootfs) is the node's main filesystem used for local disk volumes, emptyDir, log storage, and so on. By default, it is /var/lib/kubelet.

The imagefs is an optional filesystem that the container runtimes use to store container images and container writable layers. For containerd, it is the filesystem containing /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs.

If nodefs or imagefs reach the eviction thresholds (100% - nodefs.available, 100% - imagefs.available), the DiskPressure condition becomes true and the pods start being evicted from the node. So it is crucially important not to allow disk fulfillment coming to the eviction threshold for both nodefs and imagefs.

ETCD Recommendation

For a cluster with a high load on the ETCD, it is strongly recommended to mount the dedicated SSD-volumes in the ETCD-storage directory (4 GB size at least is recommended) on each Control-plane before the installation. Mount point:

/var/lib/etcd

For more information about general hardware recommendations, refer to the Official ETCD Documentation at https://etcd.io/docs/latest/op-guide/hardware/.

SSH Key Recommendation

Before working with the cluster, you need to generate an ssh key. Kubemarine supports following types of keys: RSA, DSS, ECDSA, Ed25519.

Example:

ssh-keygen -t rsa -b 4096

Private Certificate Authority

In internal environments, certificates signed by the custom CA root certificate can be used, for example, in a private repository. In this case, the custom CA root certificate should be added to all the cluster nodes.

Examples:

• CentOS/RHEL/Oracle

# yum install ca-certificates
# curl -o /etc/pki/ca-trust/source/anchors/Custom_CA.crt http://example.com/misc/Custom_CA.crt
# update-ca-trust extract

• Ubuntu/Debian:

# apt install ca-certificates
# curl -o /usr/share/ca-certificates/Custom_CA.crt http://example.com/misc/Custom_CA.crt
# echo "Custom_CA.crt" >> /etc/ca-certificates.conf
# update-ca-certificates

Inventory Preparation

Before you begin, select the deployment scheme and prepare the inventory.

Deployment Schemes

Several deployment schemes exist for the cluster installation.

There are two major deployment schemes as follows:

• Non-HA Deployment
• HA Deployment

Non-HA Deployment Schemes

This deployment provides a single Kubemarine control-plane.

All-in-one Scheme

This scheme has one node assigned as control-plane and worker roles; balancer role is optional. This scheme is used for developing and demonstrating purposes only. An example of this scheme is available in the All-in-one Inventory Example.

Note: If you install both ingress-nginx and envoy-gateway at the same time in this schema, you should explicitly provide hostPorts configuration to these plugins, otherwise they both will compete for ports 80/443.

The following image illustrates the All-in-one scheme.

One Control Plane and Many Workers Schema

This schema might be useful if:

• You want a development environment, where you do not care about HA;
• AND you want to use multiple workers (otherwise, All-in-one schema could be used);
• AND you have slow disks, so you want to avoid etcd replication overhead (otherwise, Mini-HA schema could be used);
• AND you do not want to bother with vIP configuration (otherwise, you could combine control-plane and balancer role on one node, without explicit envoy-gateway/ingress-nginx placement).

In this case, you can use schema with single control plane node (which runs envoy-gateway/ingress-nginx) and many (1 or more) workers. An example of this schema is available in the One Control Plane and Many Workers Inventory Example.

Note:

• Optionally, if you want control-plane node to be able to run workloads, you need to add "worker" role to this node.
• If you install both ingress-nginx and envoy-gateway at the same time in this schema, you should explicitly provide hostPorts configuration to these plugins, otherwise they both will compete for ports 80/443.

The following image illustrates the "One Control Plane and Many Workers" schema.

HA Deployment Schemes

This deployment type provides a highly available and reliable solution.

Mini-HA Scheme

In this scheme, the control-plane, balancer, and worker roles are all assigned to odd number of identical nodes (at least 3). In this scheme, it is mandatory to enable VRRP to leverage balancing. An example of this scheme is available in the Mini-HA Inventory Example.

The following image illustrates the Mini-HA scheme.

Full-HA Scheme

In this scheme, several nodes are assigned different roles. The number of control-plane nodes should be odd, three, or more. The number of worker nodes should be greater than one or more than three. The recommended number of balancer nodes is two, with configured VRRP, but one balancer without VRRP is also supported. An example of this scheme presented is available in the Minimal Full-HA Inventory Example and Typical Full-HA Inventory Example.

The following image illustrates the Full-HA scheme.

Taints and Toleration

A node, taint, lets you mark a node so that the scheduler avoids or prevents using it for certain pods. A complementary feature, tolerations, lets you designate pods that can be used on "tainted" nodes.

Node taints are key-value pairs associated with an effect. Following are the available effects:

• NoSchedule. The pods that do not tolerate this taint are not scheduled on the node; the existing pods are not evicted from the node.
• PreferNoSchedule. Kubernetes avoids scheduling the pods that do not tolerate this taint onto the node.
• NoExecute. A pod is evicted from the node if it is already running on the node, and is not scheduled onto the node if it is not yet running on the node.

Note: Some system pods, for example, kube-proxy and fluentd, tolerate all NoExecute and NoSchedule taints, and are not evicted.

In general, taints and tolerations support the following use cases:

• Dedicated nodes. You can use a combination of node affinity and taints/tolerations to create dedicated nodes. For example, you can limit the number of nodes onto which to schedule pods by using labels and node affinity, apply taints to these nodes, and then add corresponding tolerations to the pods to schedule them on those particular nodes.
• Nodes with special hardware. If you have nodes with special hardware, for example, GPUs, you have to repel pods that do not need this hardware and attract pods that need it. This can be done by tainting the nodes that have the specialized hardware and adding the corresponding toleration to pods that must use this special hardware.
• Taint-based evictions. New Kubernetes versions allow configuring per-pod eviction behavior on nodes that experience problems.

To set taint to any node, you can apply the following command:

kubectl taint nodes <NODENAME> <KEY>=<VALUE>:<EFFECT>

To remove the taint added by command above you can run:

kubectl taint nodes <NODENAME> <KEY>=<VALUE>:<EFFECT>-

Where:

• NODENAME is the name of the tainted node.
• KEY is the name of the taint. For example, special, database, infra, and so on.
• VALUE is the value for the taint.
• EFFECT is the effect for the taint behavior. It can be one of NoSchedule, PreferNoSchedule, or NoExecute.

To deploy pods on tainted nodes, define the toleration section:

tolerations:
- key: <KEY>
  operator: Equal
  value: <VALUE>
  effect: <EFFECT>

A toleration "matches" a taint, if the keys are the same and the effects are the same, and:

• the operator is Exists (in which case no value should be specified), or
• the operator is Equal and the values are equal.

Note: An empty key with operator Exists matches all keys, values, and effects which specifies that this tolerates everything.

CoreDNS Deployment with Node Taints

By default, CoreDNS pods are scheduled to worker nodes. If the worker nodes have taints, the CoreDNS must have tolerations configuration in cluster.yaml, otherwise, the CoreDNS pods get stuck in the Pending state. For example:

services:
  coredns:
    deployment:
      spec:
        template:
          spec:
             tolerations:
              - key: application
                operator: Exists
                effect: NoSchedule

Plugins Deployment with Node Taints

The plugins also require the tolerations section in case of node taints. The Calico pods already have tolerations to be assigned to all the cluster nodes. However, for other plugins, it should be set in cluster.yaml. For more information, see Tolerations.

If you create your own plugins, the tolerations settings should be taken into account.

Configuration

All the installation configurations for the cluster are in a single inventory file. It is recommended to name this file as cluster.yaml.

For more information about the structure of the inventory and how to specify the values, refer to the following configuration examples:

• Minimal Full-HA Inventory Example - It provides the minimum set of parameters required to install a cluster out of the box.
• Typical Full-HA Inventory Example - It provides a set of parameters that you probably want to configure.
• Full Full-HA Inventory Example - It provides almost all the possible parameters that you can configure.
• Minimal All-in-one Inventory Example - It provides the minimum set of parameters for deploying All-in-one scheme.
• Minimal Mini-HA Inventory Example - It provides the minimum set of parameters for deploying Mini-HA scheme.

Inventory validation

When configuring the inventory, you can use your favorite IDE supporting YAML validation by JSON schema. JSON schema for inventory file can be used by URL. Do not try to copy the schema content.

Make sure to use a raw URL (at raw.githubusercontent.com) without any query parameters. The JSON schema is naturally versioned by a Kubemarine version, specifically, by GitHub tag or branch that you are currently checking.

Note that the inventory file is validated against the same schema at runtime.

The known IDEs that support validation are:

• PyCharm or other IntelliJ based IDEs.
• Red Hat's extension for Visual Studio Code.

The inventory file consists of the following sections.

globals

In the globals section, you can describe the global parameters that can be overridden. For example:

globals:
  expect:
    pods:
      plugins:
        retries: 300
  nodes:
    ready:
      timeout: 10
      retries: 60

The following parameters are supported:

node_defaults

In the node_defaults section, you can describe the parameters to be applied by default to each record in the nodes section. For example, by adding the keyfile parameter in this section, it is copied to all elements of the nodes list. However, if this parameter is defined in any element of nodes list, it is not replaced in it.

For example, you can have the following inventory content:

node_defaults:
  keyfile: "/home/username/.ssh/id_rsa"
  username: "centos"

node:
  - name: "lb"
    internal_address: "192.168.0.1"
    roles: ["balancer"]
  - name: "control-plane"
    keyfile: "/home/username/another.key"
    internal_address: "192.168.0.2"
    roles: ["control-plane"]

After executing the above example, the final result is displayed as follows:

node:
  - name: "lb"
    username: "centos"
    keyfile: "/home/username/.ssh/id_rsa"
    internal_address: "192.168.0.1"
    roles: ["balancer"]
  - name: "control-plane"
    username: "centos"
    keyfile: "/home/username/another.key"
    internal_address: "192.168.0.2"
    roles: ["control-plane"]

Following are the parameters allowed to be specified in the node_defaults section:

• keyfile, password, username, connection_port, connection_timeout, gateway, and boot.timeout.
• labels, and taints - specify at global level only if the Mini-HA Scheme is used. For more information about the listed parameters, refer to the following section.

Note: To establish an ssh connection, you can use either a keyfile or a password. In case if you are specifying both, keyfile will be considered on priority.

Note: Set an environment variable with the desired password. For example, you can use export PASS="your_password". In the cluster.yaml file, specify the password field using the environment variable syntax. For instance: password: '{{ env.PASS }}'. This syntax instructs the code to fetch the value from the PASS environment variable and substitute it into the password field when reading the configuration file. See environment variables for more details.

nodes

In the nodes section, it is necessary to describe each node of the future cluster.

The following options are supported:

An example with parameters values is as follows:

node_defaults:
  keyfile: "/home/username/.ssh/id_rsa"
  password: '{{ env.PASS }}'     #Either keyfile or password can be used.
  username: "centos"
  boot:
    timeout: 900

nodes:
  - name: "k8s-lb"
    address: "10.101.0.1"
    internal_address: "192.168.0.1"
    roles: ["balancer"]
  - name: "k8s-control-plane-1"
    address: "10.101.0.2"
    internal_address: "192.168.0.2"
    roles: ["control-plane"]
    labels:
      region: asia
    taints:
      - "node-role.kubernetes.io/remove-example:NoSchedule-"
      - "node-role.kubernetes.io/add-example=add-example:NoSchedule"
  - name: "k8s-worker-1"
    address: "10.101.0.5"
    internal_address: "192.168.0.5"
    roles: ["worker"]
    labels:
      netcracker-infra: infra
      region: europe

The example is also available in Full Inventory Example.

Warning: Please be informed that the master role is obsolete and will be changed by control-planein the future. The control-plane and master roles are interchangeable at the moment. Therefore it's possible to use the control-plane and master roles in any procedure.

cluster_name

In the cluster_name variable specify the future address of the cluster. On this address, the Control Plane Endpoint is raised, and it is used in the calculated parameters.

An example is as follows:

cluster_name: "k8s-stack.sdntest.example.com"

Note: Cluster name should be a fully qualified domain name. It should not be an IP address.

For more information, refer to FQDN at https://en.wikipedia.org/wiki/Fully_qualified_domain_name

control_plain

control_plain parameter specifies which addresses are to be available for Kubernetes. The internal and external parameter is described in the following table.

This parameter is calculated in the first turn so that it can be referenced in inventory parameters. For example:

answer: '3600 IN A {{ control_plain["internal"] }}'

This is an autocalculated parameter, but you can override it if you are aware about the procedure. For example:

control_plain:
  internal: 192.168.0.1
  external: 10.101.0.1

Automatic calculation works according to the following principle: The algorithm iterates through and looks for appropriate addresses.

Addresses are taken from the following groups in order:

1. VRRP IP
2. Balancer
3. Control-plane

Note: VRRP IPs with maintenance-type: "not bind" do not participate in the control_plain calculation. For more information, see maintenance type.

Note: It is important to notice that addresses may not necessarily be taken from a single group. There may be situation that the internal address is taken from the VRRP, and the external one from the Balancer. This situation is not recommended, but it is possible. If the inventory is correctly filled in and all the addresses that are available are indicated, the algorithm automatically selects the best pair of addresses.

After detecting addresses, the algorithm automatically displays the determined addresses and their sources as follows:

Control plains:
   Internal: 192.168.0.1 (vrrp_ip[0])
   External: 10.101.1.101 (balancer "balancer-1")

control_endpoint

The algorithm of control_plain calculation chooses the very first address if there are several elements in the group. If you are not satisfied with this principle, you can "help" the algorithm in choosing which address to take by specifying the control_endpoint parameter for the group element. For example:

vrrp_ips:
- ip: 192.168.0.1
  floating_ip: 10.101.0.1
- ip: 192.168.0.2
  floating_ip: 10.101.0.2
  control_endpoint: True

The above example produces the following result:

Control plains:
   Internal: 192.168.0.2 (vrrp_ip[1])
   External: 10.101.0.2 (vrrp_ip[1])

An example with mixed groups:

vrrp_ips:
- ip: 192.168.0.1
  floating_ip: 10.101.0.1
- ip: 192.168.0.2
  control_endpoint: True

nodes:
- name: balancer-1
  internal_address: 192.168.0.3
  address: 10.101.0.3
- name: balancer-2
  internal_address: 192.168.0.4
  address: 10.101.0.4
  control_endpoint: True

The above example produces the following result:

Control plains:
   Internal: 192.168.0.2 (vrrp_ip[1])
   External: 10.101.0.4 (balancer "balancer-2")

Note: control_endpoint is not taken into account for VRRP IPs with maintenance-type: "not bind".

public_cluster_ip

Warning: public_cluster_ip is an obsolete variable, use control_plain.external variable instead.

public_cluster_ip variable specifies the Kubernetes external address to connect from an external network. This variable is optional and required if you are using Helm plugins installation.

By default public_cluster_ip inherits control_plain["external"] as shown in the following code:

public_cluster_ip: '{{ control_plain["external"] }}'

However, it is possible to change an address if the external control_plain parameter is not suitable. For example, if the cluster is behind an external balancer as shown in the following code.

public_cluster_ip: "10.102.0.1"

registry

If you want to install Kubernetes in a private environment without access to the internet, then you need to redefine the addresses of remote resources. These resources are many, so for convenience there is a single unified registry parameter that allows you to specify the registry for everything at once. To do this, you need to specify registry section in the root of the inventory and fill it with parameters.

The registry parameter automatically completes the following parameters:

Note: You can enter these parameters yourself, as well as override them, even if the registry parameter is set.

Registry section support 2 formats - new endpoints definition and old-style address-port format. We recommend to use new endpoints format as in the future we will abandon the old format. Only one format can be used.

registry (new endpoints format)

The following parameters are supported:

Endpoint value is a string with an address (protocol, host, and port). Record format example:

registry:
  endpoints:
    - https://repository-01.example.com:17001
    - https://repository-02.example.com:27001

Also, you can mix this types. Full example:

registry:
  thirdparties: https://repository-03.example.com:8080/custom_location
  endpoints:
    - https://repository-01.example.com:17001
    - https://repository-02.example.com:27001
  mirror_registry: "registry.cluster.local"

registry (old address-port format)

The following parameters are supported:

Example:

registry:
  address: example.com
  docker_port: 5443
  webserver: True
  ssl: False

This configuration generates the following parameters:

services:
  kubeadm:
    imageRepository: example.com:5443/registry.k8s.io
  cri:
    containerdRegistriesConfig:
      example.com:5443:
        host."http://example.com:5443":
          capabilities: [pull, resolve]
  thirdparties:
    /usr/bin/calicoctl:
      source: http://example.com/projectcalico/calico/{{calico-version}}/calicoctl-linux-amd64
    /usr/bin/crictl.tar.gz:
      source: http://example.com/kubernetes-sigs/cri-tools/{{crictl-version}}/crictl-{{crictl-version}}-linux-amd64.tar.gz
    /usr/bin/etcd.tar.gz:
      source: http://example.com/etcd-io/etcd/{{etcdutl-version)/etcd-{{etcdutl-version}}-linux-amd64.tar.gz
    /usr/bin/kubeadm:
      source: http://example.com/kubernetes/kubeadm/{{k8s-version}}/bin/linux/amd64/kubeadm
    /usr/bin/kubectl:
      source: http://example.com/kubernetes/kubectl/{{k8s-version}}/bin/linux/amd64/kubectl
    /usr/bin/kubelet:
      source: http://example.com/kubernetes/kubelet/{{k8s-version}}/bin/linux/amd64/kubelet
plugin_defaults:
  installation:
    registry: example.com:5443

However, if you override one of the replaced parameters, it is not replaced. For example, with the following configuration:

registry:
  address: example.com
  docker_port: 5443
  webserver: True
  ssl: False
services:
  kubeadm:
    imageRepository: 1.1.1.1:8080/test

The following configuration is produced:

services:
  kubeadm:
    imageRepository: 1.1.1.1:8080/test
  cri:
    containerdRegistriesConfig:
      example.com:5443:
        host."http://example.com:5443":
          capabilities: [pull, resolve]
...

gateway_nodes

If you do not have direct SSH-access to the cluster nodes from the deployer node and you need to connect via the gateway, you can specify the gateway nodes through which you need to create an SSH-tunnel. You can specify several gateways.

The following parameters are supported:

An example is as follows:

gateway_nodes:
  - name: k8s-gateway-1
    address: 10.102.0.1
    username: root
    password: '{{ env.PASS }}'
    keyfile: "/home/username/.ssh/id_rsa"
    
  - name: k8s-gateway-2
    address: 10.102.0.2
    username: root
    password: '{{ env.PASS }}'
    keyfile: "/home/username/.ssh/id_rsa"

You need to specify which gateways should be used to connect to nodes.

An example is as follows:

nodes:
  - name: "k8s-control-plane-1"
    address: "10.101.0.2"
    internal_address: "192.168.0.2"
    roles: ["control-plane"]
    gateway: k8s-gateway-1
  - name: "k8s-control-plane-2"
    address: "10.101.0.3"
    internal_address: "192.168.0.3"
    roles: ["control-plane"]
    gateway: k8s-gateway-2

Note: If the gateway is not specified on the node, then the connection is direct.

Note: To establish an ssh connection, you can use either a keyfile or a password.

vrrp_ips

Installation task: deploy.loadbalancer.keepalived

Can cause reboot: No

Can restart service: Always yes, keepalived

OS specific: Yes, different OS may have different default network interfaces. For interfaces with the autodetection mode selected, it is automatically detected by the internal_address property of the node on which the particular VRRP IP should be set. By default, autodetection is enabled.

In order to assign VRRP IP, you need to create a vrrp_ips section in the inventory and specify the appropriate configuration. You can specify several VRRP IP addresses. This configuration will be applied in keepalived configuration on balancer nodes. If it's needed, it's possible to specify global parameters or override whole keepalived configuration. More about it is in keepalived section in loadbalancer specification.

The following parameters are supported:

There are several formats in which you can specify values.

The following are some examples of the format:

You can specify only the address of the VRRP IP, in which case it automatically applies to all balancers in the cluster and other parameters are automatically calculated. For example:

vrrp_ips:
- 192.168.0.1

You can specify the address and which hosts it should apply to. Other parameters are automatically calculated. For example:

vrrp_ips:
- hosts:
  - name: balancer-1
    priority: 254
  - name: balancer-2
    priority: 253
  ip: 192.168.0.1
  floating_ip: 10.101.1.1

You can specify all possible parameters at one time instead of using auto-calculated. For example:

vrrp_ips:
- hosts:
  - name: balancer-1
    priority: 254
    interface: eth0
  - name: balancer-2
    priority: 253
    interface: ens3
  id: d8efc729e4
  ip: 192.168.0.1
  floating_ip: 10.101.1.1
  password: 11a1aabe
  router_id: '1'

maintenance type

Generally, the maintenance configuration is the same as the default configuration for balancer. The maintenance_type option allows to change the default behavior. The following example describes the type of traffic that is applicable for a particular IP in the maintenance mode configuration. (not bind specifies that the IP receives neither TCP nor HTTP traffic):

vrrp_ips:
- ip: 192.168.0.1
  floating_ip: 10.101.1.1
- ip: 192.168.0.2
  floating_ip: 10.101.1.2
  params:
    maintenance-type: "not bind"

services

In the services section, you can configure the service settings. The settings are described in the following sections.

kubeadm

Installation task: deploy.kubernetes

Can cause reboot: no

Can restart service: always yes, kubelet

OS specific: No

In the services.kubeadm section, you can override the original settings for kubeadm. For more information about these settings, refer to the Official Kubernetes Documentation at https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file. By default, the installer uses the following parameters:

The following is an example of kubeadm defaults override:

services:
  kubeadm:
    networking:
      podSubnet: '10.128.0.0/14'
      serviceSubnet: '172.30.0.0/16'
    imageRepository: example.com:5443/registry.k8s.io
    apiServer:
      extraArgs:
        enable-admission-plugins: NodeRestriction,PodNodeSelector
        profiling: "false"
        audit-log-path: /var/log/kubernetes/audit/audit.log
        audit-policy-file: /etc/kubernetes/audit-policy.yaml
        audit-log-maxage: "30"
        audit-log-maxbackup: "10"
        audit-log-maxsize: "100"
    scheduler:
      extraArgs:
        profiling: "false"
    controllerManager:
      extraArgs:
        profiling: "false"
        terminated-pod-gc-threshold: "1000"
    etcd:
      local:
        extraArgs:
          heartbeat-interval: "1000"
          election-timeout: "10000"

Note: Those parameters remain in manifests files after Kubernetes upgrade. That is the proper way to preserve custom settings for system services.

Note: These kubeadm parameters can be reconfigured after installation using Reconfigure Procedure.

During init, join, upgrade procedures kubeadm runs preflight procedure to do some preliminary checks. In case of any error, kubeadm stops working. Sometimes it is necessary to ignore some preflight errors to deploy or upgrade successfully.

Kubemarine allows to configure kubeadm preflight errors to be ignored.

Example:

services:
  kubeadm_flags:
    ignorePreflightErrors: Port-6443,CoreDNSUnsupportedPlugins,DirAvailable--var-lib-etcd

Note: Default settings for ignorePreflightErrors are:

services:
  kubeadm_flags:
    ignorePreflightErrors: Port-6443,CoreDNSUnsupportedPlugins

Kubernetes version

By default, the v1.34.2 version of the Kubernetes is installed. See the table of supported versions for details in Supported versions section. However, we recommend that you explicitly specify the version you are about to install. This version applies into all the dependent parameters - images, binaries, rpms, configurations. All these are downloaded and used according to your choice. To specify the version, use the following parameter as in example:

services:
  kubeadm:
    kubernetesVersion: v1.34.2

Service Account Issuer

Warning:

• Manual Service Account Issuer setup on an already installed Kubernetes cluster is not supported.
• Service Account Issuer feature is supported only on Kubernetes 1.20.

If Service Account Issuer is required, you can configure the necessary Kubernetes parameters using the kubeadm section in the cluster config. For example:

services:
  kubeadm:
    apiServer:
      extraArgs:
        feature-gates: "ServiceAccountIssuerDiscovery=true"
        service-account-issuer: "https://{{ cluster_name }}:6443"
        service-account-jwks-uri: "https://{{ cluster_name }}:6443/openid/v1/jwks"
        service-account-signing-key-file: /etc/kubernetes/pki/sa.key
        service-account-key-file: /etc/kubernetes/pki/sa.pub
    controllerManager:
      extraArgs:
        feature-gates: "ServiceAccountIssuerDiscovery=true"
    scheduler:
      extraArgs:
        feature-gates: "ServiceAccountIssuerDiscovery=true"

To be able to fetch the public keys and validate the JWT tokens against the Kubernetes cluster’s issuer, you have to allow external unauthenticated requests. To do this, bind the special role, system:service-account-issuer-discovery, with a ClusterRoleBinding to unauthenticated users. Make sure that this is safe in your environment, but only public keys are visible on the URL.

Warning: The following command opens an unauthenticated access to the endpoint receiving public tokens. Do not execute this command if you do not need to open access to the outside, or if you do not understand what you are doing. If you still decide to open an external access, make sure to provide secure access to this endpoint with external resources outside the cluster.

For example:

kubectl create clusterrolebinding oidc-reviewer --clusterrole=system:service-account-issuer-discovery --group=system:unauthenticated

If you need to test that the Service Account Issuer is working, implement the following steps:

1. Create a test pod:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  serviceAccountName: default
  containers:
    - image: nginx:alpine
      name: oidc
      volumeMounts:
        - mountPath: /var/run/secrets/tokens
          name: oidc-token
  volumes:
    - name: oidc-token
      projected:
        sources:
          - serviceAccountToken:
              path: oidc-token
              expirationSeconds: 7200
              audience: vault
EOF

2. Verify whether oidc-token is available:

kubectl exec nginx -- cat /var/run/secrets/tokens/oidc-token

3. Verify ServiceAccount JWT:

kubectl exec nginx -- cat /var/run/secrets/kubernetes.io/serviceaccount/token

4. Get the CA signing certificate of the Kubernetes API Server’s certificate to validate it:

kubectl exec nginx -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt > kubernetes_ca.crt

5. Visit well-known OIDC URLs:

curl --cacert kubernetes_ca.crt https://CLUSTER_NAME:6443/.well-known/openid-configuration

Example result:

{
  "issuer": "https://localhost:6443",
  "jwks_uri": "https://localhost:6443/openid/v1/jwks",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ]
}

6. Visit the JWKS address ("jwks_uri") to view public keys:

curl --cacert kubernetes_ca.crt https://CLUSTER_NAME:6443/openid/v1/jwks

Example result:

{
  "keys": [
    {
      "use": "sig",
      "kty": "RSA",
      "kid": "Rt3TBA31bh3rH67PQbKImg2ldwhPqBTWF2w1Hxqi84c",
      "alg": "RS256",
      "n": "vL0tjBqLDFTyqOCPBQC5Mww_3xkhlkWmeklPjSAhFuqL0U-Oie9E1z8FuhcApBaUs7UEPzja02PEZd4i1UF2UDoxKYEG9hG5vPseTXwN_xGnbhOaBdfgQ7KDvqV-WHfmlrnnCizi1VmNAHsoAg6oZMiUdOuk8kCFxpe0N6THmBKNSKnqoRnhSL4uwHSBWJ5pEyWAqyL8KYaaGYhc2MVUs3I8e-gtQE6Vlwe75_QSp9uIZNZeFr5keqiXhz8BWL76ok-vY8UZ8-rH2VIN5LzXkCvhIFI9W_UBzziSnb9l5dgSQCwGf18zVgT0yJjCz0Z9YE9A1Wgeu-LLrJz3gxR8Hw",
      "e": "AQAB"
    }
  ]
}

kubeadm_kubelet

Installation task: deploy.kubernetes

Can cause reboot: no

Can restart service: always yes, kubelet

OS specific: No

In the services.kubeadm_kubelet section, you can override the original settings for kubelet. For more information about these settings, refer to the official Kubernetes documentation at https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file. By default, the installer uses the following parameters:

podPidsLimit the default value is chosen to prevent Fork Bomb

cgroupDriver field defines which cgroup driver the kubelet controls. Configuring the kubelet cgroup driver.

serializeImagePulls parameter defines whether the images will be pulled in parallel (false) or one at a time.

Note: Some of the parameters can be reconfigured after installation using Reconfigure Procedure.

Warning: If you want to change the values of variables podPidsLimit and maxPods, you have to update the value of the pid_max (this value should not less than result of next expression: maxPods * podPidsLimit + 2048), which can be done using task prepare.system.sysctl. To get more info about pid_max you can go to sysctl section.

The following is an example of kubeadm defaults override:

services:
  kubeadm_kubelet:
    readOnlyPort: 0
    protectKernelDefaults: true
    podPidsLimit: 2048
    maxPods: 100
    cgroupDriver: systemd

kubeadm_kube-proxy

Installation task: deploy.kubernetes

Can cause reboot: No

Can restart service: Always yes, kubelet

OS specific: No

In the services.kubeadm_kube-proxy section, you can override the original settings for kube-proxy. For more information about these settings, refer to the Official Kubernetes Documentation at https://kubernetes.io/docs/reference/config-api/kube-proxy-config.v1alpha1/#kubeproxy-config-k8s-io-v1alpha1-KubeProxyConfiguration.

Note: conntrack.min inherits the services.sysctl.net.netfilter.nf_conntrack_max value from sysctl, therefore if you would like to change this variable, set the desired value in services.sysctl.net.netfilter.nf_conntrack_max.

Note: These parameters can be reconfigured after installation using the Reconfigure Procedure.

kubeadm_patches

Installation task: deploy.kubernetes

Can cause reboot: no

Can restart service: always yes, kubelet

OS specific: No

In services.kubeadm_patches section, you can override control-plane pod settings as well as kubelet settings on per node basis. This feature is implemented by using of kubeadm patches.

The following is an example of control-plane settings override:

services:
  kubeadm_patches:
    apiServer:
      - groups: [control-plane]
        patch:
          max-requests-inflight: 500
      - nodes: [master-3]
        patch:
          max-requests-inflight: 600
    etcd:
      - nodes: [master-1]
        patch:
          snapshot-count: 110001
      - nodes: [master-2]
        patch:
          snapshot-count: 120001
      - nodes: [master-3]
        patch:
          snapshot-count: 130001
    controllerManager:
      - groups: [control-plane]
        patch:
          authorization-webhook-cache-authorized-ttl: 30s
    scheduler:
      - nodes: [master-2,master-3]
        patch:
          profiling: true
    kubelet:
      - nodes: [worker5]
        patch:
          maxPods: 100
      - nodes: [worker6]
        patch:
          maxPods: 200

By default, Kubemarine sets bind-address parameter of kube-apiserver to node.internal_address via patches at every control-plane node.

Note: These parameters can be reconfigured after installation using the Reconfigure Procedure.

Note: If a parameter of control-plane pods is defined in kubeadm.<service>.extraArgs or is set by default by kubeadm and then redefined in services.kubeadm_patches, the pod manifest file will contain the same flag twice, the value taken from services.kubeadm_patches will be in the end of the arguments list.

kernel_security

This is a common section for selinux and apparmor properties.

selinux

Installation task: prepare.system.setup_selinux

Can cause reboot: Yes, only on configurations change

Can restart service: No

Overwrite files: Yes, only on configurations change: /etc/selinux/config, backup is created

OS specific: Yes, performs only on RHEL OS family.

All the SELinux settings are specified in the services.kernel_security.selinux section of the inventory.

Note: SELinux configuration is possible only on nodes running Centos or RHEL operating system.

The following parameters are available:

- haproxy_t
- container_t
- keepalived_t

Warning: It is recommended to use the default values. Using values different from the default may cause an unexpected consequences and no support is provided for consequences.

Note: Turning off and then turning on SELinux can lead to the loss of security rules, which were configured earlier.

For more information about SELinux, refer to the Official RedHat SELinux Configuration Documentation at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/index.

The procedure for applying SELinux settings is as follows:

1. The current settings on remote hosts are validated first. The detected configurations are precisely compared with the configurations from the inventory:
   • SELinux status compared with services.selinux.state - Specified if SELinux needs to be disabled or not.
   • Current mode and Mode from config file compared with the services.selinux.state parameter.
   • Loaded policy name and Policy from config file compared with the services.selinux.policy parameter.
   • Customized Permissive Types items compared with items in the services.selinux.permissive list parameter.
2. If there are no differences, then proceeds with the following tasks.
3. If there is at least one difference, the application of all SELinux settings for remote nodes begins.
4. After applying the settings, it is planned to reboot and re-validate the required configurations.
5. When re-validating, everything is checked again as described in Step 1.

apparmor

Installation task: prepare.system.setup_apparmor

Can cause reboot: Yes, only on configurations change.

Can restart service: No

Overwrite files: Yes, only on configurations change: /etc/apparmor.d, no backups.

OS specific: Yes, performs only on Ubuntu OS family.

All the AppArmor settings are specified in the services.kernel_security.apparmor section of the inventory.

Note: AppArmor configuration is possible only on nodes running Ubuntu or Debian operating system.

In the AppArmor section, you must declare the already existing AppArmor profiles, the state of which needs to be changed. It is not necessary to indicate all the available profiles - something that is not indicated is not affected. The profiles should be specified in a standard way using a path or name. It is possible to modify the following states:

• enforce (default mode) - prohibits everything according to the profile settings.
• complain - does not prohibit, but only displays violation warnings in the logs.
• disable - disables and unloads security profile.

Note: The necessary profiles are installed and configured by themselves during the installation of packages and their activation manually is not required by default. However, if some profiles are missing for you, you need to preload them on all nodes yourself and launch the AppArmor task after that.

Example:

services:
  kernel_security:
    apparmor:
      enforce:
        - /etc/cron/daily/logrotate
        - /sbin/dhclient
        - nvidia_modprobe
      complain:
        - /usr/lib/postfix/smtp
        - man_filter
      disable:
        - /usr/bin/ping
        - /bin/netstat
        - man_groff 

If you need to disable AppArmor, you cannot do this using Kubemarine. If you absolutely need it, you can uninstall AppArmor from the system through the package manager.

Note: After the installation of new repositories, the repodata is reloaded.

packages

package_manager

Installation task: prepare.package_manager.configure

Can cause reboot: No

Can restart service: No

Overwrite files: Yes, /etc/yum.repos.d/ or /etc/apt/sources.list.d/ backup is presented.

OS specific: Yes, different defaults for different OS families.

Warning: This section is specific to different OS families. Ensure that you use the proper definition format for your OS distributive - it may differ from the presented examples in this document.

If your cluster is in a closed environment or if you need to add additional package manager repositories, you can specify them in the services.packages.package_manager section of inventory. The following parameters are supported:

In the repositories section, you need to specify the new repositories to install. The contents of their configurations can be arbitrary and is directly forwarded into the yum repo files.

For example in CentOS:

services:
  packages:
    package_manager:
      replace-repositories: true
      repositories:
        kubernetes:
          name: "Kubernetes"
          enabled: 1
          gpgcheck: 0
          baseurl: "https://pkgs.k8s.io/core:/stable:/v1.33/rpm/"
        my_own_repo:
          name: "My repository"
          enabled: 1
          gpgcheck: 1
          baseurl: "https://example.com/repo"

For example in Ubuntu:

services:
  packages:
    package_manager:
      replace-repositories: true
      repositories:
        - "deb [arch=amd64 trusted=yes] http://example.com/deb/ubuntu/ focal main restricted"
        - "deb [arch=amd64 trusted=yes] http://example.com/deb/ubuntu/ focal-updates main restricted"
        - "deb [arch=amd64 trusted=yes] http://example.com/deb/ubuntu/ focal universe"
        - "deb [arch=amd64 trusted=yes] http://example.com//deb/ubuntu/ focal-updates universe"
        - "deb [arch=amd64 trusted=yes] http://example.com//deb/ubuntu/ focal multiverse"
        - "deb [arch=amd64 trusted=yes] http://example.com/deb/ubuntu/ focal-updates multiverse"
        - "deb [arch=amd64 trusted=yes] http://example.com/deb/ubuntu/ focal-backports main restricted universe multiverse"
        - "deb [arch=amd64 trusted=yes] http://example.com/deb/misc/docker-ce/debian/ buster stable"

Note: You cannot specify the repositories for different package managers. The package manager is detected automatically and the specified configuration should match it.

Note: You can use secure repositories with credentials as for thirdparties.

management

Installation task: prepare.package_manager.manage_packages

Can cause reboot: Yes, only when a list of installed packages changes.

Can restart service: No

OS specific: Yes, the necessary package manager is selected for different OS families.

mandatory

By default, the installer installs predefined list of mandatory packages from the package manager. The mandatory packages are:

• conntrack
• iptables
• curl
• openssl
• unzip
• semanage
• kmod

The exact package names are detected automatically depending on the OS family of the cluster. For more information, see associations.

Warning: Make sure to have all the mandatory packages available in the repositories. You can configure the necessary repositories in the package_manager section of the inventory.

Most of the mandatory packages are installed on all nodes with the following exceptions:

• conntrack and iptables are installed only on control-plane and worker nodes.
• unzip is installed only on nodes that require thirdparties that are packed in .zip archives. For more information, see the unpack option in thirdparties.
• semanage is installed only on RHEL nodes.

If you need to turn some mandatory packages off for some reason, it can be done in the services.packages.mandatory section. For example:

services:
  packages:
    mandatory:
      conntrack: false

custom

If you need other custom packages, you can manage them directly during the installation. You can choose any one of the following types of actions:

• remove
• install
• upgrade

All these actions are performed in a sequence as described above. You can specify only some types of actions or all at once. Short and full configuration formats are available.

Warning: Before you start working, ensure to check that you have all the necessary dependencies in the repositories you are using. You can configure the necessary repositories in the package_manager section of inventory.

Warning: This section is specific to different OS families. Ensure that you use the proper definition format for your OS distributive - it may differ from the presented examples in this document.

Warning: The packages in the install section are installed on all nodes.

The following is an example to install new packages:

services:
  packages:
    install:
      - ethtool
      - ebtables
      - socat

The following is an example to install, upgrade, and remove packages:

services:
  packages:
    remove:
      - socat
    install:
      - ebtables
    upgrade:
      - ethtool

The format of package definition is same as in the package manager. You can specify the exact version of package to install:

services:
  packages:
    install:
      - ebtables-2.0.*
      - ethtool-4.*

To update all packages, you can use an asterisk. For example:

services:
  packages:
    upgrade:
      - *

A more complex format is also available in which you can enable and exclude packages from processing:

services:
  packages:
    upgrade:
      include:
        - *
      exclude:
        - kernel
        - gluster

Warning: Be careful with managing packages, they directly affect the host operating system.

Warning: If changes in the installed packages list are detected, a reboot is scheduled.

associations

Installation task: No

Can cause reboot: No

Can restart service: No

OS specific: Yes, different defaults for different OS families.

Warning: This section is specific to different OS families. Ensure that you use the proper definition format for your OS distributive - it may differ from the presented examples in this document.

In the services.packages section, there is a services.packages.associations sub-section that allows you to configure predefined associations of package objects. It allows you to redefine the following knowledges:

• executable_name
• package_name
• service_name
• config_location

This setting is required to change the behavior of the installer such as to install a package with a different name, use the configuration file from the different path, and so on.

Note: Associations defaults automatically switches for different OS families. Do not worry about this; use those associations that are specific to your operating system in the common section - specify which one is not required in common cases.

The following associations are used by default:

RHEL and Centos

Ubuntu and Debian

Note:

• By default, the packages' versions are installed according to the Kubernetes version specified in the Supported versions section.
• In the procedure for adding nodes, the package versions are taken from the current nodes to match the nodes in the cluster. For example, if containerd.io-1.6.4-1 is installed on the nodes of the cluster, this version is installed on the new node. This behavior can be changed by setting the cache_versions option to "false". The package versions are then used only with the template from the associations section. The option can be used both in global services.packages and in specific associations sections.
• 'cache_versions' is installed to 'false' for following packages:
  • openssl;
  • curl;
  • unzip;
  • kmod;
  • semanage;
• Kubemarine disables unattended_upgrades for packages on Ubuntu/Debian nodes with "true" cache_versions option.

The following is an example of overriding containerd associations:

services:
  packages:
    cache_versions: true
    associations:
      containerd:
        cache_versions: false
        executable_name: 'containerd'
        package_name:
          - 'containerd.io-1.6*'
        service_name: 'containerd'
        config_location: '/etc/containerd/config.toml'

In case when you need to redefine associations for multiple OS families at once, you should define their names in the root of associations in the following way:

services:
  packages:
    cache_versions: true
    associations:
      debian:
        haproxy:
          executable_name: '/usr/sbin/haproxy'
          package_name: haproxy=1.8.*
      rhel9:
        haproxy:
          executable_name: '/usr/sbin/haproxy'
          package_name: haproxy-1.8*

Note: There are following supported OS families: debian, ubuntu26.04, rhel8 (for RHEL-based version 8), rhel9 (for RHEL-based version 9) and rhel10.

thirdparties

Installation task: prepare.thirdparties

Can cause reboot: No

Can restart service: No

Overwrite files: Yes, backup is not presented

OS specific: No

The installer has a mechanism to automatically deliver files from third party sources and install them in the system. For example, using it is convenient to automatically download a certain file from a repository and place it in a specific place in the system. This is configured in the services.thirdparties section. The contents of this section are as follows:

• The absolute destination path on the host system of the cluster is indicated as a key
• A set of the following parameters is indicated as values:

Warning: verify that you specified the path to the correct version of the thirdparty.

Note: Groups and nodes can be combined.

Note: If no groups and nodes are present, then the file is uploaded to control-planes and workers by default.

Note: If the file is already uploaded to hosts and its hash matches with the hash in the config, then the file is not downloaded again.

Note: The installation of the thirdparties sources that are required in the plugins are installed with the plugin. For more information, see thirdparty.

By default, the installer installs the following thirdparties with the following configuration:

services:
  thirdparties:
    /usr/bin/kubeadm:
      source: 'https://dl.k8s.io/{{k8s-version}}/bin/linux/amd64/kubeadm'
    /usr/bin/kubelet:
      source: 'https://dl.k8s.io/{{k8s-version}}/bin/linux/amd64/kubelet'
    /usr/bin/kubectl:
      source: 'https://dl.k8s.io/{{k8s-version}}/bin/linux/amd64/kubectl'
      group: control-plane
    /usr/bin/calicoctl:
      source: 'https://github.com/projectcalico/calico/releases/download/{{calico-version}}/calicoctl-linux-amd64'
      group: control-plane
    /usr/bin/crictl.tar.gz:
      source: 'https://github.com/kubernetes-sigs/cri-tools/releases/download/{{crictl-version}}/crictl-{{crictl-version}}-linux-amd64.tar.gz'
      group: control-plane
      unpack: /usr/bin/
    /usr/bin/etcd.tar.gz:
      source: 'https://github.com/etcd-io/etcd/releases/download/{{etcdutl-version}}/etcd-{{etcdutl-version}}-linux-amd64.tar.gz'
      group: control-plane

If necessary, you can redefine or add thirdparties. For example:

services:
  thirdparties:
    /usr/bin/kubeadm:
      source: https://example.com/kubernetes/kubeadm/v1.22.2/bin/linux/amd64/kubeadm

secure registries for thirdparties

If you have secure registry for thirdparties, you can specify credentials in source field using HTTP basic configuration, e.g.:

services:
  thirdparties:
    /usr/bin/kubeadm:
      source: 'http://user:password@some-registry-url/kubernetes-release/release/{{k8s-version}}/bin/linux/amd64/kubeadm'
    /usr/bin/kubelet:
      source: 'http://user:password@some-registry-url/kubernetes-release/release/{{k8s-version}}/bin/linux/amd64/kubelet'
    /usr/bin/kubectl:
      source: 'http://user:password@some-registry-url/kubernetes-release/release/{{k8s-version}}/bin/linux/amd64/kubectl'
    /usr/bin/calicoctl:
      source: 'http://user:password@some-registry-url/projectcalico/calico/releases/download/{{calico-version}}/calicoctl-linux-amd64'
    /usr/bin/crictl.tar.gz:
      source: 'http://user:password@some-registry-url/kubernetes-sigs/cri-tools/releases/download/{{crictl-version}}/crictl-{{crictl-version}}-linux-amd64.tar.gz'
    /usr/bin/etcd.tar.gz:
      source: 'http://user:password@some-registry-url/etcd-io/etcd/releases/download/{{etcdutl-version}}/etcd-{{etcdutl-version}}-linux-amd64.tar.gz'

Note: Percent-encoded (URL encoded) should be used for credentials. For example for username user and password p@$$w0rD:

services:
  thirdparties:
    /usr/bin/kubeadm:
      source: 'http://user:p%40%24%24w0rD@some-registry-url/kubernetes-release/release/{{k8s-version}}/bin/linux/amd64/kubeadm'

Also it's possible to use environment variables for such credentials. For that kubemarine provide following jinja functions to convert strings:

• b64encode to encode to base64;
• b64decode to decode from base64;
• url_quote to use percent-encoding;

Note: Such functions can be used not only for thirdparties, but in any places and order;

For example, if you use REG_USERNAME and REG_PASSWORD environments for credentials, you can use following configuration:

services:
  thirdparties:
    /usr/bin/kubeadm:
      source: 'http://{{ env.REG_USERNAME | url_quote }}:{{ env.REG_PASSWORD | url_quote }}@some-registry-url/kubernetes-release/release/{{k8s-version}}/bin/linux/amd64/kubeadm'
    /usr/bin/kubelet:
      source: 'http://{{ env.REG_USERNAME | url_quote }}:{{ env.REG_PASSWORD | url_quote }}@some-registry-url/kubernetes-release/release/{{k8s-version}}/bin/linux/amd64/kubelet'
    /usr/bin/kubectl:
      source: 'http://{{ env.REG_USERNAME | url_quote }}:{{ env.REG_PASSWORD | url_quote }}@some-registry-url/kubernetes-release/release/{{k8s-version}}/bin/linux/amd64/kubectl'
    /usr/bin/calicoctl:
      source: 'http://{{ env.REG_USERNAME | url_quote }}:{{ env.REG_PASSWORD | url_quote }}@some-registry-url/projectcalico/calico/releases/download/{{calico-version}}/calicoctl-linux-amd64'
    /usr/bin/crictl.tar.gz:
      source: 'http://{{ env.REG_USERNAME | url_quote }}:{{ env.REG_PASSWORD | url_quote }}@some-registry-url/kubernetes-sigs/cri-tools/releases/download/{{crictl-version}}/crictl-{{crictl-version}}-linux-amd64.tar.gz'
    /usr/bin/etcd.tar.gz:
      source: 'http://{{ env.REG_USERNAME | url_quote }}:{{ env.REG_PASSWORD | url_quote }}
      @some-registry-url/etcd-io/etcd/releases/download/{{etcdutl-version}}/etcd-{{etcdutl-version}}-linux-amd64.tar.gz'

CRI

Installation task: prepare.cri

Can cause reboot: No

Can restart service: Always yes, containerd

Overwrite files: Yes, by default /etc/containerd/config.toml, /etc/crictl.yaml and /etc/ctr/kubemarine_ctr_flags.conf, backup is created. Additionally, if plugins."io.containerd.grpc.v1.cri".registry.config_path is defined in services.cri.containerdConfig specified directory will be created and filled according services.cri.containerdRegistriesConfig.

OS specific: No

The services.cri section configures container runtime used for kubernetes. By default, the following parameters are used:

services:
  cri:
    containerRuntime: containerd
    containerdConfig:
      version: 2
      plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc:
        runtime_type: "io.containerd.runc.v2"
      plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options:
        SystemdCgroup: true
      # This parameter is added, if no registry.mirrors and registry.configs.tls specified:
      plugins."io.containerd.grpc.v1.cri".registry:
        config_path: "/etc/containerd/certs.d"

Note: default value of SystemdCgroup = true only in case, when cgroupDriver from kubelet config is equal to systemd.

The containerRuntime parameter configures a particular container runtime implementation used for kubernetes. Currently, only containerd is available and used by default.

It is possible to additionally define the containerdConfig section, which contains the parameters passed to config.toml, for example:

services:
  cri:
    containerRuntime: containerd
    containerdConfig:
      plugins."io.containerd.grpc.v1.cri":
        sandbox_image: registry.k8s.io/pause:3.9

Also, it is possible to specify the registries configuration in registries hosts format using containerdRegistriesConfig section:

services:
  cri:
    containerRuntime: containerd
    containerdRegistriesConfig:
      artifactory.example.com:5443:
        host."https://artifactory.example.com:5443":
          capabilities: [ "pull", "resolve" ]

If containerdRegistriesConfig is specified, registries hosts configuration is placed in /etc/containerd/certs.d directory by default. It's possible to override this value plugins."io.containerd.grpc.v1.cri".registry.config_path in containerdConfig:

services:
  cri:
    containerRuntime: containerd
    containerdConfig:
      plugins."io.containerd.grpc.v1.cri".registry:
        config_path: "/etc/containerd/registries"
    containerdRegistriesConfig:
      artifactory.example.com:5443:
        host."https://artifactory.example.com:5443":
          capabilities: [ "pull", "resolve" ]

Note: it's possible to specify registries using registry.mirrors and registries.configs.tls, but it's not recommended, because this approach is deprecated by containerd team since containerd v1.5.0:

services:
  cri:
    containerRuntime: containerd
    containerdConfig:
      plugins."io.containerd.grpc.v1.cri".registry.mirrors."artifactory.example.com:5443":
        endpoint:
        - https://artifactory.example.com:5443
      plugins."io.containerd.grpc.v1.cri".registry.configs."artifactory.example.com:5443".tls:
        insecure_skip_verify: true

Note: registry.mirrors and registries.configs.tls can't be used with config_path or containerdRegistriesConfig because it's restricted by containerd. In that case, kubemarine fails with exception during enrichment.

Although, registry.mirrors and registries.configs are deprecated, when the registry requires an authentication, it should be specified using registries.configs.auth, as in following example:

services:
  cri:
    containerRuntime: containerd
    containerdConfig:
      plugins."io.containerd.grpc.v1.cri".registry.configs."private-registry:5000".auth:
        auth: "bmMtdXNlcjperfr="
    containerdRegistriesConfig:
      private-registry:5000:
        host."https://private-registry:5000":
          capabilities: [ "pull", "resolve" ]
          skip_verify: true

Where, auth: "bmMtdXNlcjperfr=" field is username:password string in base64 encoding.

Note: it's possible to use environment variables for credentials as for thirdparies. For example, you can use REG_USERNAME and REG_PASSWORD environments with folowing configuration:

services:
  cri:
    containerRuntime: containerd
    containerdConfig:
      plugins."io.containerd.grpc.v1.cri".registry.configs."private-registry:5000".auth:
        auth: "{{ (env.REG_USERNAME ~ ':' ~ env.REG_PASSWORD ) | b64encode }}"
    containerdRegistriesConfig:
      private-registry:5000:
        host."https://private-registry:5000":
          capabilities: [ "pull", "resolve" ]
          skip_verify: true

Or more simple:

services:
  cri:
    containerRuntime: containerd
    containerdConfig:
      plugins."io.containerd.grpc.v1.cri".registry.configs."private-registry:5000".auth:
        username: "{{ env.REG_USERNAME }}"
        password: "{{ env.REG_PASSWORD }}"
    containerdRegistriesConfig:
      private-registry:5000:
        host."https://private-registry:5000":
          capabilities: [ "pull", "resolve" ]
          skip_verify: true

Note how containerdConfig and containerdRegistriesConfig.<registry> sections reflect the toml format structure. For more details on containerd configuration, refer to the Official containerd configuration file documentation at https://github.com/containerd/containerd/blob/main/docs/cri/config.md. By default, the following parameters are used for containerdConfig:

services:
  cri:
    containerdConfig:
      version: 2
      plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc:
        runtime_type: "io.containerd.runc.v2"
      plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options:
        SystemdCgroup: true
      # This parameter is added, if no registry.mirrors and registry.configs.tls specified:
      plugins."io.containerd.grpc.v1.cri".registry:
        config_path: "/etc/containerd/certs.d"

Note: crictl binary is also installed and configured as required.

Note: After applying the parameters, containerd is restarted on all nodes in the cluster.

modprobe

Installation task: prepare.system.modprobe

Can cause reboot: Yes, only when a list of kernel modules changes.

Can restart service: No

Overwrite files: Yes, only when a list of kernel modules changes, /etc/modules-load.d/predefined.conf, backup is created

OS specific: Yes

The services.modprobe section manages Linux Kernel modules to be loaded in the host operating system. By default, the following modules are loaded:

If necessary, you can redefine or add List Merge Strategy to the standard list of Kernel modules to load. For example (Debian OS family):

services:
  modprobe:
    debian:
      - my_own_module1
      - my_own_module2

There is also an extended format that allows to choose nodes on which the modules should be loaded. For example:

services:
  modprobe:
    debian:
      - modulename: my_own_module1
        groups: [control-plane, worker]
      - modulename: my_own_module2
        nodes: [balancer-1, balancer-2]

The following settings are supported in the extended format:

Note: You can specify nodes and groups at the same time.

Note: If no groups or nodes are specified, then the module is loaded on all nodes.

Warning: Be careful with these settings, they directly affect the hosts operating system.

Warning: If changes to the hosts modprobe configurations are detected, a reboot is scheduled. After the reboot, the new parameters are validated to match the expected configuration.

sysctl

Installation task: prepare.system.sysctl

Can cause reboot: Yes, only when list of Kernel parameters changes.

Can restart service: No

Overwrite files: Yes, only when list of Kernel parameters changes: /etc/sysctl.d/98-kubemarine-sysctl.conf, backup is created

OS specific: No

The services.sysctl section manages the Linux Kernel parameters for all hosts in a cluster. By default, the following key-values are configured:

Constant value equal to 2048 means the maximum number of processes that the system can require during run (only processes of the Linux virtual machine itself are implied). This value have been established empirically.

Note: You can also define the kernel.pid_max value by your own, but you need to be sure that it is at least greater than the result of the expression: maxPods * podPidsLimit + 2048. For more information about the podPidsLimit and maxPods values, refer to the kubeadm_kubelet section.

Warning: Also, in both the cases of calculation and manual setting of the pid_max value, the system displays a warning if the specified value is less than the system default value equal to 32768. If the pid_max value exceeds the maximum allowable value of 4194304, the installation is interrupted.

Note: All default parameters are installed only on control-plane, worker nodes, except net.ipv4.ip_nonlocal_bind, and net.ipv6.ip_nonlocal_bind that are installed on balancer nodes only by default.

Note: Before Kubernetes 1.21 sysctl property net.ipv4.conf.all.route_localnet have been set automatically to 1 by Kubernetes, but now it setting by Kubemarine defaults. Kubernetes 1.21 Urgent Upgrade Notes.

You can specify your own parameters instead of the standard parameters. You need to specify the parameter key and its value. If the value is empty, the key is ignored. For example:

services:
  sysctl:
    net.bridge.bridge-nf-call-iptables: 1
    net.ipv4.ip_forward: 0
    net.ipv4.ip_nonlocal_bind: 0

There is also an extended format that allows to choose nodes on which the parameters should be managed. For example:

services:
  sysctl:
    net.netfilter.nf_conntrack_max:
      value: 1000000
      groups: [control-plane, worker]
      nodes: [balancer-1]
      install: true
    vm.max_map_count:
      value: 262144

The following settings are supported in the extended format:

Note: You can specify the nodes and groups at the same time.

Note: If no groups or nodes are specified, then the parameter is installed on all nodes.

Note: Per-node patches are also supported for this section.

Note: Kernel parameters can be reconfigured after installation using Reconfigure Procedure.

Warning: Be careful with these settings, they directly affect the hosts operating system.

Warning: If the changes to the hosts sysctl configurations are detected, a reboot is scheduled. After the reboot, the new parameters are validated to match the expected configuration.

audit

Audit Kubernetes Policy

Installation task: deploy.kubernetes.audit

Can cause reboot: No

Can restart service: Always yes, container kube-apiserver.

Overwrite files: Yes, /etc/kubernetes/audit-policy.yaml backup is created.

OS specific: No

For more information about Kubernetes auditing, refer to the Official Documentation at https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/.

Logging level: None - do not log; Metadata — log request metadata: user, request time, target resource (pod, namespace, etc.), action type (verb), etc.; Request — log metadata and request body; RequestResponse - log metadata, request body and response body.

omitStages: The list of stages for which no events are created.

By default, the following policy is installed:

services:
  audit:
    cluster_policy:
      apiVersion: audit.k8s.io/v1
      kind: Policy
      # Don't generate audit events for all requests in RequestReceived stage.
      omitStages:
        - "RequestReceived"
      rules:
        # Don't log read-only requests
        - level: None
          verbs: ["watch", "get", "list"]
        # Don't log checking access by internal services
        - level: None
          userGroups:
            - "system:serviceaccounts:calico-apiserver"
            - "system:nodes"
          verbs: ["create"]
          resources:
            - group: "authorization.k8s.io"
              resources: ["subjectaccessreviews"]
            - group: "authentication.k8s.io"
              resources: ["tokenreviews"]
        # Don't log update of ingress-controller-leader ConfigMap by ingress-nginx.
        # This reproduces only for v1.2.0 and can be removed after its support stop.
        - level: None
          users: ["system:serviceaccount:ingress-nginx:ingress-nginx"]
          verbs: ["update"]
          resources:
            - group: ""
              resources: ["configmaps"]
              resourceNames: ["ingress-controller-leader"]
        # Log all other resources in core and extensions at the request level.
        - level: Metadata
          verbs: ["create", "update", "patch", "delete", "deletecollection"]
          resources:
          - group: ""
            resources:
            - configmaps
            - endpoints
            - limitranges
            - namespaces
            - nodes
            - persistentvolumeclaims
            - persistentvolumes
            - pods
            - replicationcontrollers
            - resourcequotas
            - secrets
            - serviceaccounts
            - services
          - group: "apiextensions.k8s.io"
            resources:
            - customresourcedefinitions
          - group: "apps"
            resources:
            - daemonsets
            - deployments
            - replicasets
            - statefulsets
          - group: "batch"
            resources:
            - cronjobs
            - jobs
          - group: "rbac.authorization.k8s.io"
            resources:
            - clusterrolebindings
            - clusterroles
            - rolebindings
            - roles
          - group: "autoscaling"
            resources:
            - horizontalpodautoscalers
          - group: "storage.k8s.io"
            resources:
            - storageclasses
            - volumeattachments
          - group: "networking.k8s.io"
            resources:
            - ingresses
            - ingressclasses
            - networkpolicies
          - group: "authentication.k8s.io"
            resources: ["tokenreviews"]
          - group: "authorization.k8s.io"
          - group: "projectcalico.org"
            resources:
              - bgpconfigurations
              - bgpfilters
              - bgppeers
              - blockaffinities
              - caliconodestatuses
              - clusterinformations
              - felixconfigurations
              - globalnetworkpolicies
              - globalnetworksets
              - hostendpoints
              - ipamconfigurations
              - ippools
              - ipreservations
              - kubecontrollersconfigurations
              - networkpolicies
              - networksets
              - profiles
          - group: "crd.projectcalico.org"
            resources:
              - bgpconfigurations
              - bgpfilters
              - bgppeers
              - blockaffinities
              - caliconodestatuses
              - clusterinformations
              - felixconfigurations
              - globalnetworkpolicies
              - globalnetworksets
              - hostendpoints
              - ipamblocks
              - ipamconfigs
              - ipamhandles
              - ippools
              - ipreservations
              - kubecontrollersconfigurations
              - networkpolicies
              - networksets

It is possible not only to redefine the default policy, but also to extend it. For more information, refer to List Merge Strategy.

For example, consider you have an operator that constantly updates some Pods' labels and some ConfigMap to maintain the leadership. If you do not see any benefit from logging of such events, they can be disabled by specifying the following in cluster.yaml:

services:
  audit:
    cluster_policy:
      rules:
      - level: None
        userGroups: ["system:serviceaccounts:operator-namespace"]
        verbs: ["patch", "update"]
        namespaces: ["operator-namespace"]
        resources:
        - group: ""
          resources: [pods]
        - group: ""
          resources: [configmaps]
          resourceNames: [controller-leader]
      - '<<': merge

Audit Daemon

Installation tasks:

• prepare.system.audit.install
• prepare.system.audit.configure

Can cause reboot: No

Can restart service: Always yes, auditd.

OS specific: No

services:
  audit:
    rules:
      - -w /usr/bin/containerd -k docker

ntp

This is a common section for chrony and timesyncd properties.

For Kubernetes and ETCD to work correctly, it is recommended to configure the system time synchronization on all nodes of the cluster. However, this is optional and you can do it at your own discretion.

chrony

Installation task: prepare.ntp.chrony

Can cause reboot: No

Can restart service: Always yes, chronyd

Overwrite files: Yes, /etc/chrony.conf, backup is created

OS specific: Yes, performs only on the RHEL OS family and Ubuntu 26.04.

Warning: Incorrect time synchronization can lead to incorrect operation of the cluster or services. You can validate the time synchronization via the Time difference test between the nodes from PAAS Check procedure.

To synchronize the system time, you must make a list of NTP servers. All servers must be accessible from any node of the cluster. The list should be indicated in the chrony section of the services.ntp section config file. In addition to the NTP server address, you can specify any additional configurations in the same line.

The following parameters are supported:

For more information about Chrony configuration, refer to the Official Chrony Documentation at https://chrony.tuxfamily.org/documentation.html.

The following is a configuration example:

services:
  ntp:
    chrony:
      servers:
        - ntp1.example.com iburst
        - ntp2.example.com iburst

An example is also available in Full Inventory Example.

Synchronization is configured with the prepare.ntp.chrony task. The task performs the following:

• Generates the chrony.conf file and uploads it to the /etc/chrony (or /etc/chrony/chrony.conf for Ubuntu 26.04) directory on all cluster hosts. If dumping is enabled, the config dump is saved.
• Restarts the chronyd.service service
• Checks if the synchronization is done by the first host of the cluster. Leap status should become normal.

If the configuration services.ntp.chrony.servers is absent, then the task prepare.ntp.chrony in the installation is skipped.

timesyncd

Installation task: prepare.ntp.timesyncd

Can cause reboot: No

Can restart service: Always yes, systemd-timesyncd.

Overwrite files: Yes, /etc/systemd/timesyncd.conf, backup is created.

OS specific: Yes, performs only on Debian OS family (except Ubuntu 26.04).

Warning: incorrect time synchronization can lead to incorrect operation of the cluster or services. You can validate the time synchronization via the Time difference test between the nodes from PAAS Check procedure.

To synchronize the system time, you must make a list of NTP servers. All servers must be accessible from any node of the cluster. The list should be indicated in the timesyncd.Time.NTP parameter of the services.ntp section config file. In addition to the NTP server address, you can specify any additional configurations in the same line.

The following parameters are supported:

The following is a configuration example:

services:
  ntp:
    timesyncd:
      Time:
        NTP:
          - ntp1.example.com
          - ntp2.example.com

Synchronization is configured with the prepare.ntp.timesyncd task. The task performs the following:

• Generates the timesyncd.conf file and uploads it to the /etc/systemd/ directory on all cluster hosts. If dumping is enabled, the config dump is saved.
• Restarts the systemd-timesyncd service.
• Checks if the synchronization is done by the first host of the cluster. The leap status should become normal.

If the configuration services.ntp.timesyncd.servers is absent, then the task prepare.ntp.timesyncd in the installation is skipped.

resolv.conf

Installation task: prepare.dns.resolv_conf

Can cause reboot: No

Can restart service: No

Overwrite files: Yes, /etc/resolv.conf, backup is created

OS specific: No

The services.resolv.conf section allows you to configure the nameserver addresses to which cluster systems has access. By default, this section is empty in the inventory. The following parameters are supported:

Note:

• If some network resources are located in a restricted network and are not resolved through the standard DNS, be sure to configure this section and specify your custom DNS service.
• Do not put ${cluster_name} in the search field, otherwise some microservices might work incorrectly.

For example:

services:
  resolv.conf:
    search: default
    nameservers:
      - 1.1.1.1
      - 1.0.0.1
      - 2606:4700:4700::1111
      - 2606:4700:4700::1001

etc_hosts

Installation task: prepare.dns.etc_hosts

Can cause reboot: no

Can restart service: no

Overwrite files: Yes, /etc/hosts, backup is created

OS specific: No

The installation procedure has a task that generates and applies /etc/hosts configuration file on all nodes presented in the cluster.

Warning: This task overwrites the existing original /etc/hosts file records on all hosts. If you need to save these records, manually move them into inventory file to serives.etc_hosts section.

By default, the generated file contains the following address associations:

• Localhost for IPv4 and IPv6
• Internal control-plain address as control-plain and FQDN name
• Balancers, control-planes, workers names and theirs FQDNs

In order to setup your custom address, you need to specify the IP-address as the key and DNS-name as the list item. Example:

services:
  etc_hosts:
    1.1.1.1:
      - example.com

Example of generated file:

127.0.0.1        localhost localhost.localdomain localhost4 localhost.localdomain4
::1              localhost localhost.localdomain localhost6 localhost6.localdomain6
1.1.1.1          example.com
100.100.100.101  k8s-stack.example.com control-plain balancer-1.k8s-stack.sdntest.example.com balancer-1
100.100.100.102  control-plane-1.k8s-stack.example.com control-plane-1
100.100.100.103  control-plane-2.k8s-stack.example.com control-plane-2
100.100.100.104  control-plane-3.k8s-stack.example.com control-plane-3
100.100.100.105  worker-1.k8s-stack.example.com worker-1
100.100.100.106  worker-2.k8s-stack.example.com worker-2
100.100.100.107  worker-3.k8s-stack.example.com worker-3

You can specify multiple addresses at once, for example:

services:
  etc_hosts:
    1.1.1.1:
      - example.com
      - demo.example.com

This generates the following result:

...
1.1.1.1          example.com demo.example.com
...

It is possible to define custom names for internal and external ip addresses of the cluster nodes. In this case /etc/hosts file at the nodes will contain 2 lines about the same IP address. For example, setting the following in the cluster.yaml:

services:
  etc_hosts:
    100.100.100.102:  another-name-for-control-plane-1

will lead to the following content of /etc/hosts:

...
100.100.100.102  another-name-for-control-plane-1
...

100.100.100.102  control-plane-1.k8s-stack.example.com control-plane-1
...

Warning: The content of the user-defined services.etc_hosts section is not transmitted into the CoreDNS configmap. If you would like CoreDNS to have these records, put them into the services.coredns.configmap.Hosts section.

coredns

coredns parameter configures the Coredns service and its DNS rules in the Kubernetes cluster. It is divided into the following sections:

add_etc_hosts_generated

This is a boolean parameter defining whether IP addresses of the cluster nodes and their generated domain names should be added to services.coredns.configmap.Hosts. Default is true.

Warning: The add_etc_hosts_generated: true parameter doesn't add the user-defined hosts from services.etc_hosts to the CoreDNS configmap; if necessary, they must be added explicitly to services.coredns.configmap.Hosts.

configmap

This section contains the Configmap parameters that are applied to the Coredns service. By default, the following configs are used:

• Corefile - The main Coredns config, which is converted into a template in accordance with the specified parameters.
• Hosts - IP addresses and names in the format of /etc/hosts file. Can be customized with any desired IP addresses and names. Default value is

127.0.0.1 localhost localhost.localdomain
::1 localhost localhost.localdomain
'

If services.coredns.add_etc_hosts_generated is set to true, Hosts is enriched with generated list of IP addresses and names of the cluster nodes.

If Hosts is redefined in the cluster.yaml its default value is overridden.

Before working with the Corefile, refer to the Official Coredns plugins documentation at https://coredns.io/plugins/.

The Corefile consists of the settings applied for a specific destination. By default, all settings are applied for .:53 destination. For example:

services:
  coredns:
    configmap:
      Corefile:
        '.:53':
          errors: True
          rewrite: # Not used by default, intended for GEO distributed scheme
            default:
              priority: 1
              data:
                name:
                - regex
                - (.*)\.cluster-1\.local {1}.cluster.local
                answer:
                - name
                - (.*)\.cluster\.local {1}.cluster-1.local

The following settings are supported:

Note:

• Some settings allow multiple entries. For example, template. If necessary, different priority can be set for different template sections. For example:

          template:
            test-example-com:
              enabled: true
              priority: 1
              class: IN
              type: A
              zone: 'test.example.com'
              data:
                answer: '{% raw %}{{ .Name }}{% endraw %} 100 IN A 1.1.1.1'
            example-com:
              priority: 2
              enabled: true
              class: IN
              type: A
              zone: 'example.com'
              data:
                answer: '{% raw %}{{ .Name }}{% endraw %} 100 IN A 2.2.2.2'

The lesser the priority value, the earlier in Corefile config this entry appears. Default priority is 1.

• DNS resolving is done according to the hardcoded plugin chain. This specifies that a query goes through template, then through hosts, then through kubernetes, and then through forward. By default, Corefile contains the template setting, which resolves all names like *.{{ cluster_name }} in the vIP address. Hence despite entries in Hosts, such names are resolved in the vIP address.
• You can set any setting parameter to False to disable it, no matter what type it is.
• It is possible to specify other Corefile settings in an inventory-like format. However, this is risky since the settings have not been tested with the generator. All non-supported settings have a lower priority.

Warning: It is strongly discouraged to change the configuration of the CoreDNS manually, if you need to change the configuration, you must reflect them in the cluster.yaml and call the installation procedure with --tasks="deploy.coredns" argument. This will help keep the cluster configuration consistent.

queries logs

The incomming queries could be reflected in the logs. For instance, the cluster-1.local zone queries:

services:
  coredns:
    configmap:
      Corefile:
        'cluster-1.local':
          log: true

More information could be found in coredns documention

errors logs

Error messages are managed by errors plugin. By default it doesn't have any additional settings:

services:
  coredns:
    configmap:
      Corefile:
        '.:53':
          errors: True

deployment

This section contains YAML settings that are applied to Coredns service via a patch. By default, this section contains the following data:

services:
  coredns:
    deployment:
      spec:
        template:
          spec:
            volumes:
            - configMap:
                defaultMode: 420
                items:
                - key: Corefile
                  path: Corefile
                - key: Hosts
                  path: Hosts
                name: coredns
              name: config-volume

However, it is possible to add or modify any deployment parameters of the inventory in accordance with the Kubernetes patching syntax.

loadbalancer

loadbalancer configures the balancers for the Kubernetes cluster. Currently, only the Haproxy and Keepalived configuration can be customized.

target_ports

This section describes the ports used for http/https connections from balancer nodes to workers.

• target_ports.http / target_ports.https — nginx ingress hostPorts on workers and HAProxy backend ports when target_backend is nginx.
• target_ports.envoy_http / target_ports.envoy_https — Envoy Gateway hostPorts on workers and HAProxy backend ports when target_backend is envoy.

Both ingress plugins can be installed at the same time; each binds its own port pair. HAProxy forwards traffic to the pair selected by target_backend.

Default values depend on balancer availability:

If it is needed, those parameters can be overriden in cluster.yaml, e.g.:

services:
  loadbalancer:
    target_ports:
      http: 80
      https: 443
      envoy_http: 21080
      envoy_https: 21443

Note: if you change target_ports, make sure you run install procedure with both below tasks for changes to take effect:

• deploy.loadbalancer.haproxy.configure
• deploy.plugins

target_backend

This section selects which ingress stack receives traffic from HAProxy: nginx or envoy. Default is nginx if nginx-ingress-controller is installed, otherwise envoy. If you do not install nginx and envoy plugins, this option simply selects target_ports pair.

Note: Envoy Gateway uses Gateway API resources (HTTPRoutes, etc) instead of Ingresses. Thus, when you migrate cluster from Ingress-NGINX to Envoy Gateway, your applications should also migrate from Ingresses to Gateway API resources, otherwise routing will not work.

You can specify this option in cluster.yaml as following:

services:
  loadbalancer:
    target_backend: "envoy"

To switch traffic between nginx and envoy without redeploying ingress plugins:

1. Set services.loadbalancer.target_backend to "nginx" or "envoy" in cluster.yaml.
2. Run kubemarine install --tasks=deploy.loadbalancer.haproxy.configure.

haproxy

This section describes the configuration parameters that are applied to the haproxy.cfg config file, and also some Kubemarine related parameters. By default, the following configuration is used:

services:
  loadbalancer:
    haproxy:
      global:
        maxconn: 10000
      defaults:
        timeout_connect: '10s'
        timeout_client: '1m'
        timeout_server: '1m'
        timeout_tunnel: '60m'
        timeout_client_fin: '1m'
        maxconn: 10000
      keep_configs_updated: True

These settings can be overrided in the cluster.yaml. Currently, the following settings of haproxy.cfg are supported:

For more information on Haproxy-related parameters, refer to the Official Haproxy documentation at https://www.haproxy.org/download/1.8/doc/configuration.txt.

Note: you can use either config or config_file if you need to use custom config instead of default.

Parameter config allows to specify your custom config file. The priority of this option is higher than that of config_file, and if both are specified, config will be used. Example:

services:
  loadbalancer:
    haproxy:
      keep_configs_updated: True
      config: "global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http"

Parameter config_file allows to specify path to Jinja-compiled template. Example:

services:
  loadbalancer:
    haproxy:
      keep_configs_updated: True
      config_file: '/root/my_haproxy_config.cfg.j2'

This parameter use the following context options for template rendering:

• nodes
• bindings
• config_options
• target_ports

As an example of a template, you can look at default template.

keepalived

This section describes the configuration parameters that are applied to the keepalived.conf config file in addition to those related to vrrp ips. By default, the following configuration is used:

services:
  loadbalancer:
    keepalived:
      global: {}

These settings can be overrided in the cluster.yaml. Currently, the following settings of keepalived.conf are supported:

Note: you can use either config or config_file if you need to use custom config instead of default.

Parameter config allows to specify your custom config file. The priority of this option is higher than that of config_file, and if both are specified, config will be used. Example:

services:
  loadbalancer:
    keepalived:
      keep_configs_updated: True
      config: |
        global_defs {
          vrrp_garp_master_refresh 60
          vrrp_garp_master_refresh_repeat 10
          vrrp_garp_interval 1
        }
        vrrp_script script_27a2eb32e5 {
          script       "/usr/local/bin/check_haproxy.sh"
          interval 2
          fall 2
          rise 2
        }

        vrrp_instance balancer_27a2eb32e5 {
          state BACKUP
          interface enp0s8
          virtual_router_id 130
          priority 254
          nopreempt
          virtual_ipaddress {
            10.0.2.2 dev enp0s8 label vip_27a2eb32e5
          }
        
          track_script {
            script_27a2eb32e5
          }
        
          authentication {
            auth_type PASS
            auth_pass 6f3a13e1
          }
        }

Parameter config_file allows to specify path to Jinja-compiled template. Example:

services:
  loadbalancer:
    keepalived:
      keep_configs_updated: True
      config_file: '/root/my_keepalived_config.conf.j2'

This parameter use the following context options for template rendering:

• globals (values from 'services.loadbalancer.keepalived.global`);
• modified vrrp_ips with following properties for every of them:

As an example of a template, you can look at default template.

maintenance mode

Kubemarine supports maintenance mode for HAproxy balancer. HAproxy balancer has additional configuration file for that purpose. The following configuration enable maintenance mode for balancer:

services:
  loadbalancer:
    haproxy:
      maintenance_mode: True
      mntc_config_location: '/etc/haproxy/haproxy_mntc_v1.cfg'

patches

It is possible to override the resulting configuration for specific nodes using patches. This allows to either override default parameters for specific nodes, or provide different settings for different nodes.

For example:

patches:
  - groups: [control-plane, worker]
    services:
      sysctl:
        net.ipv4.conf.default.arp_ignore: 0
  - groups: [balancer]
    services:
      sysctl:
        net.ipv4.conf.default.arp_ignore: 2

The patches in the list are merged with the global configuration one by one. Thus, the same settings have precedence in the last patch if overridden few times for the same node.

The following settings are supported:

Note: New patches can be appended after installation using Reconfigure Procedure.

RBAC pss

Installation task: deploy.kubernetes.init

There is currently one supported Pod Security implementation. Pod Security Standards (PSS) are implemented as a feature gate of kube-apiserver. This is reflected in the cluster.yaml as follows:

rbac:
  admission: pss

Originally PSS assumes only three levels (or profiles) of policies. The profiles are the following:

• Privileged - Unrestricted policy, providing the widest possible level of permissions. This policy allows for known privilege escalations.
• Baseline - Minimally restrictive policy which prevents known privilege escalations. Allows the default (minimally specified) Pod configuration.
• Restricted - Heavily restricted policy, following current Pod hardening best practices.

There are plenty of rules that included in baseline and restricted profiles. For more information, refer to Pod Security Standards.

Configuring Default Profiles

The following configuration is default for PSS:

rbac:
  admission: pss
  pss:
    pod-security: enabled
    defaults:
      enforce: baseline
      enforce-version: latest
      audit: baseline
      audit-version: latest
      warn: baseline
      warn-version: latest
    exemptions:
      usernames: []
      runtimeClasses: []
      namespaces: ["kube-system"]

There are three parts of PSS configuration:

• pod-security enables or disables the PSS installation.
• The default profile is described in the defaults section and enforce defines the policy standard that enforces the pods.
• exemptions describes the exemptions from default rules.

PSS enabling requires special labels for plugin namespaces such as nginx-ingress-controller, envoy-gateway, kubernetes-dashboard, local-path-provisioner, and calico (relevant only for calico-apiserver namespace). For instance:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: privileged
    pod-security.kubernetes.io/warn-version: latest

In case of enabling predefined plugins the labels will be set during the installation procedure automatically.

Warnings: Pay attention to the fact that PSS is implicitly enabled by default (that is reflected in kube-apiserver Feature Gates prior to Kubernetes v1.28). PSS labels on namespaces shouldn't be set if your Kubernetes cluster is deployed without PSS enabled.

Configuring Exemptions

The exemption section describes objects that are not enforced by the policy. It is possible to define User or ServiceAccount in the username section. For example, ("system:serviceaccount:my-ns:myadmin" - it is a serviceAccount, "myuser" - it is a user):

...
    exemptions:
      usernames: ["system:serviceaccount:my-ns:myadmin", "myuser"]

In this case, kube-apiserver does not enforce the default policy to any pods that are created by myuser or myadmin.

The default configuration does not enforce the default policy to any of the pods in the kube-system namespace.

...
    exemptions:
      namespaces: ["kube-system"]

Do not change the namespaces exemption list without strong necessary. In any case, check our Maintenance Guide before any implementation.

Application Prerequisites

In case of using PSS, the application that installed in Kubernetes cluster should be matched with PSS profiles (privileged, baseline, restricted). Those profiles may be set by labeling the namespace so as it described above for predefined plugins. Moreover, the application should be compatible with PSS. The restricted profile requires the following section in pod description:

...
securityContext: 
  runAsNonRoot: true
  seccompProfile: 
    type: "RuntimeDefault"
  allowPrivilegeEscalation: false
  capabilities: 
    drop: ["ALL"]
...

RBAC Accounts

Installation task: deploy.accounts

In the deploy.accounts section, you can specify the account creation settings after the cluster is installed.

RBAC account_defaults

In this section, you can describe any parameters that needs to be applied by default to each record in the RBAC accounts section. It works the same way as node_defaults.

The default settings for account_defaults are as follows:

rbac:
  account_defaults:
    namespace: kube-system
    configs:
      - apiVersion: v1
        kind: ServiceAccount
        metadata: {}
      - apiVersion: rbac.authorization.k8s.io/v1
        kind: ClusterRoleBinding
        metadata: {}
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: ClusterRole
        subjects:
          - kind: ServiceAccount
      - apiVersion: v1
        kind: Secret
        metadata:
          annotations: {}
        type: kubernetes.io/service-account-token

The yaml file that is created from the above template is applied to the cluster during the installation procedure.

RBAC Authenticated Issuer Discovery

Installation task: deploy.accounts

To configure service account issuer discovery authentication you can use following flag:

rbac:
  authenticated-issuer-discovery: false

By default, KubeMarine sets this flag to false, which allows unauthenticated access to service account issuer discovery endpoint, usually found at https://<cluster_ip>:6443/.well-known/openid-configuration/.

This is different from default K8s behavior, where this endpoint requires authentication. We disable authentication by default, because some applications do not support authentication on OIDC discovery endpoints. For more information see https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-issuer-discovery

Plugins

Installation task: deploy.plugins

In the plugins section, you can configure the parameters of plugins, as well as register your own plugins. Plugins are installed during the deploy.plugins task. If you skip the plugin installation task, no plugins are installed.

Predefined Plugins

When you want to install a plugin, the installer includes pre-configured plug-in configurations. The following plugins are available for installation out of the box:

• Network plugins
  • calico
• Ingress Controllers
  • nginx-ingress-controller
  • envoy-gateway
• kubernetes-dashboard
• local-path-provisioner

Note: It is not possible to install multiple plugins of the same type at the same time.

calico

Before proceeding, refer to the Official Documentation of the Kubernetes Cluster Network at https://kubernetes.io/docs/concepts/cluster-administration/networking/.

Calico plugin is installed by default and does not require any special enablement or configuration. However, it is possible to explicitly enable or disable the installation of this plugin through the install plugin parameter.

The following is an example to enable the calico plugin:

plugins:
  calico:
    install: true

The following is an example to disable the calico plugin:

plugins:
  calico:
    install: false

After applying the plugin configurations, the plugin installation procedure waits for the following pods to be in the Running state:

• coredns
• calico-kube-controllers
• calico-node
• calico-apiserver

If the pods do not have time to start at a specific timeout, then the plugin configuration is incorrect. In this case, the installation is aborted.

By default, no additional settings are required for the plugin. However, you can change the default settings. To do this, in the plugins section of the config file, specify the calico plugin section and list all the necessary parameters and their values in it. For example:

plugins:
  calico:
    install: true
    mtu: 1400
    typha:
      enabled: true
      nodeSelector:
        region: infra
    node:
      image: calico/node:v3.10.1
    env:
      FELIX_USAGEREPORTINGENABLED: 'true'

An example is also available in Full Inventory Example.

Calico BGP Configuration

By default, calico is installed with "full mesh" BGP topology, that is, every node has BGP peering with all other nodes in the cluster. If the cluster size is more than 50 nodes it is recommended to use the BGP configuration with route reflectors instead of full mesh. You also have to change calico BGP configuration if you are using DR schema.

Note: change BGP topology is possible for calico v3.20.1 or higher.

To enable route reflector and/or DR topology during installation the next steps are required:

1. Choose the nodes to be route reflectors and add the label route-reflector: True to their description in the cluster.yaml. It is recommended to use control-plane nodes for route reflectors, but not necessarily.

2. Add required parameters in the calico plugin section:

plugins:
  calico:
    fullmesh: false            # Full mesh will not be used, RRs will be used instead
    announceServices: true     # ClusterIP services CIDR will be announced through BGP
    defaultAsNumber: 65200     # AS Number will be 65200 for all nodes by default
    globalBgpPeers:            # Additional global BGP Peer(s) will be configured with given IP and AS Number
    - ip: 192.168.17.1
      as: 65200

It is also possible to change BGP topology at the running cluster.

Warning: short downtime is possible during BGP peering sessions reestablishing.

To switch from "full mesh" to "route reflector" topology:

• add the label route-reflector: True to the route reflector nodes manually:

$ kubectl label node <NODENAME> route-reflector=True

• add fullmesh: false parameter to the calico plugin section in the cluster.yaml
• run kubemarine install with the deploy.plugins task only. Other plugins should have install: false in the cluster.yaml at this step.

Note: for the topology with route reflectors the predefined value routeReflectorClusterID=244.0.0.1 is used.

To switch from "route reflector" to "full mesh" topology:

• change fullmesh parameter value to true in the calico plugin section in the cluster.yaml (it also may be removed so the default value of fullmesh is being used)
• run kubemarine install with the deploy.plugins task only. Other plugins should have install: false in the cluster.yaml at this step
• remove the labels route-reflector: true from the route reflector nodes manually:

$ kubectl label node <NODENAME> route-reflector-

If necessary, remove route-reflector label from the cluster.yaml as well.

Warning: For correct network communication, it is important to set the correct MTU value (For example in case ipip mode it should be 20 bytes less than MTU NIC size), see more details in Troubleshooting Guide.

Note: If the cluster size is more than 3 nodes, Calico Typha daemon is enabled by default and number of its replicas is incremented with every 50 nodes. This behavior can be overridden with cluster.yaml.

The plugin configuration supports the following parameters:

Default Typha Tolerations

- key: CriticalAddonsOnly
  operator: Exists
- key: node.kubernetes.io/network-unavailable
  effect: NoSchedule
- key: node.kubernetes.io/network-unavailable
  effect: NoExecute

Note: The CriticalAddonsOnly toleration key inherits from Calico manifest YAML, whereas the rest of toleration keys are represented by Kubemarine itself.

Calico Metrics Configuration

By default, no additional settings are required for Calico metrics. They are enabled by default.

Note: The following ports are opened for metrics: calico-node : 9091, calico-kube-controllers : 9094, and calico-typha: 9093. The ports for calico-node and calico-typha are opened on the host. These ports are currently not configurable by means of Kubemarine.

Note: If you use prometheus-operator or its resources in your monitoring solution, you can use the following ServiceMonitors to collect metrics from Calico:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    # If you use prometheus-operator,
    # specify the custom label corresponding to the configuration of your Prometheus.serviceMonitorSelector.
    # The same is for the other ServiceMonitors below.
    app.kubernetes.io/component: monitoring
  name: calico-metrics
  namespace: kube-system
spec:
  endpoints:
    # You can configure custom scraping properties, but leave `port: metrics`.
    # The same is for the other ServiceMonitors below.
    - interval: 30s
      port: metrics
      scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      k8s-app: calico-node
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: monitoring
  name: calico-kube-controllers-metrics
  namespace: kube-system
spec:
  endpoints:
    - interval: 30s
      port: metrics
      scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: monitoring
  name: calico-typha-metrics
  namespace: kube-system
spec:
  endpoints:
    - interval: 30s
      port: metrics
      scrapeTimeout: 10s
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      k8s-app: calico-typha-metrics

Calico Environment Properties

It is possible to change the default Calico environment properties. To do that, it is required to specify a key-value in the env section in the calico plugin definition. For example:

plugins:
  calico:
    env:
      WAIT_FOR_DATASTORE: 'false'
      FELIX_DEFAULTENDPOINTTOHOSTACTION: DENY

Note: In case of you use IPv6, you have to define CALICO_ROUTER_ID with value hash in env section. This uses a hash of the configured nodename for the router ID.

For more information about the supported Calico environment variables, refer to the Official Calico Documentation at https://docs.projectcalico.org/reference/node/configuration.

Calico API server

For details about the Calico API server, refer to the Official Documentation at https://docs.tigera.io/calico/latest/operations/install-apiserver.

By default, the Calico API server is not installed. To install it during the Calico installation, specify the following:

plugins:
  calico:
    apiserver:
      enabled: true