SONiC Testbed Server OS Upgrade

Introduction

This guide covers the safe, standard procedure to upgrade the Ubuntu LTS version of a testbed server (the host machine that runs the SONiC testbed VMs and topology). It includes prerequisites, step-by-step upgrade instructions, and the post-upgrade configuration checks that are required to keep the testbed fully functional (NIC port naming, netplan, and MTU).

Note: This procedure applies to the testbed server/host, not to the SONiC DUT. Drain or pause any running tests and back up critical configuration before you start.

Supported Upgrade Paths

Ubuntu supports direct upgrades between adjacent LTS (Long-Term Support) versions:

From	To	Supported?
18.04 LTS	20.04 LTS	Yes
20.04 LTS	22.04 LTS	Yes

Pre-Upgrade Checklist

System requirements:

1. The server is connected to the internet.
2. Enough free disk space (at least 2-4 GB free).
3. sudo/root privileges.

Account & permission model (read first). Run every step in this guide as the same regular Linux account that is a member of the sudo group, using sudo for privileged commands — not as root directly. During the upgrade you may still be prompted for the sudo password; passwordless sudo (NOPASSWD, restored in post-upgrade check (a)) is what the testbed automation needs afterwards, and is not required to perform the upgrade itself. Keeping one consistent account throughout is why the config prompts above recommend preserving host-specific sudoers/SSH files: it ensures the restored permission model matches what the automation expects.

Upgrade Steps

1. Fully update the current system

sudo apt update
sudo apt upgrade -y
sudo apt dist-upgrade -y
sudo apt autoremove -y

If you are upgrading from 18.04 to 20.04, make sure lxd is uninstalled first:

sudo apt remove --purge lxd lxd-client

2. Check the upgrade tool

Ensure the upgrade manager is installed:

sudo apt install update-manager-core

Check the LTS prompt setting:

sudo cat /etc/update-manager/release-upgrades

Ensure it contains:

Prompt=lts

3. Pre-upgrade checks

Back up the 100G port renaming rules (and other host-specific config). A release upgrade can remove or overwrite these files, so capture their actual contents — not just a directory listing — before you start, so the recovery steps have something to restore from:

# Copy the real udev rule files (preserving names/permissions) into a backup dir
sudo mkdir -p ~/testbed-preupgrade-backup/udev-rules.d
sudo cp -a /etc/udev/rules.d/. ~/testbed-preupgrade-backup/udev-rules.d/

# Optional: also print them so the exact rules are recorded in your session log
for f in /etc/udev/rules.d/*.rules; do echo "== $f =="; sudo cat "$f"; done

It is good practice to back up the other host-specific files the same way (so they can be diffed/restored after the upgrade), for example /etc/netplan/, /etc/sudoers and /etc/sudoers.d/, and /etc/ssh/sshd_config.

4. Start the upgrade

sudo do-release-upgrade

• For any prompt to update a package or restart a service, choose yes.
• For any prompt asking whether to keep the currently-installed config file or install the package maintainer's version, decide per file instead of applying one answer to everything. At the prompt: N/O keeps your current file (this is the default), Y/I installs the maintainer's version, and D shows the diff between the two.
  • Host-specific / testbed config — e.g. sudoers and /etc/sudoers.d/*, SSH (/etc/ssh/sshd_config), netplan, and the udev net-naming rules (/etc/udev/rules.d/*): keep your currently-installed version (N), or press D and merge the changes intentionally. Blindly taking the maintainer's version here can overwrite host-specific settings and break testbed automation or remote recovery after the reboot. Restore anything you lose from the pre-upgrade backup.
  • Generic config you have not customized: prefer the maintainer's version (Y) so you pick up upstream fixes — keeping a stale local copy of a security-relevant file (SSH/PAM defaults, etc.) can leave a known vulnerability unpatched.
  • When unsure, press D to review the diff first, then choose based on whether the file carries host-specific customizations.

5. Reboot and verify the version

The upgrade usually requests a reboot at the end. If it does not, reboot manually:

sudo reboot

After reboot, verify the version:

lsb_release -a

Post-Upgrade Configuration Checks

A release upgrade can reset NIC naming, netplan, and MTU settings. The following checks restore testbed connectivity.

a) Ensure passwordless sudo (NOPASSWD)

The testbed automation requires members of the sudo group to run commands without a password prompt. Make sure the following line is present in the sudoers configuration:

%sudo ALL=(ALL:ALL) NOPASSWD:ALL

Recommended method - edit safely with syntax checking using visudo:

sudo visudo

Look for:

%sudo ALL=(ALL:ALL) ALL

Either replace it, or comment it out and add the NOPASSWD line:

#%sudo ALL=(ALL:ALL) ALL
%sudo ALL=(ALL:ALL) NOPASSWD:ALL

Verify (this should not prompt for a password):

sudo -k sudo ls

b) Confirm 100G port renaming

Check whether the 100G port renaming rule is still present after the upgrade:

sudo ls /etc/udev/rules.d

If the rule is missing, the quickest fix is to restore it from the pre-upgrade backup:

sudo cp -a ~/testbed-preupgrade-backup/udev-rules.d/. /etc/udev/rules.d/
sudo reboot

If you do not have a backup, re-create the rule manually. For a Mellanox 100G NIC:

1. Get the PCI IDs:

   lspci -D | grep -i ethernet | grep -i Mellanox
   

   Example output:

   0000:17:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
   0000:17:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
   

2. Add the following lines to /etc/udev/rules.d/70-persistent-net.rules (adjust the PCI IDs to match your server):

   ACTION=="add", SUBSYSTEM=="net", KERNELS=="0000:17:00.0", NAME:="ens4f0"
   ACTION=="add", SUBSYSTEM=="net", KERNELS=="0000:17:00.1", NAME:="ens4f1"
   

3. Reboot the server.

4. Verify the rename succeeded:

   lspci -D | grep -i ethernet | grep -i Mellanox
   ls -la /sys/class/net/ | grep -i 0000:17:00.
   

   Example output:

   ens4f0 -> ../../devices/pci0000:16/0000:16:02.0/0000:17:00.0/net/ens4f0
   ens4f1 -> ../../devices/pci0000:16/0000:16:02.0/0000:17:00.1/net/ens4f1
   

c) Confirm netplan configuration for external_port (defined in ansible/host_vars/SERVER.yml, take ens4f0 / ens4f1 as example)

1) Check whether ens4f0 / ens4f1 exist:

ip a | egrep "ens4f0|ens4f1"
# or
ip link show ens4f0
ip link show ens4f1

If both interfaces are listed, you can stop here. If not, continue.

2) Find the physical NIC names and MAC addresses:

ip link show
# or, for a concise list:
for i in /sys/class/net/*; do printf "%-20s %s\n" "$(basename $i)" "$(cat $i/address)"; done

3) Identify which netplan YAML file to edit:

ls -l /etc/netplan/
cat /etc/netplan/<active-file>.yaml

The active file may be 00-installer-config.yaml, 01-netcfg.yaml, 50-cloud-init.yaml, etc. Use whichever exists on the host.

4) Edit the active netplan YAML:

Add the interfaces under ethernets: (2-space indentation), for example:

network:
  version: 2
  ethernets:
    ens4f0:
      dhcp4: no
    ens4f1:
      dhcp4: no

Save (Ctrl+O) and exit (Ctrl+X).

5) Apply safely:

sudo netplan try   # if the network stays stable, press ENTER to confirm before timeout
sudo netplan apply

6) Verify interfaces and link state:

ip a | egrep "ens4f0|ens4f1"
ip link show ens4f0
ip link show ens4f1

d) Verify interface MTU

On some servers, after the upgrade the DUT stops receiving BGP updates from the EOS container because the OVS rules on the server no longer forward BGP updates correctly. The root cause is the upgraded server port MTU - it should be 9216.

Check and fix the MTU:

# Check the current MTU (shows "mtu 1500" if it is wrong)
ifconfig ens4f1

# Set the correct MTU
ip link set dev ens4f1 mtu 9216

# Verify (should now show "mtu 9216")
ifconfig ens4f1

Example before/after (truncated):

ens4f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
...
ens4f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9216

To make the MTU persistent across reboots, set it in the netplan configuration for the interface (for example, add mtu: 9216 under ens4f1:).