Node upgrade to 3.2

A node is upgraded with the standard vStack.update.sh script. The script ships with the distribution and is located on the node in /root/bin/. Nodes are upgraded one at a time, and each one is rebooted after the upgrade.

This page is intended for nodes of the 3.0.x and 3.1.x lines that are moving to 3.2.0. Moving from the 2.x line changes the FreeBSD major version and is done together with vStack technical support.

Run all commands as root on the node itself.

Before the window

1. Repository access. Packages are taken from the vStack repository, which accepts only addresses from the allow list. Send technical support the addresses from which the nodes go out in advance. Without this, the script exits with code 2.
2. Node console. You need access to IPMI or BMC in case the node does not come back over the network.
3. The workload is removed from the node. VMs are shut down or moved, and the node's pools are handed over to other cluster nodes. How to do this is described in the "Moving pool services" section. If VMs in the cluster also run on nodes that do not hold their pool, the procedure is described in the "Freeing a node with remotely started VMs" section.
4. Time is synchronized on all nodes. Check ntpq -p: the time source must be reachable.
5. A node configuration backup is taken. The procedure is described on the "Node configuration backup" page. The upgrade script itself also makes a copy, but keeps it on the node system disk, and on reinstallation it is lost together with the disk.

Cluster traversal order

Upgrade nodes strictly one at a time. Move on to the next node when the previous one has come back, has been accepted by the cluster, and resilver has finished on all cluster pools.

Resilver starts on its own. While a node reboots, the pools of the other nodes lose its disks, and when the node comes back, ZFS catches up on the changes on them. If you reboot the next node earlier, a pool whose synchronization has not finished yet will lose a second disk. A pool with N+1 redundancy does not survive this.

Check on each node that holds pools:

zpool status | grep -E 'pool:|state:|scan:'

You can proceed when all pools have state: ONLINE and the scan: line does not contain resilver in progress.

Two more rules for the duration of the traversal:

• A cluster with mixed versions is a normal operating state while the traversal is in progress.
• Do not return a pool that has already run on an upgraded node to a node that has not been upgraded. Keep pools on upgraded nodes until you finish the traversal.

caution

Do not run zpool upgrade either during the traversal or after it. The upgrade script reminds you of this in the final summary.

Step 1. Update the script itself

The script is released together with the release and is signed with the same key as the packages. Take the version from the distribution you are upgrading to.

fetch -o /root/bin/vStack.update.sh https://repo.vstack.com/os/3.2.0/update.sh
fetch -o /tmp/update.sh.sig        https://repo.vstack.com/os/3.2.0/update.sh.sig
openssl dgst -sha256 \
  -verify /usr/share/keys/vstack/vstackos.pub \
  -signature /tmp/update.sh.sig /root/bin/vStack.update.sh
chmod 0755 /root/bin/vStack.update.sh

The check must print Verified OK. If it prints anything else, do not start the upgrade.

Step 2. Dry run

A dry run changes nothing on the node. The script writes a temporary repository config and shows the pkg plan.

/root/bin/vStack.update.sh --controllers --base \
  --source release --version 3.2.0 --dry-run

Make sure the repository is reachable and the plan has no unexpected package removals.

Step 3. Upgrade

/root/bin/vStack.update.sh --controllers --base \
  --source release --version 3.2.0

• --controllers upgrades the node packages, except the base system.
• --base upgrades the base system and the kernel. A reboot is mandatory after it.

There is no point in restarting controllers right after the upgrade if a reboot follows. You can disable this with the --no-restart option. The --reboot option reboots the node on completion. The full list of options is in the reference.

Before the transaction, the script takes a configuration copy to /var/backups/etc-<ts>: /etc, /usr/local/etc, and, in the boot subdirectory, the files /boot/loader.conf, /boot/loader.conf.local, and /boot/device.hints. vstack-env.conf is saved separately. After the upgrade, the script restores from the copy the accounts, ssh settings, sysctl.conf, syslog.conf, hosts, ntp.conf, vstack-env.conf, and the three boot files, that is, the node tuning profile. Other files from the copy are not restored by name; compare them manually if necessary.

Step 4. Pre-reboot checks

1. The exit code is 0. Investigate any other code before rebooting; the meaning of the codes is given below.
2. The root shell is executable. Otherwise, after the reboot sshd will not let root onto the node.

   pw usershow root | awk -F: '{ print $NF }'   # shell path
   ls -l $(pw usershow root | awk -F: '{ print $NF }')

3. Boot protection is armed.

   cat /boot/nextboot.conf                         # one-time boot of the new kernel
   cat /boot/loader.conf.d/zz-vstack-fallback.conf # default boot for the duration of the window: the previous kernel

   Look at the files specifically. The boot summary line in the log prints kernel names with an extra _prev suffix and is misleading. In reality, the new kernel boots once, and rollback goes to the previous one.
4. Run summary. It is stored in /var/backups/update-summary-<ts>.txt, next to the versions-before-*.txt and versions-after-*.txt files. The full log is written to /var/log/vstack-os-update-<ts>.log.

The boot loader is updated on only one system disk

The script copies the new loader.efi to the EFI partition from which the node is currently booted. The EFI partition of the second system disk keeps the old boot loader. If after the upgrade the firmware boots the node from the second disk, the new kernel will be loaded by the old boot loader.

Therefore, do not change the boot order in the firmware during the window, and take this into account when replacing a system disk.

Step 5. Reboot and acceptance

reboot

If the new kernel fails to boot, the next reboot brings the node up on the previous one. A node that hangs at the boot loader prompt will not reboot on its own: reset it from the IPMI or BMC console.

After the node comes up:

uname -r                       # base of the target line
cat /etc/os-release            # VERSION_ID = 3.2.0
pkg query '%n %v' | grep -i vstack
pkg check -d -a                # dependencies are consistent
ls /boot/nextboot.conf /boot/loader.conf.d/zz-vstack-fallback.conf 2>&1

The last command must report that the files do not exist. After a successful boot, the node itself removes the one-time boot and the temporary rollback. If the files remain, the node did not come up on the target kernel. Find the cause before returning the workload.

Then check the node services (service <name> status) and the node state in the cluster, wait for resilver to finish on all pools, and return the workload. The pools themselves do not return to the node that came back; move them manually.

Moving pool services

During the window, you need to move pools twice: remove the pools from the node before the work and return them afterwards.

The usual way is the vStack Center interface, the "Cluster services" section.

The exception is the pool on which the management VM (vStack Center) runs. It cannot be moved from the interface. A move starts by suspending the pool's VMs, and the management VM itself is among them: Center shuts itself down in the middle of the operation, and the move is interrupted. Such a pool is moved with the cluster utility on a node.

The management VM usually lives on the first cluster pool, z01. You can check this by the VM with number 0:

vstack-vm vm list

Moving with the cluster utility

This method is needed for the pool with the management VM and for clusters without Center. Commands are run from any cluster node.

In the cluster, a pool is represented by the zpool_<pool name> service: pool z01 corresponds to the zpool_z01 service. Commands take the service name, not the pool name. Service names and node numbers are shown by vstack-cluster status:

vstack-cluster status

Node ID:  1; name:   q100u001.example.com; state:       ONLINE; [     enabled]
	Service name:       cluster-management; state:    STARTED; [     enabled]
	Service name:                zpool_z01; state:    STARTED; [     enabled]

Node ID:  2; name:   q100u002.example.com; state:       ONLINE; [     enabled]
	Service name:                zpool_z02; state:    STARTED; [     enabled]
...

Moving pool z01 to node 2:

vstack-cluster service move -s zpool_z01 --dst-id=2

The destination node must be ONLINE and open (enabled). The command is described in detail in the reference.

While the pool is being moved, the pool's VMs are shut down, and provisioning of its volumes is interrupted. This is a maintenance window, not a seamless operation.

Freeing a node with remotely started VMs

A VM can run on a node other than the owner of its pool: then its disks are delivered over NVMe-oF/TCP. In the vstack-vm vm list output, such VMs are marked with the word remote. Before an upgrade, the node is freed from the VMs of its pools and from VMs that run on it with disks from other nodes' pools.

Stop and start VMs through vStack Center.

1. Move the node's pools as described in the "Moving pool services" section. Stop the VMs of these pools that run on other nodes before the move, and start them again after the move. You can find them on any cluster node:

   vstack-vm vm list | grep -w remote | grep -w <pool>

2. Move the remaining VMs off the node. Find them on the node itself:

   vstack-vm vm list | grep -w remote

   Stop each one and start it with the "Start on node" action, selecting a different node. Then make sure no VMs are left on the node:

   ls /dev/vmm

   The response No such file or directory means there are no VMs on the node.

3. Close the node to VM placement and check that its status is DISABLED:

   vstack-cluster node disable --id=<N>
   vstack-cluster node status

   If the vstack-cluster utility is not on the node, the same actions are performed by ccrd_cli --node --node_id <N> --command disable and ccrd_cli --node --command status.

   After closing the node, run ls /dev/vmm again. If VMs have appeared on the node, move them off as in step 2.

   caution

   During the work, do not start or move VMs to the closed node. Keep no more than one node closed.

4. Upgrade the node following steps 1–5 on this page.

5. Open the node after acceptance:

   vstack-cluster node enable --id=<N>

   Without the vstack-cluster utility: ccrd_cli --node --node_id <N> --command enable. Then return the pools if necessary. VMs do not return to the node on their own.

Exit codes

Code	Meaning
0	the upgrade is complete
1	an error in the arguments
2	the source or version is unavailable, the signing key was not obtained, or a move between major versions was requested without --allow-major-upgrade
3	a pkg error or a new kernel check error
4	failed to save or restore the configuration (/etc, /usr/local/etc, boot files)
5	an installed service did not restart
6	the upgrade is complete, but the root shell is not executable. Fix before rebooting
7	the upgrade is complete, but the /boot write is not confirmed. Do not reboot until investigated

With codes 6 and 7, the script intentionally ignores --reboot.

Rollback

The script does not roll back packages. You can go back as follows:

• Boot. The previous kernel stays on the node and is available from the boot loader menu until the upgrade window is closed by a successful boot.
• Configuration. The copy is stored in /var/backups/etc-<ts>; the five most recent are kept.
• The whole system. Reinstall from the target version media or restore from a node backup taken before the work.

The script does not save pool data, VM disks, or configuration beyond what is listed in step 3. Make a copy of them yourself before the window; see "Node configuration backup".