Node configuration backup
Before upgrading or reinstalling a node, take a copy of its configuration and
state. The vstackos.config-backup.sh script does this in a single run, in a
few seconds, and changes nothing on the node.
The script is provided by vStack technical support on request. It is not included in the release distribution.
This is not a data backup. Pool data, VM disks, and ZFS snapshots are not included in the collection. The script collects the node configuration, that is, what the node is rebuilt from or returned to its previous state with.
Why a separate collection if the upgrade makes a copy itself
Before upgrading, vStack.update.sh saves /etc, /usr/local/etc,
vstack-env.conf, and the boot profile: /boot/loader.conf,
/boot/loader.conf.local, and /boot/device.hints. After the upgrade, it
restores accounts, ssh, several system files, and these three boot
files. This is described in detail in step 3 of the
"Node upgrade to 3.2" page.
This copy has two weaknesses. It is stored in /var/backups on the node system disk,
and is lost together with it on reinstallation. And it does not cover part of the node
state:
| Not covered by the upgrade copy | What this risks |
|---|---|
| cluster layer configuration and database | nothing to restore the node's cluster membership and service layout from |
| disk partitioning, system disk mirror, EFI partitions | nothing to reproduce the boot disk from on reinstallation |
| disk export composition (LUNs, targets, initiators) | on some versions, the composition file is zeroed when services stop, and the composition is lost with it |
| descriptions of the node's VMs | nothing to restore the machine configuration from |
| package list and repository priorities | nothing to reproduce the same build from on rollback |
When to run
Before stopping the cluster layer and exporting pools. After they stop, the pools are not mounted, and there is nowhere left to take the export composition from. The script will notice this and warn you, but will not be able to capture the composition.
You need root privileges on the node and 256 MB of free space in the output directory. A
collection usually takes 5–8 MB. No maintenance window is needed: the script only reads and writes
to its own directory.
Each node has its own collection. Do not mix collections from different nodes. The node
identifier, vstack-env.conf, and the cluster layer database are specific to each node, and if you
mix them up, the cluster will break.
Step 1. Run on the node
sh vstackos.config-backup.sh --tag "3.1.0 -> 3.2.0"
The same without copying the script to the node:
ssh root@q100u001 sh -s -- --tag "3.1.0 -> 3.2.0" < vstackos.config-backup.sh
The collection goes to /root/vstackos-config-backup/<cluster>/<date-UTC>/<node>/;
the directory is created with permissions 700. Inside, it is split into sections by topic: node identity,
packages, configuration, storage, disks, cluster, export, VMs, network, runtime.
The script determines the cluster name itself and prints it at startup. First it takes
the node name prefix of the form <cluster>u<NNN> (for q100u001 this is q100), then
cluster_id from vstack-env.conf, and as a last resort the full node name. If the name
comes out wrong, set the directory with the --out option.
| Option | When you need it |
|---|---|
--dry-run | show the collection plan and write nothing |
--tag TEXT | a label in the report: target version, ticket number |
-o, --out DIR | a different directory, for example for a repeat "after" collection |
--copy-to DIR | immediately duplicate the collection to mounted media and verify the copy against checksums. The option can be specified several times |
--tar | additionally pack the collection into vstackos-config-backup-<node>-<date>.tar.gz next to the directory |
--verify DIR | verify a finished copy against its manifest and exit |
--no-secrets | do not put master.passwd, spwd.db, and the node's private ssh keys into the copy |
--with-logs | add the tails of /var/log |
Exit codes: 0 — the collection is clean, 2 — collected, but there are warnings,
1 — the script did not start.
Step 2. Read the collection report
cat <out>/00-summary/SUMMARY.txt # node profile, collection size, check results
cat <out>/00-summary/checks.txt # PASS/FAIL for critical artifacts
cat <out>/00-summary/warnings.txt # what was not collected
FAIL on the cluster layer database, vstack-env.conf, disk partitioning, or
VM descriptions means stop. Do not start the upgrade until you find out
why the artifact was not captured.
Pay separate attention to the warning that the saved export composition is empty while the subsystems are alive at runtime. It means the composition file has already been zeroed, and the only copy of the export is the runtime dump inside the collection.
Step 3. Put the collection in three places
By default, the collection is stored on the system disk, the very disk that the upgrade overwrites and that is lost on reinstallation. This one copy is not enough.
| # | Where | How | Who does it |
|---|---|---|---|
| 1 | on the node | output directory | script |
| 2 | second media: the node boot USB drive, another pool, a mounted network share | --copy-to <mount point>; the script copies and verifies against checksums | script |
| 3 | off the node: the administrator's workstation | a tar stream, see below | administrator |
At the end of the run, the script prints all three lines marked [done] or
[NOT done]. The third is always NOT done, because only the administrator can
move the collection off the node.
USB drive
Writing the installation image with dd wipes everything that was on the media.
Therefore, put the collection there after writing the image.
gpart show da0 # find the media partition
mount -t msdosfs /dev/da0s1 /mnt # or -t ufs /dev/da0p2 for the memstick image
sh vstackos.config-backup.sh --copy-to /mnt
umount /mnt
The media must be case-sensitive for file names. The node has files
whose names differ only in case. A FAT USB drive, a Windows disk, and the default
macOS volume treat such names as one file, and a copy there silently loses one
file of each pair. The script checks the media before copying and refuses to write
to a case-insensitive one. On macOS and Windows, store the collection
as an archive (--tar or the tar stream below) and unpack it where case
is distinguished.
Moving off the node: a tar stream, not scp -r
The collection contains about a thousand small files. scp -r transfers them one by one, and on
a link with latency this stretches to hours. A tar stream handles it in
seconds.
D=/root/vstackos-config-backup/<cluster>/<date>
# the receiver is case-sensitive: straight into a directory
ssh root@q100u001 "tar czf - -C $D q100u001" | tar xzf -
# macOS or Windows: store as an archive, without unpacking
ssh root@q100u001 "tar czf - -C $D q100u001" > vstackos-config-backup-q100u001-<date>.tar.gz
Off-node storage layout
It is most convenient to repeat the node layout; then the collection is moved by simple copying:
<storage>/<cluster>/<date>/vstackos-config-backup-<node>-<date>.tar.gz
CHECKSUMS.sha256
Verifying a copy
Any copy is verified against its own manifest. This is the only script mode
that does not need root: run it where the copy is stored. The script chooses the hashing
utility (sha256, sha256sum, or shasum) based on what is
available on the machine.
sh vstackos.config-backup.sh --verify <copy-dir>
# verified NNN files ... mismatched 0 -> the copy is intact (code 0)
# MISMATCH ./30-config/etc/rc.conf -> the copy is corrupted (code 2)
The manifest covers the collected material. The report files of the run itself are intentionally not included in it.
Step 4. Repeat on each node
The cluster is upgraded one node at a time, and the collection is also taken node by node. The node name is the last directory level, so collections from different nodes do not collide either on the USB drive or in shared storage.
After the upgrade: the "after" collection and comparison
ssh root@q100u001 sh -s -- \
--out /root/vstackos-config-backup/<cluster>/<date>-after/q100u001 \
--tag after-3.2.0 < vstackos.config-backup.sh
diff -r <before>/30-config <after>/30-config | head -50
diff <before>/20-packages/pkg-versions.txt <after>/20-packages/pkg-versions.txt
First of all, look at the *.pkgnew and *.pkgsave files in the configuration,
missing packages, diverged tuning parameters, and the export composition.
Compare tuning by the effective values, not by the file:
sysctl -n <variable>