Node CLI

On a vStack OS node, the administrator needs two utilities:

the cluster utility vstack-cluster shows cluster nodes and services, enables and disables nodes, and starts, stops, and moves pools;
the upgrade script vStack.update.sh upgrades the packages, base system, and kernel of the node.

Both are run as root on the node itself. Virtual machines, networks, and VDCs are managed by vStack Center; its interface and API are described in the Center documentation.

Cluster utility

vstack-cluster is the Cluster Manager client (the ccrd daemon). It comes to the node with the vStack-ccrd package and is located in /usr/local/bin. The utility has no state of its own: each command is sent to ccrd, and the utility prints its response. You can run it on any cluster node.

Invocation format:

vstack-cluster <COMMAND> [<ACTION>] {OPTIONS}

A node is specified by number (--id, --dst-id), a service by name (-s, --service-name). The utility does not accept a node name instead of a number. The node number is in the range 1–63.

Nodes and services

Node — a cluster server. It has a number (Node ID) and a name of the form q100u001.<domain>.
Service — something Cluster Manager places on one of the nodes. Each storage pool zNN is represented by the zpool_zNN service. The cluster-management service belongs to Cluster Manager itself: do not start, stop, or move it manually.

A node and a service each have two independent attributes:

Attribute	Values	Meaning
`state`	node: `ONLINE`, `OFFLINE`; service: `STARTED`, `STOPPED`, `STOPPING`, and others	the actual state at the moment
`[enabled]` / `[disabled]`	—	whether services may be placed on the node; whether the service takes part in automatic placement and failover

Commands

Command	What it does
`status`	nodes and the services running on them
`node list`	node numbers
`node status`	the state of all nodes
`node disable --id=N`	close the node to service placement
`node enable --id=N`	open the node again
`service list`	service names
`service status`	the state of all services and the nodes they run on
`service start -s <service> --id=N`	start the service on node `N`
`service stop -s <service>`	stop the service
`service move -s <service> --dst-id=N`	move the service to node `N`
`service enable -s <service>`	return the service to automatic placement
`service disable -s <service>`	exclude the service from automatic placement
`ha disk status`	the state of SCSI/NVMe reservations on the disks of the HA fencing group

The node exit command removes a node from the cluster. It cannot be undone, so it is not described here: run it only together with vStack technical support.

Common options

Option	What it does
`-h`, `--help`	help for the current level, for example `vstack-cluster service move --help`
`-v`, `--version`	utility version
`-c <file>`, `--config=<file>`	a different configuration file. Usually not needed

status

Cluster nodes and the services on each of them. With --json, the same content is output in JSON.

vstack-cluster status

Node ID:  1; name:   q100u001.example.com; state:       ONLINE; [     enabled]
	Service name:                zpool_z02; state:    STARTED; [     enabled]

Node ID:  2; name:   q100u002.example.com; state:       ONLINE; [     enabled]
	Service name:       cluster-management; state:    STARTED; [     enabled]
	Service name:                zpool_z03; state:    STARTED; [     enabled]

Node ID:  3; name:   q100u003.example.com; state:       ONLINE; [     enabled]

Node ID:  4; name:   q100u004.example.com; state:       ONLINE; [     enabled]
	Service name:                zpool_z01; state:    STARTED; [     enabled]
	Service name:                zpool_z04; state:    STARTED; [     enabled]

A node with no services at all, like q100u003 in the example, is normal for a node that failed and came back. Cluster Manager moves pools off a failed node but does not return them on its own. You can return a pool with the service move command or from vStack Center.

vstack-cluster status --json

{
   "cluster": {
      "nodes": [
         { "enabled": true, "id": 1, "name": "q100u001.example.com", "state": "ONLINE" }
      ],
      "services": [
         { "enabled": true, "name": "zpool_z02", "nodeId": 1, "state": "STARTED" }
      ]
   }
}

services[].nodeId is the number of the node on which the service runs.

node list, node status

node list prints only node numbers; node status prints numbers, names, and state.

vstack-cluster node status

Node ID:  1; name:   q100u001.example.com; state:     ONLINE; [     enabled]
Node ID:  2; name:   q100u002.example.com; state:     ONLINE; [     enabled]

node disable, node enable

disable closes the node to service placement; enable opens it again.

vstack-cluster node disable --id=2
vstack-cluster node enable --id=2

caution

After disable, services stay on the node, and the command seems to do nothing. But on the next change in cluster composition, for example after any node reboots, Cluster Manager recalculates placement and moves services off all closed nodes. If only one node is left open, all cluster pools will move to it.

Close no more than one node at a time, and open it as soon as you finish the work. To move a pool, use service move, not node disable.

service list, service status

service list prints only service names. service status shows the state of each one and the number of the node it runs on.

vstack-cluster service status

Service name:       cluster-management; state:    STARTED at node:  2; [     enabled]
Service name:                zpool_z01; state:    STARTED at node:  4; [     enabled]
Service name:                zpool_z02; state:    STARTED at node:  1; [     enabled]

You cannot select a single service or a single node in status; filter the output, for example vstack-cluster service status | grep zpool_z01.

service start, service stop

stop stops the service on the node where it runs. start starts a stopped service on the specified node.

vstack-cluster service stop -s zpool_z03
vstack-cluster service start -s zpool_z03 --id=3

Stopping a pool service means exporting the pool: the VMs of this pool stop, and provisioning of its volumes ceases.

service move

Moves a service to another node. You do not need to specify the source node.

vstack-cluster service move -s zpool_z01 --dst-id=2

Option	Required	What it sets
`-s`, `--service-name`	yes	service name
`--dst-id`	yes	destination node number, 1–63

The destination node must be ONLINE and open (enabled). During the move, the pool's VMs and the provisioning of its volumes are interrupted. The pool on which the management VM (vStack Center) runs can be moved only with this command: it cannot be moved from the Center interface. The procedure is described in the "Moving the pool with the management VM" section.

If the service stopped but did not start on the new node, the utility does not move it back. Check the state with the service status command and, while the service is not STARTED, contact technical support.

service enable, service disable

enable returns the service to automatic placement; disable excludes it from there.

vstack-cluster service enable -s zpool_z03

After enable, Cluster Manager recalculates placement and may immediately start the service on its own.

Exit codes

Code	Meaning
`0`	the command completed
`12`	an unknown command, an unknown option, or a required option is missing. The utility prints the reason and the help

ccrd_cli

ccrd_cli is installed on the node together with ccrd. Without arguments, it prints the same as vstack-cluster status. To work with the cluster, use vstack-cluster.

vStack.update.sh

The node upgrade script. It is located in /root/bin/vStack.update.sh and upgrades packages from the vStack repository: controllers, the base system, and the kernel. The upgrade procedure, checks, and rollback are described on the "Node upgrade to 3.2" page.

vStack.update.sh {--controllers | --base} [OPTIONS]

Example

/root/bin/vStack.update.sh --controllers --base --source release --version 3.2.0 --dry-run

What to upgrade

At least one of the two options is required; otherwise, the script exits with code 1.

Option	What it upgrades
`--controllers`	node packages, except the base system
`--base`	the base system and the kernel, and the controllers as well. After it, the node must be rebooted

Where to get packages

Option	Value	What it sets
`--source`	`release`	the vStack release repository. For customer nodes, this is the standard source. Requires `--version`
`--source`	`local`	a local repository on the node, `file:///local/repos/vstack/...`
`--version`	`X.Y.Z` or `X.Y.Z-rcN`	the distribution version, for example `3.2.0`
`--repo-url`	URL	a different repository address, for example an internal mirror

Access to the release repository is granted by an allow list of addresses. Send the node addresses for this list to vStack technical support.

How to upgrade

Option	What it does
`--dry-run`	shows the plan and changes nothing
`--no-restart`	does not restart services after the upgrade. Useful if a reboot follows
`--reboot`	reboots the node after upgrading the base system and kernel. Ignored with codes `6` and `7`
`--log PATH`	path to the log. Default: `/var/log/vstack-os-update-<ts>.log`
`-h`, `--help`	help

Boot protection

By default, the script saves the previous kernel to <kernel directory>_prev, boots the new one once with automatic rollback to the previous one, and updates the boot loader on the EFI partition. The boot loader is copied only to the EFI partition from which the node is currently booted; it does not get to the second system disk.

The options below disable these protections. Do not use them unless instructed by support.

Option	What it disables
`--no-kernel-backup`	the copy of the previous kernel. Without it, the script will not stop if there is not enough space for the copy
`--no-fallback`	one-time boot of the new kernel with automatic rollback
`--no-esp`	the boot loader update on the EFI partition

Moving between major versions

Needed only when moving from the 2.x line to 3.x, and done together with vStack technical support.

Option	What it does
`--allow-major-upgrade`	allows moving between major versions. Without it, the script rejects such a move with code `2`. Not needed for a regular upgrade within a line
`--install-missing`	additionally installs all packages of the target release that are missing on the node
`--rename-map PATH`	a map of renamed packages. Taken from the distribution by default

Exit codes

Code	Meaning
`0`	the upgrade is complete
`1`	an error in the arguments
`2`	the source or version is unavailable, the signing key was not obtained, or a move between major versions was requested without `--allow-major-upgrade`
`3`	a `pkg` error or a new kernel check error. There will be no reboot
`4`	failed to save or restore the configuration: `/etc`, `/usr/local/etc`, `vstack-env.conf`, boot files in `/boot`
`5`	an installed service did not restart
`6`	the upgrade is complete, but the `root` shell is not executable. Fix before rebooting, otherwise `root` will not be able to log in over ssh
`7`	the upgrade is complete, but the `/boot` write is not confirmed. Do not reboot until investigated

Files

Path	What it contains
`/var/backups/etc-<ts>/`	a copy of `/etc`; the `usr-local-etc` and `boot` subdirectories contain copies of `/usr/local/etc` and of the `loader.conf`, `loader.conf.local`, `device.hints` files. The five most recent are kept
`/var/backups/vstack-env-<ts>.conf`	a copy of `vstack-env.conf`
`/var/backups/update-summary-<ts>.txt`	the final run summary
`/var/log/vstack-os-update-<ts>.log`	the full log
`/boot/nextboot.conf`, `/boot/loader.conf.d/zz-vstack-fallback.conf`	one-time boot of the new kernel and rollback. After a successful boot, the node removes them itself

Cluster utility​

Nodes and services​

Commands​

Common options​

status​

node list, node status​

node disable, node enable​

service list, service status​

service start, service stop​

service move​

service enable, service disable​

Exit codes​

ccrd_cli​

vStack.update.sh​

Example​

What to upgrade​

Where to get packages​

How to upgrade​

Boot protection​

Moving between major versions​

Exit codes​

Files​

Cluster utility

Nodes and services

Commands

Common options

status

node list, node status

node disable, node enable

service list, service status

service start, service stop

service move

service enable, service disable

Exit codes

ccrd_cli

vStack.update.sh

Example

What to upgrade

Where to get packages

How to upgrade

Boot protection

Moving between major versions

Exit codes

Files