Bringing a new node into service

Clusters grow by adding nodes. Hardware installation, system installation on a node, and admission of the node to the cluster take place outside vStack Center, through the installation process and Cluster Manager. vStack Center enters the procedure later, when the node is already visible to the cluster but is not yet running workloads, has not been inventoried, and has not made its disks available.

Responsibilities are strictly divided. Cluster Manager maintains cluster membership and node statuses; vStack Center displays its view and forwards commands to it. The node agent reports hardware, package versions, and system settings. These are stored in the vStack Center database and displayed on the node card.

This is why all work with a node happens in two stages. Collecting inventory and software versions is a command to the node agent, and the command itself does not report the outcome. The result appears on screen later, when the node reports back. Use the Last_seen column to check how recent the data is: it shows when the node last reported.

Disks on a new node go through two independent steps, and confusing them can be costly. Bringing a disk online makes it visible to all cluster nodes. This does not make it a member of a ZFS pool: the pool is assembled on the node, and vStack Center does not provide this operation.

Stage of bringing a node into service	Performed by	Where it is performed
Hardware installation, system installation, admission to the cluster	Installation process and Cluster Manager	Outside vStack Center — see the vStack OS documentation
Enabling the node to accept workloads	Cluster Manager, on a command from vStack Center	Not available in the web interface — use the API
Collecting hardware and software versions	Node agent	Node card: Hardware section, Software button
Aligning system settings	Node agent	Node card: Configure button
Bringing disks online in the cluster	Node agent	Not available in the web interface — use the API
Assembling a ZFS pool from the node's disks	Node	Not available in vStack Center
Starting the pool service on the node	Cluster Manager	Pool card: Actions → Start pool
Verifying cluster capacity	vStack Center	Overview, Server Nodes, Storage Resources screens

Prerequisites

• The Cluster master role (role_id=1), required for every operation in this procedure, including collecting host data;
• The feature.hardware_inventory flag: without it, the collected inventory and core utilization are not displayed;
• The feature.physical_disc_management flag: without it, physical disk operations are unavailable, including listing disks, lighting indicators, bringing a disk online in the cluster, and marking a disk as a spare;
• The feature.node_config_management flag: without it, node settings and software version collection are unavailable;
• The node has already been admitted to the cluster and is visible in the Server Nodes table;
• A list of serial numbers for the new node's disks: disks are verified by serial number;
• vStack Center version 3.0.5.

Enabled flags are listed in the cluster options. Check them there.

Step 1. Make sure the node is visible to the cluster

Start by reading the current state. Until the node appears in Cluster Manager's view, any command to bring it into service will have no target.

1. Open Provider Data Center → Server Nodes. The new node appears as a new row in the table with the ID, Name, Health, CPU (Cores), and RAM (GB) columns, and as a separate tab alongside the familiar node01…node05.
2. Record the node's ID: you will need it later for both commands and verification.
3. Check Health. A node has two independent status assessments: what the administrator permits and what is actually happening. This pair is explained in "Cluster, nodes, and services".

Cluster Manager provides the table contents. If the table is empty and the screen reports a status retrieval error, cluster membership data is unavailable. Check Cluster Manager availability and do not start bringing the node into service.

Step 2. Enable the node to accept workloads

Bringing a node into service grants administrative permission; it does not start the node. vStack Center forwards the enable command to Cluster Manager, after which the node starts accepting workloads.

Enabling and disabling a node are not available in the web interface. An administrator performs the operation through the API — see the API reference. Check the result in the Server Nodes table: an enabled node is no longer marked as prohibited from operating, and the Health column shows its actual state.

An accepted command means that Cluster Manager has received it, not that the node has already changed state. Health shows the actual change and updates when the node reports back. The node identifier is a number: the cluster rejects a value containing extraneous characters with code -20122 before sending the command.

Step 3. Collect the hardware inventory

You need the inventory immediately to check the delivered hardware against the manifest and locate disks in the enclosure. This works in two stages: collection, then reading.

1. Start an inventory of the new node. Collection can target a single node or the entire cluster, and can be limited to one device class. When bringing a node into service, a full scan is appropriate: do not select a class, and let the agent collect everything.

   Device class	Data collected
   ifnet	Network interfaces
   ses	Disk enclosures
   cam	Disk subsystem
   nvme	NVMe devices
   smbios	SMBIOS data
   No class selected	Full inventory

2. Open the node card and expand the Hardware section. The Inner view tab shows the node's hardware in the Manufacturer / Description / Product / Serial number / Version table. The nested Node data, CPU, and RAM tabs switch between views.

3. Check the CPU hardware on the Utilization tab: it shows core utilization by NUMA domain, with data no more than ten minutes old. The Portgroup tab shows networking information in the Name, Hash, Last_seen, MAC, Proto, and Status columns.

4. Check disk locations in the Disks section on the Placement tab. The Enclosure_device, Enclosure_id, Dev_index, and Device columns provide the information you will use to locate a disk in the enclosure later.

An empty Utilization tab on a newly enabled node is normal: the agent has not reported yet. Reopen the card later.

Step 4. Align software versions with the rest of the cluster

Version differences between nodes often cause unexpected cluster behavior. A new node arrives with the version from its image, and that version almost always needs to be checked.

1. Click Software. The button is available in both the node list and the node card. The Name / Version / Last_seen table opens, showing each package's version and the time it was last detected. Node package versions and cluster versions are displayed separately.
2. Check Last_seen. An old value means the node has not reported for some time, and the displayed version may differ from the installed version. In that case, start software version collection and reopen the table.
3. Compare the new node with any node already in service. If required package versions differ, stop bringing the node into service and update it.

Some packages are optional: vStack-vm-prio, vStack-storage-san, and vStack-zpool may be absent from the table. Their absence on a new node does not indicate a collection failure and is not a reason to roll back bringing the node into service.

Step 5. Apply the node configuration

System settings are grouped, and each has a class: a profile within which the value applies. The new node receives the same values already configured on nodes in service, so read the reference configuration first, then apply it.

1. Open the card of a node in service and click Configure. Settings are displayed in groups, with their current values and last detection times. Record the values that must match on the new node.
2. Before editing a value, check whether the setting is numeric or a string: this determines what the form accepts.
3. Open Configure on the new node's card, enter the values, and select the class in which they should apply. Target the change to a single node: there is no need to send it to the entire cluster when bringing a node into service.
4. Confirm the change. The command is forwarded to the node agent, so the result is not visible immediately. Reopen the settings: the changed setting's value and detection time will update.

vStack Center checks the range of numeric settings itself: a value below the minimum is rejected with code -23124, and a value above the maximum with code -23125. Such values are not sent to nodes. An unknown setting is rejected with code -23121, and a setting without the requested class with code -23123.

Step 6. Bring the node's disks online in the cluster

The new node's disks are initially "raw": the node sees them, but the cluster does not. Bringing a disk online makes it visible to all nodes. The operation is destructive and not idempotent: bringing an already online disk online again is rejected with code -24706.

Bringing a disk online in the cluster, marking it as a spare, and removing it from the cluster are not available in the web interface. An administrator performs them through the API — see the API reference. The block size and serial number under which the disk will appear in the cluster are also specified there.

1. Check what the node has detected. Open the node card's Disks section. The tabs show different views of the same list: Disk props — ID, Model, Size (GB), Status; ZFS props — Pool name, Vdev_pool_name, Vdev_guid, Vdev_pool_guid; Placement — location in the enclosure; LED on panel — Led_fault and Led_locate.
2. Distinguish three independent attributes: Status, the disk's health; the disk's role (free, partitioned, pool member, spare); and the "online in the cluster" flag. None can be inferred from another. Only the API shows the flag for bringing the disk online in the cluster.
3. Bring the required disks online and reread the list. The disk's role does not change: bringing it online in the cluster does not affect its role.
4. Consider marking some disks as spares immediately so that an emergency replacement can find a replacement disk. Marking a disk as a spare writes a service label to the disk, so the node refuses this operation on a disk with live data unless the operation is forced.

caution

Bringing a disk online in the cluster does not add it to a ZFS pool. A pool is assembled from online disks on the node; vStack Center does not provide this operation. Until a pool exists, the node's disks will not add a single byte of capacity to the cluster.

You can light a disk's indicators in the enclosure directly from the interface: the Front view and Inner view tabs in the Disks section provide locate and fault indicator buttons for each disk. The result is visible on the LED on panel tab.

Step 7. Start the pool service on the new node

A pool runs on a node as a cluster service. Until the service is enabled and started, the pool provides no capacity, even if the disks are online and the pool has been assembled.

1. Open Provider Data Center → Storage Resources. The ZFS pools table shows the cluster's pools with the GUID, Name, Health, Consuming, Scrubbing, Resilvering, Current node, and Capacity (RAW) columns. Current node indicates which node currently hosts the pool.
2. Open the pool from the table and, in the Actions menu, select Start pool.
3. Return to Storage Resources and make sure the pool is shown as running and Current node points to the new node.

A service has two independent attributes: permission, whether it is allowed to operate, and state, whether it is running now. A disabled service will not start, no matter how many times you try to start it. Enabling and disabling a service are not available in the web interface; an administrator performs these operations through the API. The operation sequence and common failures are explained in "Managing cluster services".

An accepted command confirms that Cluster Manager received it, not that the service actually started. A pool service name consists of letters, digits, and underscores, for example zpool_ssd01; the cluster does not accept hyphens, periods, or spaces.

Step 8. Verify cluster capacity

The final step answers whether the node is actually in service or merely enabled. The node's resources must appear both in the per-node view and in the cluster totals.

1. Open Provider Data Center → Overview. The screen aggregates capacity, provisioning, and actual consumption for CPU, memory, and storage. The totals should increase by the new node's resources.
2. Go to the Resources tab: the table shows nodes with the Node name, Node health, and Consuming columns. The new node should appear in the list.
3. Verify the per-node view in Server Nodes using the CPU (Cores) and RAM (GB) columns, and the node's pools in Storage Resources.

Empty CPU (Cores) and RAM (GB) values on a newly enabled node mean that its metrics are not yet available. Provisioned values are still accurate. The node's pools will also be absent until the pool has been assembled and its service started.

Finishing the work

1. Turn off any indicators you lit during installation: node card → Disks section → Front view or Inner view tabs. Check the result on the LED on panel tab in the Led_fault and Led_locate columns.
2. Rerun the inventory after the final physical changes, using the ses and cam classes.
3. Record the reference configuration: open Configure and Software for the new node. This is the state against which you will check the node next time.
4. If you are rolling back bringing the node into service, revoke its permission to accept workloads and remove its disks from the cluster. Both operations use the API. A disk that belongs to a live pool cannot be removed this way: the pool would become degraded, and the node will refuse the operation.

Verifying the result

What to check	Where to look	Expected result
The node is a cluster member	Server Nodes	A new row with the node's ID and Name
The node is enabled and running	Server Nodes, Health column	The node is not marked as prohibited from operating and is shown as running
Hardware is inventoried	Node card, Hardware section	The Inner view tab is populated, and Last_seen is recent
Versions match the cluster	Software button	Kernel version and required packages match a node in service
Configuration is applied	Node card, Configure	Values match the reference configuration, and detection times are recent
Disks are visible to the cluster	Node card, Disks section	The required disks appear in the list; the API shows the "online in the cluster" flag
The pool service is running	Storage Resources	The pool appears in the ZFS pools table, Current node is the new node, and Health = ONLINE
Capacity is accounted for	Overview	CPU, memory, and storage totals have increased

If something goes wrong

What you see	Cause	What to do
The node is absent from the Server Nodes table	Cluster Manager has not yet admitted the node to the cluster	Do not start bringing the node into service in vStack Center; return to the installation stage
The node table is empty, error code -2	Communication with Cluster Manager failed	Check Cluster Manager availability: cluster membership data is unavailable
Error code -713 for node and service commands	The command could not be forwarded to Cluster Manager	Same as above; the command did not reach the nodes
Error code -20122	Node identifier validation failed	Specify the node identifier as a number without extraneous characters
Error code -20512, -20522	Service name or node identifier validation failed	Use the service name from the Name column in the ZFS pools table and the node from Server Nodes
Error code -403 "Access denied"	The operation is available only to the Cluster master role	Sign in with a cluster-level role
Error code -26001, -26101	The feature.hardware_inventory flag is disabled	Enable the flag in the cluster options
Error code -23110 or -23301	The feature.node_config_management flag is disabled	Do the same for configuration management
Error code -24201	The feature.physical_disc_management flag is disabled	Do the same for physical disk management
Error code -200	The cluster status map is unavailable, and there are no nodes from which to collect data	Wait for the nodes to return to ONLINE and retry
Error code -23203	An invalid device class is selected	Select a class from the table in step 3
Error code -23124 or -23125	The numeric setting value is below the minimum or above the maximum	Set the value within the setting's limits
Error code -24702, -24704	Device path or disk serial number format validation failed	Check the path and serial number against the Disks section
Error code -24703	Invalid block size	Specify 512 or 4096
Error code -24706	The disk is already online in the cluster	No retry is needed; check the flag for bringing the disk online in the cluster
Error code -24707	The requested serial number is already used by another disk on the node	Choose another value or omit it so the node assigns one
Error code -24504	A destructive operation is already in progress on the disk	Wait for the node's response to the previous command
Error code -24503	The disk is already marked as a spare	No retry is needed
The command is accepted, but the data has not changed	Collection and writing are asynchronous: the command has only been forwarded to the node agent	Reopen the screen; check Last_seen to see how recent the values are
The disk is online, but cluster capacity has not increased	Bringing a disk online in the cluster does not assemble a ZFS pool	Assemble the pool on the node, then enable and start its service
The node is enabled, but CPU (Cores) and RAM (GB) are empty	The metrics store has no data for the node yet	Wait for metrics collection and check the Perfdata section on the node card

See also

• Cluster architecture, the pair of statuses for each object, and feature flags — "Cluster, nodes, and services".
• The complete set of node operations — "Managing cluster nodes".
• Pool services and their node assignments — "Managing cluster services".
• Planned disk replacement on a node in service — "Planned disk replacement on a node".

API methods

API operations on this page: node-list, get-statusmap, cluster-node-enable, cluster-node-disable, node-inventory, host-inventory, node-hw-list, node-cpu-cores, node-software-inventory, node-software, node-config-map, node-config-get, node-config-set, pd-list, pd-status-map, pd-use, pd-unuse, pd-spare, pd-led, cluster-service-enable, cluster-service-start, cluster-overview, cluster-totals — see the API reference.