Planned disk replacement on a node

Planned replacement is performed while the disk is still working: it has reached the end of its service life, its model is no longer supported, or it shows signs of impending failure. It differs from emergency replacement in one important way: you determine the sequence, rather than a failure. This means you can prepare a replacement disk, light the indicator in the correct enclosure, and replace the disk without leaving the pool without redundancy even for a minute.

In the vStack Center model, a disk has three independent attributes. Confusing them is the main source of errors in this procedure. None can be inferred from another.

Attribute	Where it is visible	What it means	What changes it
Health	Status column on the Disk props tab	The disk's actual condition	The node, based on actual conditions
Role	Through the API only	Disk role: free, partitioned, pool member, spare	Querying the disk, marking it as a spare, removing the spare label, replacement
Online in the cluster	Through the API only	The disk is made available in the cluster fabric and is visible to all nodes	Bringing the disk online in the cluster and removing it from the cluster

Replacement itself is a single operation. It does not "remove" the disk from the enclosure: it switches the pool to the replacement device and starts resilvering. The old disk is physically removed later, when the pool no longer uses it.

Physical disk operations, except for lighting indicators, are not available in the web interface. An administrator uses the API to replace a disk, mark it as a spare, and remove it from the cluster — see the API reference. The interface serves a different purpose in this procedure: showing the disk, its enclosure location, pool health, and resilvering status.

Replacement is the most dangerous operation in the disk subsystem, so it has its own protection against data loss. Before sending the command, vStack Center queries the node about the replacement disk's actual role and sends the command only if the replacement disk is free: spare or unused. If the replacement disk is in use, its state cannot be determined, or the faulty device and the new device are the same, replacement is rejected before the command is sent to the node. Concurrent replacements in the same pool are not allowed.

All disk subsystem commands are asynchronous: an accepted command means "forwarded to the node agent". The actual outcome arrives in the node's response and is visible the next time you read the disk list or pool state.

Prerequisites

• The Cluster master role (role_id=1), required for all physical disk operations;
• The feature.physical_disc_management flag: without it, the entire disk subsystem is unavailable. The flag is in the cluster options;
• A replacement disk on the same node: free or already marked as a spare. A replacement disk on another node cannot be used; replacement operates within a single node;
• The pool name and the path of the device being replaced are known;
• No other replacement is in progress in the pool: only one replacement per pool at a time;
• vStack Center version 3.0.5: physical disk operations were introduced in this release.

Step 1. Find the disk and its enclosure location

Identify the disk in three ways: by its entry in the disk list, by its device path (required by the replacement command), and by its enclosure location so you remove the correct disk.

1. Open Provider Data Center → Server Nodes and select the node, then expand the Disks section. The same list, filtered by pool, is available on the pool card in the Pool Disks section. Open the pool from the ZFS pools table in Storage Resources.

2. Switch between tabs: they show different views of the same list. The ID column links rows across all tabs, and search is available above the table (Search in table).

   Tab	Columns	Purpose at this step
   Disk props	ID, Model, Size (GB), Status	Identify the model, size, and health
   ZFS props	Pool name, Vdev_pool_name, Vdev_guid, Vdev_pool_guid	Make sure the disk belongs to the correct pool
   Placement	Enclosure_device, Enclosure_id, Dev_index, Device	Find the enclosure and slot
   LED on panel	Led_fault, Led_locate	Check that the indicator is lit
   Timestamps	Created, Last_seen	Determine how recent the disk record is

3. Record the node and Device for the replacement command, and Enclosure_device and Dev_index for lighting the indicator. Populated Pool name and Vdev_pool_name fields confirm that the disk is a pool member rather than free.

These tables do not show the disk's role or the "online in the cluster" flag; the API provides them. Check them before replacement: the role determines which disk can be used as a replacement.

Step 2. Light the disk's enclosure indicator

Choosing the wrong slot costs a second disk, so light the indicator before going to the server room. There are two indicators: locate ("this one") and fault.

1. In the node card's Disks section, go to the Front view or Inner view tab. Each disk has locate and fault indicator buttons there.
2. Turn on the locate indicator for the disk being replaced.
3. Check that the indicator is actually lit on the LED on panel tab in the Led_fault and Led_locate columns.

You can perform the same operation through the API. In that case, provide the enclosure name in the format /dev/sesN, rather than as displayed in the Enclosure_device column. Another format is rejected with code -24307, and an unknown enclosure with -24316.

Step 3. Record pool health before the operation

Record the initial state before making changes; otherwise, you will have nothing to compare against after replacement.

1. Open Provider Data Center → Storage Resources. The ZFS pools table contains the GUID, Name, Health, Consuming, Scrubbing, Resilvering, Current node, and Capacity (RAW) columns.
2. Record Health, Resilvering, and Current node for the pool in which the disk is being replaced. Start planned replacement only on a healthy pool. A degraded pool requires a different procedure, described in "Recovering a degraded pool".
3. Open the pool and check the Properties section: Capacity, Provisioned, Used, and Free indicate whether there is enough space for resilvering.

The pool table and page read from the vStack Center database and display the last recorded state. The API provides the current pool tree with each device's state. You will need it if the interface information differs from the actual state.

Step 4. Prepare the replacement disk

The replacement disk must be free; otherwise, protection will block the command. "Free" means a role of "spare" or "unused". The role in the database may be outdated, so first update it by querying the node directly.

1. Find candidates in the node card's Disks section: on the ZFS props tab, a free disk has empty Pool name and Vdev_pool_name fields. Check Size (GB): the replacement disk must be at least as large as the disk being replaced.
2. Update the candidate's role by querying the node directly through the API: the node reports on the disk by device path, and vStack Center records the current role. For a disk that has not yet been inventoried, the query changes nothing; first inventory that disk from the node page.
3. If the replacement disk is simply free, you can leave it as is: replacement accepts both spare and unused disks. If you keep a designated spare on the node, mark the disk as a spare through the API. This is a destructive operation: the node writes a service label to the disk, so it refuses the operation on a disk with live data unless labeling is forced.
4. Reread the disk list and make sure the replacement disk's Pool name is still empty.

Step 5. Replace the disk

Now issue a single command. An administrator performs it through the API. All checks occur before the command is sent to the node, so an invalid call will not reach the disk.

Prepare the following:

• The node containing both disks;
• The name of the pool in which the replacement is being performed;
• The path of the device being replaced, from the Device column;
• The replacement disk's path, which must differ from the path of the device being replaced.

An accepted command means only "sent to the node". The actual outcome arrives in the node's response. On success, the new device is marked as a pool member. On failure, an error is returned, and the old disk is not automatically marked as faulty: its role remains unchanged.

Step 6. Wait for resilvering

The pool restores redundancy itself; vStack Center only displays the process.

1. Open Storage Resources and monitor the Resilvering and Health columns for the pool in which the disk is being replaced. A value of one in Resilvering means the pool is restoring redundancy after replacement.
2. Replacement is complete when Resilvering returns to zero and Health remains healthy.
3. Verify pool membership: on the pool card, in the Pool Disks section's Placement tab, the replacement disk should appear in place of the old disk.

vStack Center does not show progress as a percentage; there is only an indicator of whether it is in progress. Estimate the duration from the pool size in the Properties section.

Step 7. Disconnect and remove the old disk

While a disk is online in the cluster, the cluster considers it part of the cluster. Before physically removing it, clear this flag. The operation uses the API and removes the LUN from the fabric. Specify the disk by serial number (preferred) or device path.

warning

Do not remove a disk from the cluster while it is still a member of an active pool: the pool will become degraded. Check the Pool Disks section to make sure the pool uses the replacement disk rather than the old disk. The node protects live data layouts and will refuse the operation, but you should not rely on that refusal instead of checking.

After the call, reread the disk list: the disk's "online in the cluster" flag is cleared. You can now physically remove the disk.

Finishing the work

1. Turn off both indicators: node card → Disks section → Front view or Inner view tab. Check the result on the LED on panel tab.
2. Restore the spare disk reserve. The replacement disk is now in the pool, leaving no spare on the node: bring a new disk online in the cluster and mark it as a spare through the API.
3. Update the inventory after physical changes: run a node inventory for the ses class, then cam. Otherwise, the disk list will retain an entry for the removed disk with an outdated Last_seen.
4. Verify the final state on the ZFS props tab: the new pool member has populated Pool name and Vdev_pool_name fields, while the spare's Pool name is empty.

Verifying the result

What to check	Where to look	Expected result
The pool uses the new disk	Pool card, Pool Disks → Placement	The replacement disk has replaced the old disk
Resilvering has finished	Storage Resources	Resilvering = 0, Health is healthy
The new disk is recorded as a pool member	Pool Disks → ZFS props	The replacement disk's Pool name and Vdev_pool_name fields are populated
The old disk has been removed from the cluster	Through the API	The "online in the cluster" flag is cleared
Indicators are off	Disks section → LED on panel	Led_fault and Led_locate are off
The spare disk reserve is restored	Through the API	The node has a disk with the "spare" role and an empty Pool name

If something goes wrong

What you see	Cause	What to do
Error code -24907	The replacement disk is not free: its role is neither "spare" nor "unused"	Free the disk or choose another; update the role by querying the node directly
Error code -24908	The replacement disk's state could not be determined: the node did not respond or the request timed out	Replacement was deliberately rejected. Check connectivity to the node and query the disk again
Error code -24905	The device being replaced and the new device are the same	Specify different devices
Error code -24906	A replacement is already in progress in this pool	Wait for the current replacement to finish: one replacement per pool
Error code -24902	The pool is empty or does not exist on the node	Check the pool name against the Name column in the ZFS pools table; the pool may not be imported on this node
Error code -24903	The path of the device being replaced was not provided	Use the path from the Device column
Error code -24904, -24502, -24402, -24803	Device path format validation failed	Use a path in the form /dev/daN
Error code -24504	A destructive operation is already in progress on the disk	Wait for the node's response: two destructive operations cannot run on the same disk
Error code -24503	The disk is already marked as a spare	No retry is needed; check the disk's role
Error code -24603	The disk is not marked as a spare, so there is no label to remove	Check the disk's role before removing the label
Error code -24307	Incorrect enclosure name format	Use a name in the form /dev/sesN, rather than as displayed in the Enclosure_device column
Error code -24316	The enclosure was not found	Check Enclosure_id and Enclosure_device on the Placement tab; if they differ, recollect the inventory
Error code -24311, -24313, -24315	The indicator value is neither 0 nor 1, or no indicator was provided	Provide at least one indicator with a value of 0 or 1
Error code -24802	Neither a serial number nor a disk path is specified	Provide at least one disk identifier
Error code -24901, -24501, -24601, -24801, -24401, -24301, -24201	The feature.physical_disc_management flag is disabled	Enable physical disk management in the cluster options
Error code -403	The operation is available only to the Cluster master role	Sign in with a cluster-level role
The command is accepted, but nothing has changed in the disk list	The command has only been forwarded to the node agent; the outcome arrives later	Reopen the screens; check Last_seen to see how recent the data is
Marking the disk as a spare succeeded, but its role did not change	The disk has not yet been inventoried, so there is no record in which to store the role	Run a node inventory for the cam class, then retry
The pool entered DEGRADED after the disk was removed	The disk was removed before the pool switched to the replacement disk	Return the disk to the enclosure and follow the recovery procedure
Replacement succeeded, but there is still no redundancy	The pool uses the replacement disk, leaving no spare on the node	Bring a new disk online in the cluster and mark it as a spare

Limitations

You cannot start a pool integrity check (scrub) from vStack Center; this operation is not available. The Scrubbing column only reports whether a process started on the node is running.

vStack Center also cannot add a disk to a pool as a new device. Bringing a disk online in the cluster makes it visible to nodes, but pool membership is changed on the node. Replacing an existing pool member is the only pool membership operation available from vStack Center.

The API provides the pool's autoreplace property. If it is disabled, a spare disk will not be substituted automatically, and replacement must always be started by an administrator.

See also

• The storage layer model and where pool capacity and health come from — "Storage pools".
• The pool map and direct node queries — "Viewing cluster pools".
• The emergency procedure when the pool has already lost redundancy — "Recovering a degraded pool".
• Bringing a new node's disks online in the cluster — "Bringing a new node into service".

API methods

API operations on this page: pd-list, pd-status-map, pd-led, pd-info, pd-spare, pd-unspare, pd-replace, pd-use, pd-unuse, pd-pools-status, pool-map-by-guid, pools-get, node-inventory — see the API reference.