Recovering a degraded pool

A degraded pool is running but has lost redundancy: another disk failure in the same device group will cause data loss. This procedure has one goal: restore redundancy without losing anything along the way. Speed matters here, but the sequence matters more: misidentifying a disk turns an incident into data loss.

vStack Center obtains pool health from two sources, and they can disagree during an incident. The vStack Center database stores the state collected by inventory; the interface screens display it. Direct node queries bypass the database: they query the nodes now and write nothing to the database. During an incident, trust the direct query and use the screens for capacity information and indicators of current processes.

Source	Where it is available	What it provides	When to use it
Direct node query	Through the API only	A tree of pools and devices with each element's state	Confirm the incident and find the failed disk
Pool properties as reported by ZFS	Pool card, Properties section	Capacity, used capacity, efficiency, and the node hosting the pool	Examine a specific pool's properties
vStack Center database	Storage Resources, ZFS pools table	Health, used capacity, resilvering and scrub indicators	Determine whether recovery is in progress and whether there is enough space

A direct query without a specified node queries all cluster nodes at once and combines the responses into one list. Partial responses are allowed: an unavailable node appears as a separate entry with the UNAVAILABLE state and does not cause the entire query to fail. This leads to an important rule for reading the response: "the node did not respond" and "the pool has failed" are different events and must not be confused.

Restoring redundancy uses the same operation as planned replacement: the pool switches to a replacement disk, then resilvering begins. Protection against data loss also applies during an incident: vStack Center queries the node about the replacement disk and sends the command only if that disk is free. If its state is unknown, the operation is refused.

Physical disk operations, except for lighting indicators, are not available in the web interface. An administrator uses the API to query pools directly, query a disk's role, mark a disk as a spare, replace a disk, and remove a disk from the cluster — see the API reference.

Prerequisites

• The Cluster master role (role_id=1), required for physical disk operations and reading cluster pools;
• The feature.physical_disc_management flag: without it, the disk subsystem is unavailable, including direct pool queries, the disk list, replacement, marking a disk as a spare, and lighting indicators;
• A replacement disk on the same node as the degraded pool: free or spare. A replacement disk on another node cannot be used;
• No other replacement is in progress in this pool: only one replacement per pool at a time;
• vStack Center version 3.0.5.

Step 1. Confirm that the pool is degraded

First, make sure the degradation is real and not caused by outdated database information.

1. Open Provider Data Center → Storage Resources. The ZFS pools table contains the GUID, Name, Health, Consuming, Scrubbing, Resilvering, Current node, and Capacity (RAW) columns. The Health column identifies an unhealthy pool.

2. Query the pools directly through the API. The query is read-only and safe to repeat, and it answers a question the screen does not: which device inside the pool has failed.

3. Read the response as a tree flattened into a list.

   Entry level	What it describes	What to check
   Pool on a node	The entire pool	Pool name, state, node
   Device group	A group within a pool, such as a mirror	Group state
   Device	A specific disk	Device path and state

An entry showing a node in the UNAVAILABLE state means the node did not respond to the query, not that its pool has failed. Check that node separately in the Server Nodes table's Health column; the incident may not involve disks.

Data in the ZFS pools table comes from the database, so during an incident, check it against a direct query: the screen may show the last recorded state.

Step 2. Determine whether recovery is already in progress

If resilvering has already started, for example because the pool automatically substituted a spare disk, intervening will cause harm.

1. In the ZFS pools table, check the required pool's Resilvering column. A value of one means "recovery is in progress": wait for it to finish, then decide whether replacement is needed.
2. Open the pool and read the Properties section: Capacity, Provisioned, Used, and Free. An overfilled pool takes longer to recover, and there must be enough space for resilvering.
3. Also check Scrubbing: an integrity check started on the node also adds load to the pool.

Step 3. Locate the failed disk

A direct query provides the device path; the screens provide the model, enclosure location, and disk health. Match them precisely: you will be removing a physical disk, which is identified by its enclosure and slot.

1. Take the device path from the direct query entry with an abnormal state.
2. Open the pool card's Pool Disks section (or the Disks section on the node card) and find this path on the Placement tab in the Device column. The adjacent Enclosure_device, Enclosure_id, and Dev_index fields give the disk's location in the enclosure.
3. Check the model and disk health on the Disk props tab in the Model, Size (GB), and Status columns.
4. Light the disk's indicator so the engineer in the server room selects the correct slot. In the node card's Disks section, go to the Front view or Inner view tab. Each disk has locate and fault indicator buttons there. During an incident, the fault indicator is more appropriate: it explicitly indicates "this disk has failed".
5. Check that the indicators are lit on the LED on panel tab in the Led_fault and Led_locate columns.

If the path from the direct response is missing from the disk list, the disk has not been inventoried. Run a node inventory for the cam class, then ses, and reopen the list.

Step 4. Check whether the pool is imported

A separate type of incident looks like degradation but requires a different remedy: the pool is not running on the node. A pool runs in the cluster as a service. While the service is stopped, the cluster cannot see the pool.

1. In the ZFS pools table, check the Current node column: it indicates which node currently hosts the pool. An empty value or an absent pool in the table indicates that the pool service is not running.
2. Open the pool from the table and, in the Actions menu, select Start pool.
3. Make sure the pool appears in the table as running, then repeat the direct query: the pool should appear in that response as well.

Service permission is a second, independent attribute: a disabled service will not start, no matter how many times you try to start it. Enabling and disabling are not available in the web interface; an administrator performs these operations through the API. The procedure is explained in "Managing cluster services".

An accepted command confirms receipt by Cluster Manager, not that the service actually started. The pool table shows the actual state when the node reports back.

Step 5. Find or prepare a replacement disk

Replacement will not start until a free disk is available on the node. Free means a role of "spare" or "unused". The role in the database may be outdated, so confirm it with a direct query.

1. Find candidates in the node card's Disks section: on the ZFS props tab, a free disk has empty Pool name and Vdev_pool_name fields. Check Size (GB): the replacement disk must be at least as large as the disk being replaced.
2. Update the candidate's role by querying the node directly through the API: the node reports on the disk by device path, and vStack Center records the current role.
3. If no disk is explicitly marked as a spare, mark a free disk as a spare through the API. The node refuses this operation on a disk with live data unless the operation is forced. This is a protection mechanism.
4. Reread the disk list and make sure the replacement disk's Pool name is empty.

caution

Do not use a disk from another pool as a replacement, even if it appears free. Protection will reject the command with code -24907, but checking the role before the call is easier than investigating a refusal during an incident.

Step 6. Start the replacement

An administrator performs replacement through the API. All checks occur before the command is sent to the node: an invalid call will not reach the disk.

Prepare the following:

• The node containing the failed disk and the replacement disk;
• The pool name from the Name column in the ZFS pools table;
• The failed device's path from the direct query;
• The replacement disk's path, which must differ from the failed device's path.

An accepted command means "sent to the node". The actual outcome arrives in the node's response: on success, the replacement disk is marked as a pool member; on failure, an error is returned, and the old disk is not automatically marked as faulty.

Do not send the same command again. A second replacement in the same pool is rejected with code -24906, and a second destructive operation on the same disk with -24504.

Step 7. Wait for redundancy to be restored

Recovery runs on the node; vStack Center only displays it. Check two places: the current tree state from a direct query and the process indicator on the pools screen.

1. In the ZFS pools table, monitor the Health and Resilvering columns. A value of one in Resilvering means recovery is in progress; zero with a healthy Health value means it is complete.
2. Repeat direct queries for this node. Success means that the replacement disk's path has replaced the old path, and the pool and device group states have returned to normal.
3. Verify pool membership on the pool card in the Pool Disks section's Placement tab.

Until the pool returns to a healthy state, treat it as vulnerable: redundancy has not yet been restored, and another failure in the same device group will cause data loss.

Finishing the work

1. Turn off the indicators: node card → Disks section → Front view or Inner view tab. Check the result on the LED on panel tab.
2. Remove the failed disk from the cluster before physically removing it. This operation uses the API with the serial number or device path. Confirm with a direct query that the pool no longer uses this disk.
3. Restore the spare disk reserve. The replacement disk is now in the pool, leaving no spare on the node: bring a new disk online in the cluster and mark it as a spare.
4. Update the inventory after physical changes: run a node inventory for the ses class, then cam. Otherwise, the disk list will retain an entry for the removed disk with an outdated Last_seen.
5. Read the pool's autoreplace property through the API. If it is disabled, a spare disk will not be substituted automatically, and the next replacement will again need to be started manually.

Verifying the result

What to check	Where to look	Expected result
The pool has regained redundancy	Storage Resources, Health column	The pool is shown as healthy
The pool uses the replacement disk	Direct query through the API	The replacement disk's path has replaced the failed device's path
Resilvering has finished	Storage Resources, Resilvering column	A value of 0 with a healthy Health value
The replacement disk is recorded as a pool member	Pool card, Pool Disks → ZFS props	The replacement disk's Pool name and Vdev_pool_name fields are populated
The failed disk has been removed from the cluster	Through the API	The "online in the cluster" flag is cleared
Indicators are off	Disks section → LED on panel	Led_fault and Led_locate are off
The spare disk reserve is restored	Through the API	The node has a disk with the "spare" role and an empty Pool name
All nodes respond	Direct query through the API	The response contains no entries in the UNAVAILABLE state

If something goes wrong

What you see	Cause	What to do
A node is in the UNAVAILABLE state in the direct query response	The node did not respond to the query. This does not indicate a degraded pool	Check the node in the Server Nodes table; the incident may not involve disks
Error code -25003	The cluster node list could not be retrieved when querying all nodes	Query nodes individually and check Cluster Manager availability
Error code -24907	The replacement disk is not free: its role is neither "spare" nor "unused"	Choose another disk or free this one; update the role with a direct query
Error code -24908	The replacement disk's state could not be determined: the node did not respond or the request timed out	Replacement was deliberately rejected. Check connectivity to the node and query the disk again
Error code -24906	A replacement is already in progress in this pool	Wait for the current replacement to finish: one replacement per pool
Error code -24905	The failed device and the new device are the same	Specify different devices
Error code -24902	The pool is empty or does not exist on the node	The pool may not be imported. Check Current node and start the pool from the Actions menu
Error code -24903	The failed device's path was not provided	Use the path from the direct query
Error code -24904, -24502, -24402, -24803	Device path format validation failed	Use a path in the form /dev/daN
Error code -24504	A destructive operation is already in progress on the disk	Wait for the node's response: two destructive operations cannot run on the same disk
Error code -24503	The disk is already marked as a spare	No retry is needed; check the disk's role
Error code -24307, -24316	The enclosure name format is incorrect or the enclosure was not found	Use a name in the form /dev/sesN; if information differs, recollect the enclosure inventory
Error code -24311, -24313, -24315	The indicator value is neither 0 nor 1, or no indicator was provided	Provide at least one indicator with a value of 0 or 1
Error code -25002, -24901, -24501, -24401, -24301, -24201, -24801	The feature.physical_disc_management flag is disabled	Enable physical disk management in the cluster options
Error code -403	The operation is available only to the Cluster master role	Sign in with a cluster-level role
Error code -29	The pool map could not be retrieved from the database	Rely on direct queries: the database is currently unavailable
Error code -20512, -20522, -713	Service name or node validation failed, or the command did not reach Cluster Manager	Use the pool name from the Name column and check Cluster Manager availability
The command is accepted, but the state does not change	The command has only been forwarded to the node agent	Reopen the screens and repeat the direct query; the outcome arrives in the node's response
The direct query reports "healthy", but the ZFS pools table reports "degraded"	The database information is outdated: inventory updates it	Trust the direct query; update the database by collecting host data
Replacement succeeded, but the pool remains degraded	More than one disk has failed in the pool, or there is a failure in another device group	Reread the entire tree from the direct query and replace the remaining failed devices one at a time
The pool is degraded, but the tree contains no failed disks	The pool may have lost access to some devices because of a node rather than a disk	Check node states and the pool service; no disk replacement is needed

Limitations

You cannot start a pool integrity check from vStack Center; this operation is not available. The Scrubbing column only reports whether a process started on the node is running.

Current pool health from direct queries is not written to the vStack Center database: a direct query returns the state at the time of the call. It cannot provide a history of degradation events.

Resilvering progress is not available as a percentage; there is only an indicator of whether it is in progress. Estimate the duration from the pool size in the Properties section.

See also

• The planned procedure when the disk is still working — "Planned disk replacement on a node".
• The storage layer model and sources of pool state — "Storage pools".
• The pool map, pool page, and direct node queries — "Viewing cluster pools".
• Pool services and their node assignments — "Managing cluster services".

API methods

API operations on this page: pd-pools-status, pd-list, pd-status-map, pd-led, pd-info, pd-spare, pd-replace, pd-use, pd-unuse, pool-map, pool-map-by-guid, pools-list, pools-get, get-statusmap, node-list, node-inventory, host-inventory, cluster-service-enable, cluster-service-start — see the API reference.