Skip to main content

An asynchronous operation remains in progress

Symptom​

A long-running operation (such as migration or volume creation) remains in progress longer than expected.

Cause​

Long-running actions are asynchronous: the client starts the operation and then polls its status. An apparent hang usually means the operation is actually still running on the node or waiting for resources to become available.

For some operations, a long period without a response is normal. VM migration to another pool responds once, after the entire cycle: copying through a ZFS stream, then re-registering disks, snapshots, and the machine, then deleting the source dataset. The cluster does not report copy progress at all, so the absence of updates does not indicate the operation's state.

How to check​

  1. Object state. Check the State column in the VM list and the state field on the card. While the operation is running, the machine is locked and other commands targeting it are rejected.
  2. The operation log panel, LOG button: it shows the progress of the operation passed to the node agents.
  3. Machine state history: the vm-history method returns it, showing when the object entered its current state, and therefore whether the operation has been running for a minute or an hour.
  4. Node status: Provider Data Center → Server Nodes. A command sent to a node that is not ONLINE will not receive a response.

Resolution​

Check the current operation status and the load on the affected node; if the operation is not progressing and resources are available, escalate with the monitoring data.

Repeating a request with the same ID is unnecessary and counterproductive: since 3.0.5, the dispatcher recognizes redelivery by the broker and does not run the work a second time. A synchronous call receives -1000, "this request already exists"; an asynchronous call receives no response to the repeat. The saved response from the first attempt is not returned again; the result arrives for the original request. A proper retry uses a new request ID.

Migration has a separate case: code -25310 means that the cycle did not finish and failed during re-registration. This is already a failure rather than a hang. Do not start a new migration until the cause is determined; check the state of the machine and its disks on the card.

API methods​

The API operations covered on this page are vm-statuses, vm-history, vm-get, node-list, and task. See the API reference.

See also​