OS image library
Problem and purpose
A virtual machine can be started from a ready-made OS image in seconds, whereas installation media takes tens of minutes. But the new machine's disk is created as a ZFS clone, and a clone resides on the same pool as its parent. The image must therefore already be on the pool where the machine is created. Downloading gigabytes from an external server every time a VM is created makes no sense.
The image library separates these two events in time. First, the administrator imports the image into the cluster once and distributes it across pools. Then machines are created from local copies, without accessing the external network.
This divides the model into two parts. Outside the cluster is the library
source: a standard web server with a manifest.json file listing available
images. Inside the cluster is the image catalog: images that have already
been imported, with a distribution map across pools and the state of each copy.
Synchronization reads the first part; import populates the second.
The model's third characteristic is less obvious. Source registration has no
"which VDC" (virtual data center) parameter: the Cluster master role creates a
shared cluster source, a VDC administrator creates a source of their own VDC.
Import has such a parameter, but rights decide. The Cluster master role imports
only shared images, and a specified VDC is ignored. A VDC administrator must
specify their own VDC: without it, import is rejected with code -49005, and
with another VDC, with code -49006.
In the vStack Center web interface, the library opens from the «Image library» section with the «Images» and «Sources» tabs: the cluster image catalog, adding an image from a source, removing an image, and registering, synchronizing, and removing sources. Every library operation is also available through the cluster API, by API calls or an automation system that makes those calls. The operations are listed at the end of this page; request and response structures are in the API reference.
This page explains the model. For importing an image, see How to import an OS image into the cluster; for converting a stopped VM into an image, see How to convert a VM to an image; for removing an image from the cluster, see How to remove an OS image.
Model and entities
Library source. An external server with images. A source has a name, a library root URL, and the time of its last synchronization, which is empty if it has not yet been synchronized. The URL must use http or https; https is required by default. Unencrypted http is allowed only with explicit permission. The URL is unique within the cluster: the same root URL cannot be registered twice.
Manifest. The manifest.json file at the source root is a catalog of what
the source offers. The library owner writes the manifest, and each entry contains
the complete image metadata: required name, version, format, url_path,
size, and sha256, and optional description, os_type, net_link_pattern,
zero_based_link, default_shell, sector_size_logical, sector_size_physical,
guest_customization, uefi, min_size, and uuid. The cluster determines
import eligibility itself: only images in zfs format are eligible.
The uuid field is formally optional, but import uses it to find the image in
the manifest: an entry without uuid cannot be imported. The identifier is
globally unique: the manifest must not contain two entries with the same uuid.
Image in the cluster catalog. An imported image is addressed by a surrogate identifier (UUID). Image names are not unique, so operations use the identifier, rather than the name. An image has a lifecycle stage and a map of its actual distribution across pools.
Image from a VM. Import is not the only way into the catalog. A stopped VM can be converted into an image: its boot disk becomes the image copy on its pool, and the VM itself and its snapshots are deleted. Such an image has no library source and a single copy, on the pool of the former VM.
Copy on a node → pool pair. Distribution is performed by pool, and each node → pool pair has its own outcome. This accounts for the "not distributed everywhere" state and the reverse dependency: VM disks on a pool are ZFS clones of the image copy on that specific pool.
| Entity | Addressed by | Belongs to |
|---|---|---|
| Library source | Integer identifier | The cluster (shared) or one VDC |
| Image in the catalog | UUID identifier | The owning VDC or the cluster |
| Image copy | A node → pool pair | The pool |
How it works
Working with the library involves nine operations, each with its own privilege.
| Task | Mode | Privilege |
|---|---|---|
| Register or reconfigure a source | Synchronous | Manage Image Library |
| Remove a source | Synchronous | Manage Image Library |
| View available sources | Synchronous | View Image Library |
| Read the source catalog | Synchronous | Sync Image Library |
| Import an image into the cluster | Asynchronous | Import Golden Image |
| Convert a stopped VM into an image | Asynchronous | Create Image From VM and Remove VM together |
| View the cluster image catalog | Synchronous | View Image Catalog |
| Retrieve an image dependency map | Synchronous | View Image Dependencies |
| Remove an image from the cluster | Synchronous | Remove OS Image |
All privileges are scoped to the respective VDC; the Cluster master role has global scope. The right to view dependencies is separate from the right to remove images: viewing can be granted broadly, whereas removal cannot.
Synchronization reads the manifest, not images
Synchronization downloads only manifest.json and returns the source image
catalog. No image bytes are transferred, so the operation is inexpensive and
safe. It also updates the source's last synchronization time. An image in an
unsupported format does not disappear from the catalog: it is returned with a
"not eligible for import" flag, which shows why import will not accept it.
Requests to external URLs are protected against target substitution. The scheme (http or https only) and name resolution result are checked. The cluster rejects loopback, link-local including the metadata service address, private RFC 1918 ranges, carrier-grade NAT shared address space (CGNAT), the "zero network", and the broadcast address. Disguised IPv4-mapped IPv6 forms are reduced to IPv4 and checked against the same list; native IPv6 loopback, link-local, and unique local addresses (ULA) are blocked. The manifest body is limited to 100 MB.
Import distributes the image across pools
Import responds immediately, acknowledging job acceptance rather than image
readiness: work continues in the background. The dedicated distribution service
vstack-center-image-distributor transfers the bytes using a two-phase protocol;
the cluster dispatcher only orchestrates the process. One component writes to
the image catalog, so there are no competing writes to the image state.
Progress and the outcome are visible in the cluster image catalog: an image has a stage and a distribution map with an entry for each pool.
| Stage | Meaning | Action |
|---|---|---|
importing | The image is being downloaded and distributed | Wait |
active | The image has been distributed to all selected pools | VMs can be created |
partial_active | Some pools did not receive a copy | Complete distribution by importing the same version again |
failed | The attempt failed entirely | Investigate the cause and restart the import |
deleting | The image is being removed | Wait for the outcome; import and another removal are rejected with code -48004 |
There are no automatic retries, by design. Any failure (download, checksum,
distribution, or the service losing the job) sets the image to failed and
creates a log entry. The administrator initiates a retry by importing again.
The same logic applies after a dispatcher restart: incomplete imports are
marked as failed, rather than silently retried.
Partial success is not rolled back. Pools that received a copy keep it; importing the same version again delivers the remaining copies.
Converting a VM into an image does not copy the disk
Conversion replies at once, like import, and its progress is visible in the
catalog: the image goes through importing to active, or to failed with a
reason. No bytes move: the VM's boot volume is renamed into the image volume on
the same pool. All VM snapshots are deleted before that, and the VM itself
after. If a failure happens before the rename, the VM and its disk stay intact,
though its snapshots may already be deleted.
If the VM was created from a catalog image, the link between them flips after conversion: the new image becomes the root of the clone chain, and the source catalog image becomes its dependant. That is why the new image cannot be removed while the source image exists. Details are in How to convert a VM to an image.
VM creation waits for a ready pool
As long as the image has a distribution map, its copy on the pool selected for
machine creation must be active. Otherwise, VM creation is rejected with code
-259 before anything is created. The operation order therefore reverses the
usual approach: first the image on the pool, then the machine on that pool.
For creation details, see
How to create, remove, and migrate a virtual machine.
Visibility and ownership
Images and sources have the same visibility scope. Cluster master sees everything. A VDC administrator sees shared cluster objects plus objects assigned to their VDCs. A principal without a VDC sees only shared objects. The source URL is shown only to its owner; others cannot see it.
Visibility and ownership are different rights. A VDC administrator cannot remove an image made visible to their VDC through a visibility grant: removal requires ownership. Importing a new version of an existing image is also available only to the owner or the Cluster master role. Nonexistent and invisible images return the same response, "not found", so the response cannot reveal the existence of someone else's restricted image.
Every change is logged first
Source registration and removal, synchronization, import, VM-to-image
conversion, and image removal write an intent record to the security log
before the action itself. Writing the record has a time
limit of two seconds by default. If the log does not accept the event, the
operation does not start and returns code -503. Nothing changes in either the
catalog or on the nodes.
Successful reads of the source list and image catalog are not logged,
but an access denial for any image operation leaves an
auth.image_library_access_denied entry. Viewing dependencies is recorded as a
config.image_dependencies_viewed event. Removal stages have their own entries:
config.image_remove_initiated, config.image_removed,
config.image_remove_partial, config.image_remove_failed, and
config.image_remove_blocked_dependencies. Converting a VM into an image
writes the intent record config.image_from_vm_initiated; on success,
vm.converted_to_image, vm.deleted (the VM removal), and
config.image_from_vm_done; on failure, config.image_from_vm_failed. Each
record contains a VDC
association or an indication that the object is shared across the cluster,
the caller's access scope, and their identifier. The log contains no storage
paths or pool topology, only node identifiers and aggregate counts.
Limitations
Only zfs format can be imported. An image copy on a pool is a ZFS dataset
from which machine disks are cloned, so there is nowhere to distribute another
format. An image in another format appears in the source catalog as ineligible
for import, and import itself returns code -48005. Workaround: include a zfs
variant of the image in the manifest.
Unencrypted http is blocked by default. The library supplies the data from
which all cluster machines are built, so the channel must be verifiable. Without
explicit permission, an http URL is rejected with code -46011. Workaround:
publish the source over https; reserve permission for an isolated test environment.
Internal addresses are prohibited. A source host that resolves to loopback,
link-local, the metadata service address, a private RFC 1918 range, CGNAT, the
"zero network", or the broadcast address is rejected with code -46012.
The restriction cannot be lifted: it prevents the cluster from accessing itself
at someone else's direction. A library on an internal network falls entirely
under this restriction. Workaround: move the library to an address outside the
listed ranges.
Removing a source leaves its images. Removing a source deletes only the registration and its VDC bindings; images imported from it stay in the catalog but lose their link to the source. A VDC administrator removes only their VDC's source; a shared cluster source is removed by the Cluster master role. To change the URL, there is no need to remove the source: registration with an existing source identifier acts as an overwrite of its name and URL.
Import requires the distribution service. Without a deployed
vstack-center-image-distributor (and its required vhost imgdistr_vhost),
there is nothing to transfer the bytes. The only workaround is to deploy the
service; synchronization does not involve the distribution service and reads
only the manifest.
Dependency checks can be incomplete. The clone map is assembled by querying all nodes, and an unreachable node is not included in the response; it is marked as skipped. Workaround: wait for the node to return and retrieve the map again; removal in this situation will still have a partial outcome.
Further reading
- How to import an OS image into the cluster — reading the source catalog, starting import, and monitoring distribution across pools.
- How to convert a VM to an image — VM requirements, the web interface form, and refusals; guest OS preparation for Windows and Linux.
- How to remove an OS image — checking dependencies, irreversible removal, and investigating a partial outcome.
- How to create, remove, and migrate a virtual machine — selecting an image and a pool when creating a machine.
API methods
The API operations covered on this page are os-library-set,
os-library-delete, os-library-list, os-library-sync, os-image-import,
os-image-from-vm, golden-images-list, golden-image-clones, and
golden-image-remove; see the
API reference.