feat(vhost-user): add a generic vhost-user device - #6147
Draft
1stvamp wants to merge 9 commits into
Draft
Conversation
This was referenced Aug 22, 2026
The VIRTIO_BLK_F_RO check read acked_features after it had been reassigned to acked_features & PROTOCOL_FEATURES, so it tested bit 5 against 0x4000_0000 and never matched. read_only was therefore always false, even though the device does ask the backend for VIRTIO_BLK_F_RO, and docs/api_requests/block-vhost-user.md documents the backend as the place a readonly vhost-user drive is configured. read_only feeds Block::read_only(), which decides whether a root device gets ro or rw on the guest kernel cmdline, so a readonly vhost-user root device was given rw. The guest still sees VIRTIO_BLK_F_RO and retries the root mount readonly, so it boots, but /proc/cmdline is wrong and a later remount,rw fails. There is no way to correct this from config: VhostUserBlockConfig::try_from rejects is_read_only, which leaves the backend as the only source of the flag. Compute read_only before acked_features is narrowed, so the shadowing cannot swallow the bit again. test_new_all_features already advertises VIRTIO_BLK_F_RO from its mock backend and asserted the broken result, so it now asserts read_only is set, and test_vhost_user_block asserts ro reaches the guest cmdline. Signed-off-by: Wes Mason <wes@1stvamp.org>
The vhost-user frontends duplicate everything that is not specific to the virtio device type they implement: connecting to the backend socket, negotiating virtio and protocol features, fetching the config space, allocating queues and eventfds, and setting up the backend's vrings on activation. Add VhostUserDevice to hold that shared state, built from a VhostUserDeviceSpec carrying the parts that do vary: socket path, queue count and size, the features to offer, the config space size, whether CONFIG is mandatory, and the metrics name. Device types embed it and keep their own state alongside. Two details worth noting for the callers that follow. Fetching the config space is skipped when the backend does not ack CONFIG, leaving an empty config space, since only a frontend with no device-specific fallback needs to insist on it. And activate() sets up the queues the guest marked ready rather than every allocated queue, because the number a driver initialises is dictated by the backend-owned config space, so initializing an unconfigured queue returns NotReady and aborts activation. Real vring indices are preserved: queues 0 and 2 being ready maps to vrings 0 and 2, not 0 and 1. Activating with no ready queues is rejected rather than quietly setting up nothing. The spec is validated before the socket is touched, since a frontend built from it may be configured at runtime rather than from constants: the queue count is bounded by what the PCI notification region can address, the queue size has to be a power of two, and the config space size has to fit the protocol's own limit. The config space size is a size rather than an upper bound, in that the backend has to return exactly that many bytes. The vhost crate checks the size the reply declares but not the length of the payload behind it, so the length is checked here too. Otherwise a backend could declare the size that was asked for, send fewer bytes, and leave the guest reading past the end of what it sent. No device type uses this yet. Signed-off-by: Wes Mason <wes@1stvamp.org>
The block device carried its own copy of the feature negotiation, config
space fetch, queue allocation and vring setup, none of which is specific
to block. Move that state into VhostUserDevice and keep only the
block-specific fields here: the drive id, partuuid, cache type and the
root and readonly flags.
Most of the behaviour is unchanged. Block still offers the same feature
set, including FLUSH when the cache type is Writeback, still treats
CONFIG as optional, and still reports metrics under block_{drive_id}.
Activating without a ready queue reports QueueError::NotReady as before,
though it now comes from the generic readiness check rather than from
initializing queue 0. The double-activation assert moves into the
generic device, which owns the state it guards.
read_only is now read from the features the backend acked rather than
from a local that a later shadowing narrowed, so the ordering trap fixed
in "fix(vhost-user-block): honour a readonly backend" cannot come back.
VhostUserBlockError keeps its existing variants and their messages,
which surface through the API, and gains one for the generic errors
block itself cannot produce.
One behaviour does change. Fetching the config space now goes through
the shared frontend, which requires the backend to return as many bytes
as were asked for, so a backend that answers a 60 byte request with
fewer now fails to attach where before it was accepted and left the rest
of the config space short.
Signed-off-by: Wes Mason <wes@1stvamp.org>
Add a device that carries no knowledge of the virtio device type it implements. The backend owns the config space entirely, so a device type Firecracker has no frontend for, e.g. virtio-fs or virtio-scsi, can be attached by pointing this device at a backend that speaks it. It is built on the device-type agnostic frontend added earlier, so feature negotiation, the config space fetch, queue allocation and vring setup are the same code vhost-user-block now uses. What it adds on top is a device type that comes from configuration rather than from a Rust type, and a config space that is mandatory rather than optional, since there is no device-specific fallback to fall back on. Features are passed through from the backend rather than filtered to a fixed list. A frontend that knows nothing about the device type has no basis for deciding which of that type's features the guest may have, and withholding them leaves the guest unable to use the device the way it was configured. The exceptions are the bits that change how the frontend itself drives the queues, translates addresses or reports dirty pages, since offering those would promise the guest behaviour Firecracker does not implement: packed rings, a platform IOMMU, notification data, per-queue reset, an admin queue, SR-IOV and dirty page logging. A consequence worth stating: a guest offered the backend's features may size itself from the backend's config space, which this frontend cannot parse. So the configured queue count has to agree with what the backend serves, and that is the operator's to get right. The host-side device type is a sentinel, VhostUserGeneric, which never reaches the guest: virtio_device_type_id() reports the configured type instead, feeding the MMIO device-type register and both the PCI device ID and its class code. The PCI class code is now keyed on that rather than on device_type(), which is the same value for every other device and the real type rather than the sentinel for this one, so a generic device carrying a block type is reported as a mass storage controller instead of falling back to an unassigned class. The size of the config space to fetch is configurable, defaulting to 256 bytes. Only the device type knows how large its config space is, and the protocol requires the backend to answer with exactly as many bytes as were asked for, so a frontend agnostic to the device type cannot work it out and whoever configures the device says instead. The default suits a backend that pads its reply. Device types 0 and 0xFF are rejected. The virtio-mmio driver reads 0 as no device, so the guest would ignore it and whoever configured it would be told nothing, and 0xFF is the value Firecracker keys this device type by internally, so allowing it would leave the guest-visible type and the host-side one indistinguishable by value. Snapshotting is refused, as it is for vhost-user-block. Skipping the device instead would write a snapshot missing a device that the guest driver still expects to be there on restore. Co-authored-by: Amit Patil <iamitpatil2001@gmail.com> Signed-off-by: Amit Patil <iamitpatil2001@gmail.com> Signed-off-by: Wes Mason <wes@1stvamp.org>
Expose the device through `PUT /vhost-user-devices/{id}`: the request
handler and its parsed-request plumbing, the swagger definition, the
builder that attaches configured devices at boot, and the request
metrics, registered in the integration suite's metrics schema so that
validation still matches what Firecracker emits.
`GET /vm/config` reports the configured devices back, reporting the
queue size the device was built with rather than the smaller one a guest
driver may since have selected, so the config round-trips after boot as
well as before it.
The response now carries a `vhost-user-devices` list for every microVM,
empty when none are configured. Every place that compares a whole config
response is updated for that: both full-config tests and the config
fixture the command-line start test compares against.
The swagger definition is bounded to what the code actually accepts,
rather than to the range of the underlying types, and gains the optional
config space size alongside the queue size.
Signed-off-by: Amit Patil <iamitpatil2001@gmail.com>
Signed-off-by: Wes Mason <wes@1stvamp.org>
Cover booting a guest off a generic vhost-user block device over both MMIO and PCI, read-only and read-write, a device configured with more queues than the guest driver ends up using, and a backend that goes away while the microVM is running. The read-only case is the end-to-end check that the backend's own feature bits reach the guest: the frontend knows nothing of VIRTIO_BLK_F_RO, it just passes on whatever the backend offered, and the guest mounting the device read-only is what proves it arrived. The queue counts are matched to what the backend serves rather than set to 1, because a guest offered the backend's features sizes itself from the backend's config space. The multi-queue case configures more than the backend serves, leaving a surplus for the frontend to skip. Signed-off-by: Amit Patil <iamitpatil2001@gmail.com> Signed-off-by: Wes Mason <wes@1stvamp.org>
Assert that snapshotting a microVM with a generic vhost-user device fails, so that the device cannot silently go missing from a snapshot the guest driver would still expect it in. The microVM has to be paused first. A running one is refused earlier for running, and never reaches the device save path, so the test would pass without exercising the thing it is there for. Signed-off-by: Wes Mason <wes@1stvamp.org>
Describe what the device is for, how to configure it over the API, and a worked virtiofsd example. Records the limitations a user will otherwise hit as a confusing runtime failure: snapshotting is refused, config space writes are not forwarded to the backend and changes to it cannot be pushed to the guest, the queue count has to match what the backend serves, a few feature bits are never offered because the frontend would have to implement them, the backend has to be listening before the device is attached, and the configured config space size has to match what the backend answers with. Lists the new endpoint in the device API matrix. The per-device columns there are left alone, since the device is a developer preview. Signed-off-by: Amit Patil <iamitpatil2001@gmail.com> Signed-off-by: Wes Mason <wes@1stvamp.org>
`GET /vm/config` reporting a configured device back was claimed but not covered, and the response gaining a `vhost-user-devices` list changes every microVM's config output, so it is worth pinning down. Signed-off-by: Wes Mason <wes@1stvamp.org>
1stvamp
force-pushed
the
feat/vhost-user-generic-device
branch
from
August 25, 2026 01:17
2ec2e80 to
535e268
Compare
zulinx86
added
the
Status: Awaiting assignee
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds a generic vhost-user device, configured over
PUT /vhost-user-devices/{id}, that carries no knowledge of the virtio device type it implements: the type comes from configuration and the backend owns the config space.Snapshotting a microVM with one of these devices is refused, as it already is for vhost-user-block.
Reason
Attaching a vhost-user backend for a device type Firecracker has no frontend for (e.g. virtio-fs, virtio-scsi) means writing that frontend first. A frontend that passes the config space through to the backend can serve any device type without one.
Closes #5687. Supersedes #5773, @meAmitPatil's original implementation, which went quiet with the author absent; the commits that are his keep his authorship.
This is step 3 of the 3 @ShadowCurse asked for in #6072.
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.PR Checklist
tools/devtool checkbuild --allto verify that the PR passesbuild checks on all supported architectures.
tools/devtool checkstyleto verify that the PR passes theautomated style checks.
how they are solving the problem in a clear and encompassing way.
in the PR.
CHANGELOG.md.Runbook for Firecracker API changes.
integration tests.
TODO.rust-vmm.This is part 4 of 4 in a stack made with GitButler: