feat(vhost-user): add a generic vhost-user device

CHANGELOG.md

@@ -10,6 +10,13 @@ and this project adheres to
 
 ### Added
 
+- [#5687](https://github.com/firecracker-microvm/firecracker/issues/5687): Added
+  developer preview support for a generic vhost-user frontend device. This
+  allows attaching any virtio device type (e.g. virtio-fs, virtio-scsi) via the
+  vhost-user protocol without requiring a dedicated Firecracker frontend for
+  each device type, configured via the `PUT /vhost-user-devices/{id}` API
+  endpoint. Config space is owned by the backend via the mandatory CONFIG
+  protocol feature. Snapshotting is not supported.
 - [#5891](https://github.com/firecracker-microvm/firecracker/pull/5891): Added
   support for virtio device reset.
 - [#5983](https://github.com/firecracker-microvm/firecracker/pull/5983): Add two
@@ -82,6 +89,11 @@ and this project adheres to
   Terminating a connection now also discards its TX buffer, so the device stops
   advertising `EPOLLOUT` for a host stream it will never write to again, which
   could otherwise busy-spin the event thread indefinitely.
+- [#6083](https://github.com/firecracker-microvm/firecracker/pull/6083): Fixed a
+  vhost-user-block device backed by a readonly backend not being treated as
+  readonly. The `VIRTIO_BLK_F_RO` check read the acked feature set after it had
+  been narrowed to the vhost-user protocol bit, so it never matched, and a
+  readonly vhost-user root device was given `rw` on the guest kernel cmdline.
 - [#6086](https://github.com/firecracker-microvm/firecracker/pull/6086),
   [#6143](https://github.com/firecracker-microvm/firecracker/pull/6143): Fixed a
   deadlock in the logger: a signal handler that logs while the interrupted

docs/device-api.md

@@ -33,6 +33,7 @@ BadRequest - HTTP response.
 | `entropy`                 |    O     |       O        |      O       |        O         |     O      |      O       |   **R**    |      O      |     O      |
 | `pmem/{id}`               |    O     |       O        |      O       |        O         |     O      |      O       |     O      |    **R**    |     O      |
 | `serial`                  |    O     |     **R**      |      O       |        O         |     O      |      O       |     O      |      O      |     O      |
+| `vhost-user-devices/{id}` |    O     |       O        |      O       |        O         |     O      |      O       |     O      |      O      |     O      |
 
 ## Input Schema
 

docs/vhost-user.md

@@ -0,0 +1,128 @@
+# Using generic vhost-user devices
+
+## What is a vhost-user device
+
+The
+[vhost-user protocol](https://qemu-project.gitlab.io/qemu/interop/vhost-user.html)
+allows virtio device emulation to be offloaded to a separate backend process
+communicating over a Unix domain socket. The backend handles the actual device
+logic while Firecracker acts as the frontend, managing virtqueues and guest
+memory.
+
+A generic vhost-user frontend knows nothing about the specific virtio device
+type being implemented. The backend is fully responsible for the device
+configuration space. This allows using device types that Firecracker would never
+support natively (e.g. virtio-fs, virtio-scsi) without requiring a dedicated
+frontend for each.
+
+## Prerequisites
+
+- The vhost-user backend process must be running and listening on the configured
+  Unix domain socket **before** configuring the device in Firecracker.
+- The backend must support the `VHOST_USER_PROTOCOL_F_CONFIG` protocol feature,
+  as Firecracker relies on the backend to provide the device configuration
+  space.
+- The guest kernel must include the driver for the virtio device type being
+  emulated (e.g. `CONFIG_VIRTIO_FS=y` for virtio-fs).
+
+## Configuration
+
+The following options are available:
+
+- `id` - unique identifier of the device.
+- `device_type` - the virtio device type ID as defined in the
+  [virtio specification](https://docs.oasis-open.org/virtio/virtio/v1.3/csd01/virtio-v1.3-csd01.html#x1-1930005).
+  For example: `26` for virtio-fs, `8` for virtio-scsi.
+- `socket` - path to the vhost-user backend Unix domain socket.
+- `num_queues` - number of virtqueues to configure for this device. This must
+  match what the backend and the guest driver expect for the device type, and
+  must be at least 1. For example, virtio-fs needs one hiprio queue plus at
+  least one request queue, so `num_queues` must be 2 or more; configuring fewer
+  queues than the guest driver sets up leaves the device unusable.
+- `queue_size` (optional) - size of each virtqueue. Defaults to 256.
+
+### Config file
+
+```json
+"vhost-user-devices": [
+    {
+        "id": "fs0",
+        "device_type": 26,
+        "socket": "/tmp/virtiofsd.sock",
+        "num_queues": 2,
+        "queue_size": 256
+    }
+]
+```
+
+### API
+
+```console
+curl --unix-socket $socket_location -i \
+    -X PUT 'http://localhost/vhost-user-devices/fs0' \
+    -H 'Accept: application/json' \
+    -H 'Content-Type: application/json' \
+    -d "{
+         \"id\": \"fs0\",
+         \"device_type\": 26,
+         \"socket\": \"/tmp/virtiofsd.sock\",
+         \"num_queues\": 2,
+         \"queue_size\": 256
+    }"
+```
+
+## Example: virtio-fs with virtiofsd
+
+Start the [virtiofsd](https://gitlab.com/virtio-fs/virtiofsd) backend:
+
+```console
+virtiofsd \
+    --socket-path=/tmp/virtiofsd.sock \
+    --shared-dir=/path/to/shared \
+    --tag=myfs
+```
+
+> [!NOTE]
+>
+> The `--tag` flag is required to enable the `VHOST_USER_PROTOCOL_F_CONFIG`
+> protocol feature in virtiofsd.
+
+Then configure the device in Firecracker as shown above. Inside the guest, mount
+the shared directory:
+
+```console
+mount -t virtiofs myfs /mnt
+```
+
+## Limitations
+
+- **Snapshotting is not supported.** Creating or restoring snapshots of a VM
+  with generic vhost-user devices will fail.
+- **Configuration space writes are not yet forwarded** to the backend via
+  `VHOST_USER_SET_CONFIG`. The protocol supports it, we just haven't wired it
+  up: backends such as virtiofsd and SPDK do not rely on guest-initiated config
+  writes, so it is deferred, matching the existing vhost-user block device.
+- **`num_queues` must match what the backend serves.** The backend's own feature
+  bits are offered to the guest, so a guest that accepts a multi-queue feature
+  sizes itself from the backend's configuration space, which this frontend
+  cannot parse. Configure fewer queues than the backend serves and the guest
+  will try to use queues that do not exist; the surplus in the other direction
+  is skipped harmlessly.
+- **A few feature bits are never offered**, whatever the backend advertises,
+  because honouring them is the frontend's job and Firecracker does not
+  implement them: packed rings, a platform IOMMU, notification data, per-queue
+  reset, an admin queue, SR-IOV, and dirty page logging.
+- **`config_space_size` must match the backend's configuration space.** Both the
+  vhost-user protocol and Firecracker require the backend to answer with exactly
+  as many bytes as were asked for, and a frontend agnostic to the device type
+  cannot work out how many that is. It defaults to 256, which suits a backend
+  that pads its reply; set it to the device type's own size otherwise, e.g. 44
+  for virtio-fs or 60 for virtio-block. Attaching the device fails if the reply
+  is a different length.
+- **Configuration space changes cannot be pushed to the guest.** The config
+  space is read once when the device is attached and there is no API to refresh
+  it, so a backend whose configuration changes afterwards has no way to tell the
+  guest.
+- **The backend must be started before the device is attached.** Firecracker
+  connects to the socket when processing the `PUT /vhost-user-devices/{id}`
+  request and will return an error if the backend is not available.

src/firecracker/src/api_server/parsed_request.rs

@@ -28,6 +28,7 @@ use super::request::net::{parse_patch_net, parse_put_net};
 use super::request::pmem::{parse_patch_pmem, parse_put_pmem};
 use super::request::snapshot::{parse_patch_vm_state, parse_put_snapshot};
 use super::request::version::parse_get_version;
+use super::request::vhost_user_device::parse_put_vhost_user_device;
 use super::request::vsock::parse_put_vsock;
 use crate::api_server::request::hotplug::memory::{
     parse_get_memory_hotplug, parse_patch_memory_hotplug, parse_put_memory_hotplug,
@@ -109,6 +110,9 @@ impl TryFrom<&Request> for ParsedRequest {
                 parse_put_net(body, path_tokens.next())
             }
             (Method::Put, "snapshot", Some(body)) => parse_put_snapshot(body, path_tokens.next()),
+            (Method::Put, "vhost-user-devices", Some(body)) => {
+                parse_put_vhost_user_device(body, path_tokens.next())
+            }
             (Method::Put, "vsock", Some(body)) => parse_put_vsock(body),
             (Method::Put, "entropy", Some(body)) => parse_put_entropy(body),
             (Method::Put, "hotplug", Some(body)) if path_tokens.next() == Some("memory") => {

src/firecracker/src/api_server/request/mod.rs

@@ -18,5 +18,6 @@ pub mod pmem;
 pub mod serial;
 pub mod snapshot;
 pub mod version;
+pub mod vhost_user_device;
 pub mod vsock;
 pub use micro_http::{Body, Method, StatusCode};

src/firecracker/src/api_server/request/vhost_user_device.rs

@@ -0,0 +1,107 @@
+// Copyright 2024 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+use vmm::logger::{IncMetric, METRICS};
+use vmm::rpc_interface::VmmAction;
+use vmm::vmm_config::vhost_user_device::VhostUserDeviceConfig;
+
+use super::super::parsed_request::{ParsedRequest, RequestError, checked_id};
+use super::{Body, StatusCode};
+
+pub(crate) fn parse_put_vhost_user_device(
+    body: &Body,
+    id_from_path: Option<&str>,
+) -> Result<ParsedRequest, RequestError> {
+    METRICS.put_api_requests.vhost_user_count.inc();
+    let id = if let Some(id) = id_from_path {
+        checked_id(id)?
+    } else {
+        METRICS.put_api_requests.vhost_user_fails.inc();
+        return Err(RequestError::EmptyID);
+    };
+
+    let device_cfg =
+        serde_json::from_slice::<VhostUserDeviceConfig>(body.raw()).inspect_err(|_| {
+            METRICS.put_api_requests.vhost_user_fails.inc();
+        })?;
+
+    if id != device_cfg.id {
+        METRICS.put_api_requests.vhost_user_fails.inc();
+        Err(RequestError::Generic(
+            StatusCode::BadRequest,
+            "The id from the path does not match the id from the body!".to_string(),
+        ))
+    } else {
+        Ok(ParsedRequest::new_sync(VmmAction::InsertVhostUserDevice(
+            device_cfg,
+        )))
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::api_server::parsed_request::tests::vmm_action_from_request;
+
+    #[test]
+    fn test_parse_put_vhost_user_device_request() {
+        parse_put_vhost_user_device(&Body::new("invalid_payload"), None).unwrap_err();
+        parse_put_vhost_user_device(&Body::new("invalid_payload"), Some("id")).unwrap_err();
+
+        let body = r#"{
+            "id": "bar",
+            "device_type": 26,
+            "socket": "/tmp/test.sock",
+            "num_queues": 2
+        }"#;
+        parse_put_vhost_user_device(&Body::new(body), Some("1")).unwrap_err();
+
+        let body = r#"{
+            "foo": "1"
+        }"#;
+        parse_put_vhost_user_device(&Body::new(body), Some("1")).unwrap_err();
+
+        let body = r#"{
+            "id": "fs0",
+            "device_type": 26,
+            "socket": "/tmp/virtiofsd.sock",
+            "num_queues": 2
+        }"#;
+        let r = vmm_action_from_request(
+            parse_put_vhost_user_device(&Body::new(body), Some("fs0")).unwrap(),
+        );
+
+        let expected_config = VhostUserDeviceConfig {
+            id: "fs0".to_string(),
+            device_type: 26,
+            socket: "/tmp/virtiofsd.sock".to_string(),
+            num_queues: 2,
+            queue_size: None,
+            config_space_size: None,
+        };
+        assert_eq!(r, VmmAction::InsertVhostUserDevice(expected_config));
+
+        // The optional fields are parsed when they are given.
+        let body = r#"{
+            "id": "fs0",
+            "device_type": 26,
+            "socket": "/tmp/virtiofsd.sock",
+            "num_queues": 2,
+            "queue_size": 128,
+            "config_space_size": 44
+        }"#;
+        let r = vmm_action_from_request(
+            parse_put_vhost_user_device(&Body::new(body), Some("fs0")).unwrap(),
+        );
+
+        let expected_config = VhostUserDeviceConfig {
+            id: "fs0".to_string(),
+            device_type: 26,
+            socket: "/tmp/virtiofsd.sock".to_string(),
+            num_queues: 2,
+            queue_size: Some(128),
+            config_space_size: Some(44),
+        };
+        assert_eq!(r, VmmAction::InsertVhostUserDevice(expected_config));
+    }
+}