From 906114719657df7bcd5c03d7b84ef6b254c111a4 Mon Sep 17 00:00:00 2001 From: Hector Cao Date: Wed, 10 Jun 2026 11:39:53 +0200 Subject: [PATCH 01/12] sru-exception: add sru exception page for HWE virtualization stack Signed-off-by: Hector Cao --- .../exception-Virtualization-Updates.rst | 177 ++++++++++++++++++ docs/SRU/reference/package-specific.rst | 13 ++ 2 files changed, 190 insertions(+) create mode 100644 docs/SRU/reference/exception-Virtualization-Updates.rst diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst new file mode 100644 index 000000000..0f4b5e4eb --- /dev/null +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -0,0 +1,177 @@ +.. _reference-exception-VirtualizationUpdates: + +Virtualization Updates +====================== + +This document describes the policy for updating virtualization components in a stable, +supported release starting from Ubuntu Resolute 26.04. + +This cover the following list of source packages: + +- `qemu-hwe `__ +- `libvirt-hwe `__ +- `edk2-hwe `__ +- `seabios-hwe `__ + +Rationale +--------- + +With the advent of hardware-assisted virtualization in modern CPUs, the +virtualization stack (kernel KVM, QEMU and libvirt) evolves quickly alongside +silicon, particularly with the addition of major hardware features such as +confidential computing. Production environments predominantly rely on Ubuntu LTS +releases, whose two-year cadence makes it challenging to deliver the latest +virtualization features to LTS-only users in a timely manner. + +To address this, and mirroring the long-standing +:ref:`HWE kernel ` +model, Starting from 26.04 LTS, Ubuntu provides a rolling **HWE virtualization stack** for the +LTS that tracks the upstream versions shipped in the latest Ubuntu interim release. + +Key notes +--------- + +- The HWE virtualization stack is **opt-in** + + It is separate from the base stack that remains the default in Ubuntu. Users have to + explicitly install the HWE packages to switch to the HWE virtualization stack. + +- The HWE virtualization stack will be **upgraded once every 6 months** + + The upgrade follows the same schedule as the HWE kernel. For example, when the next Ubuntu + interim release (e.g. 26.10) is released, the HWE virtualization stack will be updated in + the current LTS (e.g. 26.04) to match the versions shipped in that interim release. + +- The HWE virtualization stack rolls to have the **same upstream version the last Ubuntu release**. + + When the HWE stack rolls, it will be upgraded to have the same upstream version as the one shipped in + the latest Ubuntu release. For example, when 26.10 is released, the 26.04 HWE stack will be upgraded + to have the same upstream version as the one shipped in 26.10. + +- The HWE virtualization stack is **stopped from rolling after 2 years**. + + For example, the 26.04 LTS HWE virtualization stack will stop rolling when the next LTS (e.g. 28.04) is released, + and the last upgrade will be to match the versions shipped in the 28.04 LTS. + +Here is the illustrative timeline of a HWE component's upgrade across future Ubuntu releases: + +.. image:: https://assets.ubuntu.com/v1/4d67e99d-timeline.png + :alt: Timeline of the HWE virtualization stack release and upgrade across future Ubuntu releases + +Upgrade contents +---------------- + +- The upgrade will include the **upstream version** along with **relevant bug & security fixes** from the latest Ubuntu release. + +- Some **adjusments** may be needed to the new upstream version to make it fit in the older LTS. + +- With the above adjustments are not possible or enough to ensure the compability of the virtualization stack, **additional packages** might be needed in the SRU. For example, the build system or other dependencies of the virtualization stack (libvirt-python, virt-manager, meson, ...etc). + +Risk and mitigation +------------------- + +Upgrading the virtualization stack to have a whole new upstream version in a stable release is a major change that +might break our traditional commitments of stability and security in stable releases. This section covers the mitigations +that we put in place to both minimize the risk of such breakage. + +1 - The new version will bring in new features, deprecate/modify existing features. These should be detailed both in the file d/NEWS and the SRU bug that is linked to the d/changelog entry. + +2 - Any adjustments that are needed to make the new upstream version fit in the older LTS should be detailed in the SRU bug, and should be as minimal as possible to reduce the risk of regressions. + +3 - The upgrade might break compatiblity or exising usage. This should be avoided by extensive testing: + +Since there is complex packaging dependencies between the base and the HWE stacks to ensure they are not mixed, the upgrade +might break the base stack installation and usage. Extensive testing should be done to ensure that the upgrade does +not break the base stack installation and usage, and that switching between the base and HWE stacks with `ubuntu_virt_helper switch` +performs a complete one-to-one replacement, preserving dependencies and each package's auto/manual install mark. + +Introducting a new version might affect VM migration as Ubuntu defines new machine types; extensive testing +on migration is important to ensure we do not break existing migration scenarios, and to identify any necessary +adjustments to the new upstream version to ensure the compatibility of the stack. + +The new version might also break runtime compatibility with existing components in the LTS. This runtime +breakage is hard to identify and occurs in production, so we need to do our best to identify and fix it before the +upgrade is released. This is done by the extensive integration test suite that the virtualization components contain, +which covers a wide range of scenarios and use cases, and is ran using the SRU package for each release. The results +of these tests are attached to the SRU bug. + + +SRU Process +----------- + +The SRU should be done with a single bug, for all the source packages. This is because the components +are tightly coupled and need to be upgraded together to ensure the compatibility of the stack. +The SRU process should be followed as usual, with the following additional requirements to ensure +that the upgrade is done smoothly and with minimal regressions: + +- The SRU should be requested per the StableReleaseUpdates + documented process +- The template at the end of this document should be used and all + ‘TODO’ filled out +- For each release (e.g. resolute, etc.) that is proposed to + be updated by the SRU a link to the results of integration + testing, via autopkgtest, successfully completed using the + proposed package with no unexplained errors or failures +- Any packaging changes (e.g. a dependency changes) need to be + stated + +If backwards compatibility is to be broken, this should be clearly +written at the top of the bug description for the SRU, as well as in the +title with "[breaks-compat]". Furthermore, an email to ubuntu-release +will be sent to point the release / SRU teams to the bug in order to get +approval before uploading to the release's upload queue. + + +SRU Template +------------ + +:: + + [Impact] + This release contains bug fixes, security fixes and/or new features (and, for + the HWE stack, may bring it in line with a newer upstream version) and we + would like to make sure all of our supported customers have access to these + improvements. The notable ones are: + + *** + + See the changelog entry below for a full list of changes and bugs. + + [Test Plan] + The following development and SRU process was followed: + https://documentation.ubuntu.com/sru/en/latest/reference/exception-Virtualization-Updates + + Virtualization components contain an extensive integration test suite that is ran using + the SRU package for each releases. This test suite's results are available here: + http://autopkgtest.ubuntu.com/packages/n/virtualization-components + A successful run is required before the proposed virtualization components package + can be let into -updates. + + The virtualization team will be in charge of attaching the artifacts and console + output of the appropriate run to the bug. Virtualization team members will not + mark ‘verification-done’ until this has happened. + + For HWE stack uploads, verification must also confirm that switching between + the base and HWE variants with `ubuntu_virt_helper switch` performs a complete + one-to-one replacement, preserving dependencies and each package's auto/manual + install mark. + + [Where problems could occur] + In order to mitigate the regression potential, the results of the + aforementioned integration tests are attached to this bug. + + The HWE and base stacks are mutually exclusive and the HWE packages are + separate source packages; particular attention is required to ensure that no + variant or supported release misses a relevant change, including security + fixes. + + + + [Other Info] + + + [Changelog] + + diff --git a/docs/SRU/reference/package-specific.rst b/docs/SRU/reference/package-specific.rst index bcd4a9871..35058842a 100644 --- a/docs/SRU/reference/package-specific.rst +++ b/docs/SRU/reference/package-specific.rst @@ -68,6 +68,7 @@ the Technical Board. exception-UbuntuDevTools-Updates exception-UbuntuImage-Updates exception-Virtualbox-Updates + exception-Virtualization-Updates exception-autopkgtest-Updates exception-ec2-hibinit-agent-Updates exception-firmware-updates @@ -576,6 +577,18 @@ This is not a standing approval or policy exception, but a general pattern to update OpenVPN upstream microreleases consistently under existing SRU policy. +Virtualization +~~~~~~~~~~~~~~ + +The core virtualization components (``qemu``, ``libvirt``, ``edk2`` and +``seabios``), including their Hardware Enablement (HWE) counterparts +(``qemu-hwe``, ``libvirt-hwe``, ``edk2-hwe`` and ``seabios-hwe``), may be +uploaded according to the procedure documented in +:ref:`Virtualization Updates `. +The HWE stack provides a rolling, opt-in virtualization stack on the LTS that +tracks the upstream versions of the latest Ubuntu release, mirroring the HWE +kernel model. + Language Packs (language-pack-\*) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From b0a74a54a2b8290eac56c5bb59bad4882c16b3f1 Mon Sep 17 00:00:00 2001 From: Hector Cao <122458375+hector-cao@users.noreply.github.com> Date: Wed, 17 Jun 2026 14:44:05 +0200 Subject: [PATCH 02/12] Update docs/SRU/reference/exception-Virtualization-Updates.rst Co-authored-by: Christian Ehrhardt --- docs/SRU/reference/exception-Virtualization-Updates.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst index 0f4b5e4eb..9e0edd16c 100644 --- a/docs/SRU/reference/exception-Virtualization-Updates.rst +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -3,7 +3,7 @@ Virtualization Updates ====================== -This document describes the policy for updating virtualization components in a stable, +This document describes the policy for updating [hardware enablement virtualization components](https://ubuntu.com/server/docs/how-to/virtualisation/virt-hwe/) in a stable, supported release starting from Ubuntu Resolute 26.04. This cover the following list of source packages: From d5f6fc5060872500f481493158fbf720a4dfc6f5 Mon Sep 17 00:00:00 2001 From: Hector Cao <122458375+hector-cao@users.noreply.github.com> Date: Wed, 17 Jun 2026 14:44:22 +0200 Subject: [PATCH 03/12] Update docs/SRU/reference/exception-Virtualization-Updates.rst Co-authored-by: Christian Ehrhardt --- docs/SRU/reference/exception-Virtualization-Updates.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst index 9e0edd16c..b92863fce 100644 --- a/docs/SRU/reference/exception-Virtualization-Updates.rst +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -4,7 +4,7 @@ Virtualization Updates ====================== This document describes the policy for updating [hardware enablement virtualization components](https://ubuntu.com/server/docs/how-to/virtualisation/virt-hwe/) in a stable, -supported release starting from Ubuntu Resolute 26.04. +supported release. These opt-in components are available in Ubuntu starting from Ubuntu Resolute 26.04. This cover the following list of source packages: From 20d7cfc96ebd3d96c927200b8f2b4e0c4efd3d58 Mon Sep 17 00:00:00 2001 From: Hector Cao <122458375+hector-cao@users.noreply.github.com> Date: Wed, 17 Jun 2026 14:44:32 +0200 Subject: [PATCH 04/12] Update docs/SRU/reference/exception-Virtualization-Updates.rst Co-authored-by: Christian Ehrhardt --- docs/SRU/reference/exception-Virtualization-Updates.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst index b92863fce..967b357c1 100644 --- a/docs/SRU/reference/exception-Virtualization-Updates.rst +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -25,7 +25,7 @@ virtualization features to LTS-only users in a timely manner. To address this, and mirroring the long-standing :ref:`HWE kernel ` -model, Starting from 26.04 LTS, Ubuntu provides a rolling **HWE virtualization stack** for the +model, Starting from 26.04 LTS, Ubuntu provides an opt-in rolling **HWE virtualization stack** for the LTS that tracks the upstream versions shipped in the latest Ubuntu interim release. Key notes From a9a186bad4e2edecc4b11b3ef96d294e8c2e85eb Mon Sep 17 00:00:00 2001 From: Hector Cao <122458375+hector-cao@users.noreply.github.com> Date: Wed, 17 Jun 2026 14:44:48 +0200 Subject: [PATCH 05/12] Update docs/SRU/reference/exception-Virtualization-Updates.rst Co-authored-by: Christian Ehrhardt --- docs/SRU/reference/exception-Virtualization-Updates.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst index 967b357c1..75e95e180 100644 --- a/docs/SRU/reference/exception-Virtualization-Updates.rst +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -38,7 +38,7 @@ Key notes - The HWE virtualization stack will be **upgraded once every 6 months** - The upgrade follows the same schedule as the HWE kernel. For example, when the next Ubuntu + The upgrade follows the same schedule as the [HWE kernel](https://documentation.ubuntu.com/kernel/reference/hwe-kernels/). For example, shortly after the next Ubuntu interim release (e.g. 26.10) is released, the HWE virtualization stack will be updated in the current LTS (e.g. 26.04) to match the versions shipped in that interim release. From 2e51e5f4d0c3322b9a95155abd6db3fd03dc2445 Mon Sep 17 00:00:00 2001 From: Hector Cao <122458375+hector-cao@users.noreply.github.com> Date: Wed, 17 Jun 2026 14:45:14 +0200 Subject: [PATCH 06/12] Update docs/SRU/reference/exception-Virtualization-Updates.rst Co-authored-by: Christian Ehrhardt --- docs/SRU/reference/exception-Virtualization-Updates.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst index 75e95e180..8323b6f35 100644 --- a/docs/SRU/reference/exception-Virtualization-Updates.rst +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -42,7 +42,7 @@ Key notes interim release (e.g. 26.10) is released, the HWE virtualization stack will be updated in the current LTS (e.g. 26.04) to match the versions shipped in that interim release. -- The HWE virtualization stack rolls to have the **same upstream version the last Ubuntu release**. +- The HWE virtualization stack rolls to have the **same upstream version that is in the latest Ubuntu release**, it has already been tested and available there for a while to collect additional feedback before rolling its counterpart in the LTS. When the HWE stack rolls, it will be upgraded to have the same upstream version as the one shipped in the latest Ubuntu release. For example, when 26.10 is released, the 26.04 HWE stack will be upgraded From 7597b3dd90f39fcd5d5681500559927893c5dd40 Mon Sep 17 00:00:00 2001 From: Hector Cao <122458375+hector-cao@users.noreply.github.com> Date: Wed, 17 Jun 2026 14:46:25 +0200 Subject: [PATCH 07/12] Apply suggestion from @cpaelzer Co-authored-by: Christian Ehrhardt --- docs/SRU/reference/exception-Virtualization-Updates.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst index 8323b6f35..367deadce 100644 --- a/docs/SRU/reference/exception-Virtualization-Updates.rst +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -100,7 +100,7 @@ SRU Process ----------- The SRU should be done with a single bug, for all the source packages. This is because the components -are tightly coupled and need to be upgraded together to ensure the compatibility of the stack. +are tightly coupled and should to be upgraded together to ensure the compatibility of the stack. The SRU process should be followed as usual, with the following additional requirements to ensure that the upgrade is done smoothly and with minimal regressions: From 8aeee08b56125ca87759c444084f36cd9ffce784 Mon Sep 17 00:00:00 2001 From: Hector Cao <122458375+hector-cao@users.noreply.github.com> Date: Thu, 18 Jun 2026 09:06:28 +0200 Subject: [PATCH 08/12] Apply suggestion from @cpaelzer Co-authored-by: Christian Ehrhardt --- docs/SRU/reference/exception-Virtualization-Updates.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst index 367deadce..7d9dbeccb 100644 --- a/docs/SRU/reference/exception-Virtualization-Updates.rst +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -48,7 +48,7 @@ Key notes the latest Ubuntu release. For example, when 26.10 is released, the 26.04 HWE stack will be upgraded to have the same upstream version as the one shipped in 26.10. -- The HWE virtualization stack is **stopped from rolling after 2 years**. +- The HWE virtualization stack **stops rolling after 2 years** when the next LTS becomes available. For example, the 26.04 LTS HWE virtualization stack will stop rolling when the next LTS (e.g. 28.04) is released, and the last upgrade will be to match the versions shipped in the 28.04 LTS. From e11b68caf720b887de7ba1d4da12e3bad5b184ab Mon Sep 17 00:00:00 2001 From: Hector Cao <122458375+hector-cao@users.noreply.github.com> Date: Thu, 18 Jun 2026 09:07:08 +0200 Subject: [PATCH 09/12] Apply suggestion from @cpaelzer Co-authored-by: Christian Ehrhardt --- docs/SRU/reference/exception-Virtualization-Updates.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst index 7d9dbeccb..c31b65f37 100644 --- a/docs/SRU/reference/exception-Virtualization-Updates.rst +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -63,7 +63,7 @@ Upgrade contents - The upgrade will include the **upstream version** along with **relevant bug & security fixes** from the latest Ubuntu release. -- Some **adjusments** may be needed to the new upstream version to make it fit in the older LTS. +- Some **adjustments** (e.g. not enabling a particular new feature in the backport) may be needed to the new upstream version to make it build in fit into the older LTS. - With the above adjustments are not possible or enough to ensure the compability of the virtualization stack, **additional packages** might be needed in the SRU. For example, the build system or other dependencies of the virtualization stack (libvirt-python, virt-manager, meson, ...etc). From 42059f10962f138e3793cce01cc7f39f4c7dfe6d Mon Sep 17 00:00:00 2001 From: Hector Cao <122458375+hector-cao@users.noreply.github.com> Date: Thu, 18 Jun 2026 09:14:46 +0200 Subject: [PATCH 10/12] Apply suggestions from code review Co-authored-by: Christian Ehrhardt --- .../reference/exception-Virtualization-Updates.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst index c31b65f37..20ce17ea8 100644 --- a/docs/SRU/reference/exception-Virtualization-Updates.rst +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -65,7 +65,7 @@ Upgrade contents - Some **adjustments** (e.g. not enabling a particular new feature in the backport) may be needed to the new upstream version to make it build in fit into the older LTS. -- With the above adjustments are not possible or enough to ensure the compability of the virtualization stack, **additional packages** might be needed in the SRU. For example, the build system or other dependencies of the virtualization stack (libvirt-python, virt-manager, meson, ...etc). +- Sometimes it might happen that adjustments are not enough to ensure the build and compatibility of the virtualization stack. In that case updating it might depend on the availability of **additional packages**, it is expected that build dependencies like toolchains are the most likely to come up like that. We'd need to wait for (by toolchain team) or drive such updates (if the components are on us) to unblock the hardware enablement virtualization stack updates. Risk and mitigation ------------------- @@ -78,11 +78,11 @@ that we put in place to both minimize the risk of such breakage. 2 - Any adjustments that are needed to make the new upstream version fit in the older LTS should be detailed in the SRU bug, and should be as minimal as possible to reduce the risk of regressions. -3 - The upgrade might break compatiblity or exising usage. This should be avoided by extensive testing: +3 - The upgrade might break compatibility or existing usage. Where possible that should be avoided and to do so detected by extensive testing: -Since there is complex packaging dependencies between the base and the HWE stacks to ensure they are not mixed, the upgrade -might break the base stack installation and usage. Extensive testing should be done to ensure that the upgrade does -not break the base stack installation and usage, and that switching between the base and HWE stacks with `ubuntu_virt_helper switch` +There are complex packaging dependencies between the *base* and the *HWE* stack, to ensure they are not mixed. The upgrade of the HWE stack could affect the base installation and usage. +Extensive testing should be done to ensure that the upgrade does +not do so, and that switching between the base and HWE stacks with `ubuntu_virt_helper switch` performs a complete one-to-one replacement, preserving dependencies and each package's auto/manual install mark. Introducting a new version might affect VM migration as Ubuntu defines new machine types; extensive testing @@ -116,7 +116,7 @@ that the upgrade is done smoothly and with minimal regressions: stated If backwards compatibility is to be broken, this should be clearly -written at the top of the bug description for the SRU, as well as in the +written in the bug description for the SRU, as well as in the title with "[breaks-compat]". Furthermore, an email to ubuntu-release will be sent to point the release / SRU teams to the bug in order to get approval before uploading to the release's upload queue. @@ -138,7 +138,7 @@ SRU Template See the changelog entry below for a full list of changes and bugs. [Test Plan] - The following development and SRU process was followed: + The following additional detail to the SRU process for this case shall hereby be followed: https://documentation.ubuntu.com/sru/en/latest/reference/exception-Virtualization-Updates Virtualization components contain an extensive integration test suite that is ran using From 263acd20129b3cc6bff60697c39fd532ef37d738 Mon Sep 17 00:00:00 2001 From: Hector Cao Date: Thu, 18 Jun 2026 09:05:38 +0200 Subject: [PATCH 11/12] Improvements based on paelzer feedback Signed-off-by: Hector Cao --- docs/SRU/reference/exception-Virtualization-Updates.rst | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst index 20ce17ea8..f7589c597 100644 --- a/docs/SRU/reference/exception-Virtualization-Updates.rst +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -40,13 +40,7 @@ Key notes The upgrade follows the same schedule as the [HWE kernel](https://documentation.ubuntu.com/kernel/reference/hwe-kernels/). For example, shortly after the next Ubuntu interim release (e.g. 26.10) is released, the HWE virtualization stack will be updated in - the current LTS (e.g. 26.04) to match the versions shipped in that interim release. - -- The HWE virtualization stack rolls to have the **same upstream version that is in the latest Ubuntu release**, it has already been tested and available there for a while to collect additional feedback before rolling its counterpart in the LTS. - - When the HWE stack rolls, it will be upgraded to have the same upstream version as the one shipped in - the latest Ubuntu release. For example, when 26.10 is released, the 26.04 HWE stack will be upgraded - to have the same upstream version as the one shipped in 26.10. + the current LTS (e.g. 26.04) to **match the version shipped in that interim release**. - The HWE virtualization stack **stops rolling after 2 years** when the next LTS becomes available. From c852e45f2b7a2701d4cfa164ff2b128a6d1c681f Mon Sep 17 00:00:00 2001 From: Hector Cao Date: Thu, 18 Jun 2026 10:30:42 +0200 Subject: [PATCH 12/12] Rework the testing section and improvements Signed-off-by: Hector Cao --- .../exception-Virtualization-Updates.rst | 153 +++++++++++------- docs/SRU/reference/package-specific.rst | 6 +- 2 files changed, 93 insertions(+), 66 deletions(-) diff --git a/docs/SRU/reference/exception-Virtualization-Updates.rst b/docs/SRU/reference/exception-Virtualization-Updates.rst index f7589c597..80e61c0f6 100644 --- a/docs/SRU/reference/exception-Virtualization-Updates.rst +++ b/docs/SRU/reference/exception-Virtualization-Updates.rst @@ -6,7 +6,7 @@ Virtualization Updates This document describes the policy for updating [hardware enablement virtualization components](https://ubuntu.com/server/docs/how-to/virtualisation/virt-hwe/) in a stable, supported release. These opt-in components are available in Ubuntu starting from Ubuntu Resolute 26.04. -This cover the following list of source packages: +This covers the following list of source packages: - `qemu-hwe `__ - `libvirt-hwe `__ @@ -25,7 +25,7 @@ virtualization features to LTS-only users in a timely manner. To address this, and mirroring the long-standing :ref:`HWE kernel ` -model, Starting from 26.04 LTS, Ubuntu provides an opt-in rolling **HWE virtualization stack** for the +model, starting from 26.04 LTS, Ubuntu provides an opt-in rolling **HWE virtualization stack** for the LTS that tracks the upstream versions shipped in the latest Ubuntu interim release. Key notes @@ -57,7 +57,7 @@ Upgrade contents - The upgrade will include the **upstream version** along with **relevant bug & security fixes** from the latest Ubuntu release. -- Some **adjustments** (e.g. not enabling a particular new feature in the backport) may be needed to the new upstream version to make it build in fit into the older LTS. +- Some **adjustments** (e.g. not enabling a particular new feature in the backport) may be needed for the new upstream version to make it build and fit into the older LTS. - Sometimes it might happen that adjustments are not enough to ensure the build and compatibility of the virtualization stack. In that case updating it might depend on the availability of **additional packages**, it is expected that build dependencies like toolchains are the most likely to come up like that. We'd need to wait for (by toolchain team) or drive such updates (if the components are on us) to unblock the hardware enablement virtualization stack updates. @@ -66,54 +66,82 @@ Risk and mitigation Upgrading the virtualization stack to have a whole new upstream version in a stable release is a major change that might break our traditional commitments of stability and security in stable releases. This section covers the mitigations -that we put in place to both minimize the risk of such breakage. +that we put in place to both minimize the risk of such breakage. The mitigations are of 3 kinds: -1 - The new version will bring in new features, deprecate/modify existing features. These should be detailed both in the file d/NEWS and the SRU bug that is linked to the d/changelog entry. +- Documentation +- Feature drop: case by case decision on dropping support for a particular feature or component if it is too risky to upgrade it. +- Testing -2 - Any adjustments that are needed to make the new upstream version fit in the older LTS should be detailed in the SRU bug, and should be as minimal as possible to reduce the risk of regressions. +.. _virt-updates-documentation: -3 - The upgrade might break compatibility or existing usage. Where possible that should be avoided and to do so detected by extensive testing: +Documentation +~~~~~~~~~~~~~ +Can you add the link to the Documentation section +The places where the upgrade is documented are: -There are complex packaging dependencies between the *base* and the *HWE* stack, to ensure they are not mixed. The upgrade of the HWE stack could affect the base installation and usage. -Extensive testing should be done to ensure that the upgrade does -not do so, and that switching between the base and HWE stacks with `ubuntu_virt_helper switch` -performs a complete one-to-one replacement, preserving dependencies and each package's auto/manual install mark. + - d/NEWS, d/changelog + - `Launchpad`_ SRU bug. + - `Ubuntu Discourse `_ -Introducting a new version might affect VM migration as Ubuntu defines new machine types; extensive testing +For each upgrade, the following information should be documented: + +- any adjustments that are needed to make the new upstream version fit in the current LTS. +- any compatibility breakage with existing components or usage. + +Features drop +~~~~~~~~~~~~~ + +To mitigate the risk of breakage, we might decide to drop support for a particular +feature or component if it is too risky or impossible to have it in the current LTS. +This decision should be made on a case by case basis after careful consideration of the potential impact. + +Testing +~~~~~~~ + +In addition to the usual testing that the HWE stack naturally inherits +from the last Ubuntu release (since it has the same upstream version), some specific testing is needed to ensure +that the upgrade does not break existing usage or compatibility in the current LTS. +The following are the main areas of concern: + +1) There are complex packaging dependencies between the *base* and the *HWE* stack, to ensure they are not mixed. +The upgrade of the HWE stack could affect the installation and usage of the base stack. +Testing should be done to ensure that: + +- installing the HWE stack should remain opt-in and needs to be explicitly requested by the user. +- switching between the base and HWE stacks performs a complete one-to-one replacement, preserving dependencies and each package's auto/manual install mark. + +2) The new version might also break runtime compatibility with existing components in the LTS. This runtime +breakage is hard to identify and occurs in production, so we need to do our best to identify and fix it before the +upgrade is released. This is done by the integration test suite that makes sure that the HWE stack is compatible +with the depending applications (virt-manager, libvirt-python, ...). + +3) Introducing a new version might affect VM migration as Ubuntu defines new machine types; extensive testing on migration is important to ensure we do not break existing migration scenarios, and to identify any necessary adjustments to the new upstream version to ensure the compatibility of the stack. -The new version might also break runtime compatibility with existing components in the LTS. This runtime -breakage is hard to identify and occurs in production, so we need to do our best to identify and fix it before the -upgrade is released. This is done by the extensive integration test suite that the virtualization components contain, -which covers a wide range of scenarios and use cases, and is ran using the SRU package for each release. The results -of these tests are attached to the SRU bug. +.. important:: + As of now, we cannot run the autopkgtests for the reverse dependencies against the HWE stack since the HWE stack + is opt-in and by default the base stack will be installed for these tests. We need to figure out a way to force + the HWE stack to be installed for these tests, or to run the tests against the HWE stack in a different way. SRU Process ----------- The SRU should be done with a single bug, for all the source packages. This is because the components -are tightly coupled and should to be upgraded together to ensure the compatibility of the stack. +are tightly coupled and should be upgraded together to ensure the compatibility of the stack. The SRU process should be followed as usual, with the following additional requirements to ensure that the upgrade is done smoothly and with minimal regressions: -- The SRU should be requested per the StableReleaseUpdates - documented process +- The SRU should be requested per the :ref:`Stable Release Updates ` + documented process. - The template at the end of this document should be used and all ‘TODO’ filled out - For each release (e.g. resolute, etc.) that is proposed to be updated by the SRU a link to the results of integration - testing, via autopkgtest, successfully completed using the + testing, via autopkgtest or other testing frameworks, successfully completed using the proposed package with no unexplained errors or failures -- Any packaging changes (e.g. a dependency changes) need to be - stated - -If backwards compatibility is to be broken, this should be clearly -written in the bug description for the SRU, as well as in the -title with "[breaks-compat]". Furthermore, an email to ubuntu-release -will be sent to point the release / SRU teams to the bug in order to get -approval before uploading to the release's upload queue. +- Documentation as described in the :ref:`Documentation ` section. SRU Template @@ -122,50 +150,51 @@ SRU Template :: [Impact] - This release contains bug fixes, security fixes and/or new features (and, for - the HWE stack, may bring it in line with a newer upstream version) and we - would like to make sure all of our supported customers have access to these - improvements. The notable ones are: - *** + This release upgrades the HWE stack to the upstream version shipped in Ubuntu . + + *** + + [Notable changes and changelog] + + To have a complete list of changes, please refer to the Ubuntu release notes: + *** . + + Following are the notable adjustments that have been done to make the new upstream version fit + in the current LTS: - See the changelog entry below for a full list of changes and bugs. + *** + + Following are the breaking changes with existing components or usage: + + *** [Test Plan] - The following additional detail to the SRU process for this case shall hereby be followed: - https://documentation.ubuntu.com/sru/en/latest/reference/exception-Virtualization-Updates - Virtualization components contain an extensive integration test suite that is ran using - the SRU package for each releases. This test suite's results are available here: - http://autopkgtest.ubuntu.com/packages/n/virtualization-components - A successful run is required before the proposed virtualization components package - can be let into -updates. + - Autopkgtests with -proposed from the PPA - The virtualization team will be in charge of attaching the artifacts and console - output of the appropriate run to the bug. Virtualization team members will not - mark ‘verification-done’ until this has happened. + - Regression tests : migration - For HWE stack uploads, verification must also confirm that switching between - the base and HWE variants with `ubuntu_virt_helper switch` performs a complete - one-to-one replacement, preserving dependencies and each package's auto/manual - install mark. + - Additional integration tests with the depending applications (virt-manager, libvirt-python, ...) [Where problems could occur] - In order to mitigate the regression potential, the results of the - aforementioned integration tests are attached to this bug. - The HWE and base stacks are mutually exclusive and the HWE packages are - separate source packages; particular attention is required to ensure that no - variant or supported release misses a relevant change, including security - fixes. + This update brings a whole new upstream version into a stable release, so the + regression potential is higher than a typical SRU. The main areas where + problems could occur are: - + - The HWE and base stacks are mutually exclusive. A packaging mistake could + mix packages from both stacks, or cause the HWE stack to be pulled in + without the user explicitly opting in. - [Other Info] - + - Runtime compatibility with depending applications (virt-manager, + libvirt-python, ...) could break. - [Changelog] - + - New machine types may affect VM migration, potentially breaking existing + migration scenarios. + To mitigate these risks, the integration test suite, migration regression + tests and autopkgtests described in the [Test Plan] are run with + the proposed packages, and their results are attached to this bug. + + [Other Info] diff --git a/docs/SRU/reference/package-specific.rst b/docs/SRU/reference/package-specific.rst index 35058842a..709322b53 100644 --- a/docs/SRU/reference/package-specific.rst +++ b/docs/SRU/reference/package-specific.rst @@ -580,14 +580,12 @@ policy. Virtualization ~~~~~~~~~~~~~~ -The core virtualization components (``qemu``, ``libvirt``, ``edk2`` and -``seabios``), including their Hardware Enablement (HWE) counterparts +The Hardware Enablement (HWE) virtualization components (``qemu-hwe``, ``libvirt-hwe``, ``edk2-hwe`` and ``seabios-hwe``), may be uploaded according to the procedure documented in :ref:`Virtualization Updates `. The HWE stack provides a rolling, opt-in virtualization stack on the LTS that -tracks the upstream versions of the latest Ubuntu release, mirroring the HWE -kernel model. +tracks the upstream versions of the latest Ubuntu release. Language Packs (language-pack-\*) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~