test(ci): batched rpm delivery through a single add_content_units modify [do not merge]#6306
Draft
kduret wants to merge 28 commits into
Draft
test(ci): batched rpm delivery through a single add_content_units modify [do not merge]#6306kduret wants to merge 28 commits into
kduret wants to merge 28 commits into
Conversation
TEMP test branch (do not merge). In plugins.yml: - disable the Artifactory deliver-packages job (if: false) - force the Pulp deliver to stability: unstable, drop the stability gates (a PR runs with canary stability) and decouple the twin from test-plugins - add a marker file in the os::linux::snmp plugin packaging so get-plugins packages something to deliver All changes tagged TEMP(test-pulp-unstable). (cherry picked from commit 9174df1)
TEMP(test-pulp-unstable): reproduce the volume of the failed develop run (~30 packages per distrib) so the batched task wait is validated under a realistic load.
TEMP(test-pulp-unstable): unit-tests and test-plugins bring nothing to the batched-delivery validation and would consume up to 25 runners for the 31 marked plugins; fatpacker/package tolerate the skipped dependency.
TEMP(test-pulp-unstable): replace the 31 packaging markers with a touch of the common packaging template (changes_common=true), so get-plugins lists all 675 plugins and the batched Pulp delivery is exercised at full nightly scale (~675 packages per distrib across the 8 parallel deliver jobs).
… into MON-200796-plugins-test-batch-wait
TEMP(test-pulp-unstable) experiment: upload the rpm packages as unassociated content (repository-less create tasks parallelize across the pulp workers instead of serializing on the repository lock), resolve the created content hrefs (sha256 fallback for job re-runs), then add the whole batch to the repository with a single modify task. Two repository-serialized tasks per delivery (modify + publication) instead of one per package. Requires the reconciled rpm/packages access policy (delivery-tooling#209).
… into MON-200796-plugins-test-batch-modify
TEMP(test-pulp-unstable): same scope as the sibling batch-wait test for a direct comparison (675 packages per distrib).
…e [skip ci] The previous restriction hit the first matching matrix, which is the disabled artifactory job's; anchor on the twin's job name this time. [skip ci]: no point running until delivery-tooling#209 is merged (the orphan upload is rejected by the default access policy).
This reverts commit cf52ee2.
…nc PRC create The 2026071x-1 packages are not leftovers: #6301 is merged, so every push to develop delivers the changed plugins to Pulp with nightly versioning - the purge would fight the real pipeline. The verification misses on this test branch are an artifact of the PR versioning (20260700-<ts>) losing the RPM version comparison against develop's YYYYMMDD-1 builds. The package_release_components create is a plain synchronous DRF create (201 with the created unit, no task): take the href straight from the response, fall back to a lookup for a re-run.
A single limit=1000 page silently truncates once the repository holds more module packages than the page size (the shared apt-plugins repository is past 5000), flagging delivered packages as missing.
… into MON-200796-plugins-test-batch-modify
List the module packages newest first and stop as soon as every expected package has been seen: the shared plugins repository holds 10k+ module packages, so an unbounded offset walk is slow, eventually fails server-side, and the expected packages sit in the tail with the default ordering. The unexpected-extras report becomes count-based for the same reason.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment: batched rpm delivery through a single
add_content_unitsmodify task, on top of the full-scale test setup (675 plugins, Pulp unstable only, Artifactory and test jobs disabled).Method (deliver-rpm.sh, TEMP)
modifytask, then publish: 2 repository-serialized tasks per delivery instead of one per package (~675).deb keeps the per-package flow (the structured upload needs its per-upload
distributionparameter; release-component creation is superuser-only).Prerequisite
delivery-tooling#209 (reconciled rpm/packages access policy allowing repository-less upload for model-perm holders) must be merged and the bootstrap job re-synced; until then the uploads fail with
Destination upload repository was not provided(non-blocking warnings).Comparison target
The sibling test PR #6305 measures the same full-scale delivery with per-package repository tasks (batched waits). Refs: MON-200796