perf(ci): batch pulp delivery and promotion into single modify tasks#3557
Draft
kduret wants to merge 20 commits into
Draft
perf(ci): batch pulp delivery and promotion into single modify tasks#3557kduret wants to merge 20 commits into
kduret wants to merge 20 commits into
Conversation
Waiting for each upload task before sending the next upload pays the pulp task-queue latency once per package; under concurrent deliveries the queue grows and a single slow task aborts the delivery (~10 min wait_task cap) after minutes of sequential waits. Collect the task hrefs during the upload loop and await them as a batch before publishing (new wait_tasks helper); the batch guard aborts only when no task completes for ~10 min, so a draining queue is not mistaken for a hang. Refs MON-200796
kduret
requested review from
mushroomempires and
tuntoja
and removed request for
a team
July 13, 2026 10:11
Contributor
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The GitHub Actions OIDC token expires ~5 minutes after issuance, so any delivery longer than that started failing mid-loop with HTTP 401 (full plugins delivery: ~5400 uploads across 8 jobs died at exactly +5 min), and task polls with a stale token read as 'did not complete in time'. Refresh the token before it goes stale from the api helpers (upload attempts and task-wait sweeps), propagate it to the following steps and rewrite the pulp-cli config so the pulp commands keep working; setup-pulp-cli exports the issuance time and audience the refresh needs. Refs MON-200796
pulp_upload runs in a command substitution, so its internal token refresh only lives in the subshell: the parent shell kept the stale token, killing the rpm stable-guardrail curl with a 401 at +5 min and forcing every post-expiry deb upload to re-mint a token and rewrite the pulp-cli config (seconds per package). Refresh once per loop iteration from the parent shell instead — the guardrail and the inherited subshell token stay fresh, and the per-package refresh overhead disappears. Refs MON-200796
tuntoja
previously approved these changes
Jul 15, 2026
kduret
marked this pull request as draft
July 15, 2026 06:15
Large deliveries can wait minutes to hours on the server task queue with no output. Emit a progress heartbeat from wait_tasks, at most every 30 seconds (time-based: a single sweep over a large batch can take minutes on its own), counting the completed tasks over the batch total. Refs MON-200796
… E2BIG) Two failures surfaced by a 675-package delivery once the task queue drains: - refresh_pulp_token echoed the ::add-mask:: workflow command on stdout; when the refresh triggers inside pulp_upload's command substitution the command line is captured into the task href, which then never resolves (the batch wait stalls on a single corrupted href). Emit it on stderr. - manifest_write passed the packages array as an --argjson argument, which exceeds the kernel argv size limit with hundreds of packages (jq: Argument list too long, exit 126, right after a successful publish). Slurp the entries from stdin instead. Refs MON-200796
Contributor
|
Caution Review failedAn error occurred during the review process. Please try again later. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The published-metadata resolvers piped the index body into an awk that exits on the first match, so the printf writer took a SIGPIPE and bash printed one "printf: write error: Broken pipe" diagnostic per verified package. Read the index from a temporary file instead - same early exit, no pipe writer. Refs: MON-200796
A single limit=1000 page silently truncates once the repository holds more module packages than the page size (the shared apt-plugins repository is past 5000), flagging delivered packages as missing. Refs: MON-200796
List the module packages newest first and stop as soon as every expected package has been seen: the shared plugins repository holds 10k+ module packages, so an unbounded offset walk is slow, eventually fails server-side, and the expected packages sit in the tail with the default ordering. The unexpected-extras report becomes count-based for the same reason. Refs: MON-200796
grep -q exits on the first match, the printf writer takes a SIGPIPE, and pipefail turns the pipeline status into 141: every package that IS present gets flagged missing once the listing outgrows the pipe buffer (the shared plugins repositories are past 12k module packages). Write the listing to a file and let grep read the file - no writer left to kill. Refs: MON-200796
…ng wait) pulp-cli reads its token once at startup, so its built-in wait dies with "Authentication failed for tasks_read" as soon as the publication of a large repository outlives the OIDC token validity (~5 minutes). Start the publication in the background and poll it with wait_task, which re-authenticates on every attempt. Refs: MON-200796
The retry window only covers publication propagation: once a round reads the published metadata and resolves nothing new while some packages already resolved, the remaining ones are not part of the publication at all (e.g. evicted by the retention policy) - report right away instead of burning the whole 5-minute window, and log the per-round progress. Refs: MON-200796
One flaky HEAD out of hundreds (content-app/S3 hiccup) must not fail the whole verification. Refs: MON-200796
Replace the one-task-per-package association (serialized on the pulp repository lock) with the batched flow validated at full scale on the plugins matrix (8 distributions, 675 packages each, delivery + promotion, zero error): - delivery: packages are uploaded as repository-less content (create tasks parallelize across pulp workers), resolved concurrently, then associated with a single repository modify task followed by one publication; - deb flavor: the first package of each architecture goes through the legacy repository code path to materialize the suite associations, the rest of the batch gets its PackageReleaseComponent units created synchronously before the single modify; - promotion: the testing packages are listed (latest build only), their stable suite associations deduced, and promoted with a single modify task instead of one download/re-upload task per package. Refs: MON-200796
kduret
changed the title
The promotion sweep selected testing packages by their relative_path prefix (pool/testing/<module>/). Content reused across suites breaks that assumption: a byte-identical package first delivered to another suite is deduplicated by pulp and keeps its original upload path, so the sweep missed it (20 of 27 jammy perl-cpan-libraries packages in the pulp-only test PR). Membership now comes from the testing suite's PackageReleaseComponent associations. The release-component hrefs themselves are now resolved directly from the release_components listing (?distribution=<suite>&component=main), opened to authenticated users by the pulp bootstrap (delivery-tooling#230), instead of being deduced from the first package's associations - a deduction that was likewise ambiguous for reused content. Refs: MON-200796
The release_components viewset ships without an access policy upstream (pulp_deb 3.9, admin-only fallback) and pulpcore offers no way to create one, so the previous commit's direct listing can never work for the CI user. Replace it with mechanisms built on endpoints the CI user can read: - promotion candidates are selected by joining the module content listing with the published dists/<testing suite> Packages indexes on sha256 (fail-closed on index fetch errors), instead of trusting relative_path; - when every candidate is already published in the stable suite, the promotion short-circuits to a republish (idempotent reruns); - the delivery picks a FRESH package (content not yet in pulp) as the legacy representative of each architecture whenever possible: its single post-upload association identifies the suite release component exactly; reused representatives fall back to a before/after association diff; - the promotion deduces the stable release component from the same before/after diff on its reference package, subtracting the associations of not-yet-promoted candidates on reruns. Refs: MON-200796
The guarded *business-internal distributions answer 403 to anonymous requests, so the delivery verification could not check the served metadata or fetchability of business modules (map showed 3/3 present, 0/3 resolvable), and the deb suite-index reads would fail on guarded suites. Every content-endpoint fetch now goes through content_curl, which sends the same GitHub OIDC credentials as the api calls (the CI user holds the guard downloader role, delivery-tooling#232); unguarded distributions ignore the header. Refs: MON-200796
Same change as the feature branch (see its history for the rationale). Refs: MON-200796
Refs: MON-200796
Align the shared pulp delivery actions with centreon-plugins: the deb plugins stable packages live in a dedicated repository (apt-plugins-stable / ubuntu-plugins-stable) with plain-codename suites, matching the artifactory client layout. For every other deb family the new stable_repository_name/stable_base_path outputs default to the delivery repository, so behavior is unchanged. Also picks up the latest shared-script fixes (index fetch retries, partial-rerun release component fallback). Refs: MON-200796
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Speed up and harden the Pulp delivery and promotion by replacing the one-task-per-package flow with batched single-
modifyoperations.Pulp serializes the tasks of a repository server-side (per-repository lock). With one association task per package, a full delivery paid the queue latency N times: on a develop centreon-plugins run (8 distributions in parallel), deliveries timed out, and the sequential promotion was estimated at ~90 minutes per distribution.
Changes
Delivery (
deliver-rpm.sh/deliver-deb.sh):modifytask, followed by one publication.Promotion (
promote-rpm.sh/promote-deb.sh):modifytask instead of one download/re-upload task per package (~2-3 min instead of ~90 min per distribution).Shared (
api.sh,check-*.sh):Validation
Validated at full scale on the plugins matrix: 8 distributions in parallel with no concurrency cap, 675 packages each, delivery to testing + promotion to stable,
675/675 present · 675/675 resolvableon every repository, zero error and zero warning (run 29748876765).Applied identically to centreon, centreon-collect, centreon-modules and centreon-plugins (shared pulp files kept byte-identical).
Refs: MON-200796