test(ci): pulp-only delivery and promotion for collect and monitoring-agent [do not merge]#3572
Draft
kduret wants to merge 22 commits into
Draft
test(ci): pulp-only delivery and promotion for collect and monitoring-agent [do not merge]#3572kduret wants to merge 22 commits into
kduret wants to merge 22 commits into
Conversation
Waiting for each upload task before sending the next upload pays the pulp task-queue latency once per package; under concurrent deliveries the queue grows and a single slow task aborts the delivery (~10 min wait_task cap) after minutes of sequential waits. Collect the task hrefs during the upload loop and await them as a batch before publishing (new wait_tasks helper); the batch guard aborts only when no task completes for ~10 min, so a draining queue is not mistaken for a hang. Refs MON-200796
The GitHub Actions OIDC token expires ~5 minutes after issuance, so any delivery longer than that started failing mid-loop with HTTP 401 (full plugins delivery: ~5400 uploads across 8 jobs died at exactly +5 min), and task polls with a stale token read as 'did not complete in time'. Refresh the token before it goes stale from the api helpers (upload attempts and task-wait sweeps), propagate it to the following steps and rewrite the pulp-cli config so the pulp commands keep working; setup-pulp-cli exports the issuance time and audience the refresh needs. Refs MON-200796
pulp_upload runs in a command substitution, so its internal token refresh only lives in the subshell: the parent shell kept the stale token, killing the rpm stable-guardrail curl with a 401 at +5 min and forcing every post-expiry deb upload to re-mint a token and rewrite the pulp-cli config (seconds per package). Refresh once per loop iteration from the parent shell instead — the guardrail and the inherited subshell token stay fresh, and the per-package refresh overhead disappears. Refs MON-200796
Large deliveries can wait minutes to hours on the server task queue with no output. Emit a progress heartbeat from wait_tasks, at most every 30 seconds (time-based: a single sweep over a large batch can take minutes on its own), counting the completed tasks over the batch total. Refs MON-200796
… E2BIG) Two failures surfaced by a 675-package delivery once the task queue drains: - refresh_pulp_token echoed the ::add-mask:: workflow command on stdout; when the refresh triggers inside pulp_upload's command substitution the command line is captured into the task href, which then never resolves (the batch wait stalls on a single corrupted href). Emit it on stderr. - manifest_write passed the packages array as an --argjson argument, which exceeds the kernel argv size limit with hundreds of packages (jq: Argument list too long, exit 126, right after a successful publish). Slurp the entries from stdin instead. Refs MON-200796
The published-metadata resolvers piped the index body into an awk that exits on the first match, so the printf writer took a SIGPIPE and bash printed one "printf: write error: Broken pipe" diagnostic per verified package. Read the index from a temporary file instead - same early exit, no pipe writer. Refs: MON-200796
A single limit=1000 page silently truncates once the repository holds more module packages than the page size (the shared apt-plugins repository is past 5000), flagging delivered packages as missing. Refs: MON-200796
List the module packages newest first and stop as soon as every expected package has been seen: the shared plugins repository holds 10k+ module packages, so an unbounded offset walk is slow, eventually fails server-side, and the expected packages sit in the tail with the default ordering. The unexpected-extras report becomes count-based for the same reason. Refs: MON-200796
grep -q exits on the first match, the printf writer takes a SIGPIPE, and pipefail turns the pipeline status into 141: every package that IS present gets flagged missing once the listing outgrows the pipe buffer (the shared plugins repositories are past 12k module packages). Write the listing to a file and let grep read the file - no writer left to kill. Refs: MON-200796
…ng wait) pulp-cli reads its token once at startup, so its built-in wait dies with "Authentication failed for tasks_read" as soon as the publication of a large repository outlives the OIDC token validity (~5 minutes). Start the publication in the background and poll it with wait_task, which re-authenticates on every attempt. Refs: MON-200796
The retry window only covers publication propagation: once a round reads the published metadata and resolves nothing new while some packages already resolved, the remaining ones are not part of the publication at all (e.g. evicted by the retention policy) - report right away instead of burning the whole 5-minute window, and log the per-round progress. Refs: MON-200796
One flaky HEAD out of hundreds (content-app/S3 hiccup) must not fail the whole verification. Refs: MON-200796
Replace the one-task-per-package association (serialized on the pulp repository lock) with the batched flow validated at full scale on the plugins matrix (8 distributions, 675 packages each, delivery + promotion, zero error): - delivery: packages are uploaded as repository-less content (create tasks parallelize across pulp workers), resolved concurrently, then associated with a single repository modify task followed by one publication; - deb flavor: the first package of each architecture goes through the legacy repository code path to materialize the suite associations, the rest of the batch gets its PackageReleaseComponent units created synchronously before the single modify; - promotion: the testing packages are listed (latest build only), their stable suite associations deduced, and promoted with a single modify task instead of one download/re-upload task per package. Refs: MON-200796
…-agent TEMP(test-pulp-only) commit, not to be merged: test and docker jobs skipped, Artifactory, jfrog and download.centreon.com deliveries disabled, packaging forced, pulp delivery forced to testing followed by an immediate promotion to stable. Refs: MON-200796
Contributor
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
skip-workflow-monitoring-agent
skip-workflow-collect
and removed
skip-workflow-monitoring-agent
skip-workflow-collect
labels
Same change as the feature branch (see its history for the rationale). Refs: MON-200796
Refs: MON-200796
…matrix) Refs: MON-200796
TEMP(test-pulp-only): the label is re-added on every run of a workflow-only PR and would skip the next retrigger. Refs: MON-200796
Refs: MON-200796
Refs: MON-200796
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
TEMP test PR, not to be merged: validates the Pulp delivery and promotion for centreon-collect and monitoring-agent on top of #3557 (updated with develop).
if: false)if: false)trigger_packagingbypassed) so the full package set builds on a workflow-only PRtesting, immediately followed by a promotion tostable(forcedreleasetype)continue-on-errorremoved so failures are loudRefs: MON-200796