MWPW-192311: Add bulk publish runtime action for IO Studio#758
MWPW-192311: Add bulk publish runtime action for IO Studio#758
Conversation
|
Hello, I'm the AEM Code Sync Bot and I will run some actions to deploy your branch.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #758 +/- ##
==========================================
+ Coverage 87.44% 87.46% +0.02%
==========================================
Files 211 214 +3
Lines 63410 63670 +260
==========================================
+ Hits 55447 55690 +243
- Misses 7963 7980 +17
Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
Create a new bulk-publish IO Runtime action that accepts fragment paths and optional locales, publishes them to Odin with skip-already-published checks, exponential backoff retry, concurrency limiting, and in-memory queuing. Backend only — no UI in this ticket. - resolver.js: expands paths × locales using getTargetPath from common.js - skip-check.js: compares fragment.published.at vs fragment.modified.at - publisher.js: per-path publish with 3 retries via fetchOdin - queue.js: in-memory serial mutex for concurrent invocation safety - index.js: orchestrates via processBatchWithConcurrency (default 5) - 27 unit tests (proxyquire + sinon + chai), 83/83 full io/studio suite - E2E verified against prod Odin (sandbox surface): fresh publish, skip-already-published, and locale expansion all confirmed
- Add path prefix validation (must start with /content/dam/mas/) - Add input size caps: max 500 paths, 50 locales, 5000 resolved - Clamp concurrencyLimit to max 20 (prevent caller bypass) - Standardize all logging to structured JSON format - Add STATUS constants to eliminate stringly-typed comparisons - Remove dead PUBLISH_URI/WORKFLOW_MODEL_ID exports from publisher - Sanitize error response (no internal details in 500 body) - Add 3 new security validation tests (30 total, 86 full suite)
…dation - app.config.yaml: add limits.timeout: 300000 (5 min) to bulk-publish action so large batches (up to 5000 resolved paths) do not silently hit the 60s OpenWhisk default web action timeout - publisher.js: skip exponential backoff on non-retryable 4xx errors; extract HTTP status from error message via regex and break immediately on anything that is not 429 or 5xx; wrap publishPath body in outer try/catch so an unexpected throw from fetchFragmentByPath returns a structured failed result instead of rejecting the entire Promise.all batch - index.js: tighten path validation predicate so non-string entries (null, number, object) produce a clear 400 error instead of silently falling through to a confusing "No valid paths after resolution" message - Tests: add coverage for 401 no-retry, fetchFragmentByPath rejection recovery, and non-string path validation (158 passing, zero regressions)
yesil
left a comment
yesil
left a comment
There was a problem hiding this comment.
Why not use a single api call from the frontend with the list of all the fragment IDs?
as discussed in slack, io action is beneficial for logging/monitoring/etc. |
please also consider bulk publishing per locale or something, not all items |
|
Addressed in the latest push. Verified end-to-end: 13,885 paths to stage and 17,829 paths to prod, zero failures. IO action kept for logging/monitoring. |
…le chunk - publisher.js: replace per-path publishPath with publishChunk; one Odin /publish POST per chunk (up to 50 paths, Odin maxItems cap) - index.js: group resolved paths by locale first, then sub-chunk each locale into <=50-path pieces; one workflow instance per chunk - index.js: accept aemOdinEndpoint body param alongside odinEndpoint to avoid OpenWhisk "reserved properties" 400 when the package input is unbound in the workspace - Drop skip-check (Odin is idempotent for already-published fragments); remove per-path GET to fetchFragmentByPath - Tests: 32 passing (added dual-key fallback coverage) - Addresses PR #758 comments from yesil, 3ch023, npeltier
- bulk-publish.mjs: invoke the deployed action from a paths file, handle IMS token, summarize published/skipped/failed, four-layer acom/en_US guard - bulk-publish-discover.mjs: walk MAS surfaces via Odin search API, filter by modelIds (card/collection/dictionary entry+index), group by surface/locale, guard against acom/en_US leakage - bulk-publish-segment.mjs: split a paths file into per-locale files respecting the action 500-path request cap - debug-search-pagination.mjs: cursor-follow probe for verifying Odin search pagination end-to-end - README: usage for bulk-publish.mjs Used for the MWPW-192311 rollout: 17,829 fragments published across 107 locales (prod), 13,885 across 62 (stage), zero failures.
…lize discovery
- scripts/content/common.js: export CARD/COLLECTION/DICTIONARY model
IDs as the single source; add parseArgs() helper returning
{ getFlag, hasFlag }
- Four CLI scripts: replace per-script getFlag/hasFlag copies with
the shared parseArgs helper; drop mirrored model-id constants
- bulk-publish-discover.mjs: parallelize per-locale Odin search via
Promise.all (was sequential for-loop)
- io/studio/src/bulk-publish/index.js: publishOneChunk counts status
in a single reduce pass instead of two filter scans
- Tests: 160 passing across full io/studio suite (+2 vs baseline),
coverage on bulk-publish/index.js 92.25% -> 92.81%
There was a problem hiding this comment.
@Axelcureno has the OOTB bulk publish AOI been ruled out ? I would like to know why before I continue the review on this.
- index.js: replace processBatchWithConcurrency with sequential for-of - delete queue.js + queue.test.js (mutex no longer needed) - app.config.yaml: drop concurrencyLimit input Per Ilyas's PR feedback - Odin's scheduled_activation_with_references workflow handles its own queuing. The in-process queue + concurrency were leftovers from the per-path implementation; now that each chunk is one bulk POST, they don't add value.
Spec covers the 5-screen UI that pairs with the merged PR #758 backend: overview, create/edit, post-validate, confirm modal, published state. Scheduling intentionally out of MVP scope.
8 participants
Resolves https://jira.corp.adobe.com/browse/MWPW-192311
QA Checklist: https://wiki.corp.adobe.com/display/adobedotcom/M@S+Engineering+QA+Use+Cases
New IO Runtime action for bulk-publishing content fragments, plus a CLI toolchain for discovering and batching paths. Backend only - no UI changes. The Runtime layer exists for logging/observability; batching is delegated to Odin's bulk endpoint.
What this does
Action (
io/studio/src/bulk-publish/):index.js- validates input, groups paths by locale, chunks at 50, runs chunks sequentially. Limits: 500 paths, 50 locales, 5000 resolved.resolver.js- expandspaths[]xlocales[]viagetTargetPathfromcommon.js.publisher.js- single bulkPOST /adobe/sites/cf/fragments/publishper chunk withscheduled_activation_with_referencesworkflow; 3-retry exponential backoff on HTTP 0/429/5xx.app.config.yaml(runtime: nodejs:22,web: yes,final: true). IMS-validated againstallowedClientId.CLI (
scripts/content/):bulk-publish-discover.mjs(query Odin search by surface),bulk-publish-segment.mjs(split into <= 500-path files),bulk-publish.mjs(invoke the action),debug-search-pagination.mjs. Docs inscripts/content/README.md.E2E validation
Tests
io/studio/test/bulk-publish/), full studio suite green (167 passing).Test URLs
Checklist