Skip to content

perf(core): load workflows lazily from the generated entrypoint - #476

Merged
szanata merged 4 commits into
mainfrom
perf/lazy-workflow-entrypoint
Oct 30, 2026
Merged

szanata merged 4 commits into
mainfrom
perf/lazy-workflow-entrypoint

Conversation

@derencius

@derencius derencius commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Problem

The generated workflow entrypoint statically re-exports every workflow (export { default as X } from 'P';). Temporal's reusable VM caches modules loaded during an activation in that workflow's own moduleCache, so every cached workflow evaluates the whole catalog.

Mechanism

@temporalio/worker/lib/workflow/reusable-vm.js:138-156: __webpack_module_cache__ is a Proxy. A set before any activator exists goes into the shared sharedModules map; a set during an activation goes into that workflow's moduleCache (lines 150-156). A static entrypoint is evaluated inside each activation, so each cached workflow holds its own copy of every workflow module.

Change

Each user workflow and alias export in the entrypoint becomes an async wrapper that does await import( /* webpackMode: "eager" */ 'P' ) and calls the default export. Eager mode keeps everything in one bundle. $catalog stays a static export. Workflow type resolution is by export name, which is unchanged. The framework reads workflow metadata (METADATA_ACCESS_SYMBOL) only in Node (importComponents in the loader and the catalog), never from the bundle, so the wrapper loses nothing. --check bundles the same entrypoint through bundleWorkflowCode; a one-workflow lazy entrypoint bundled successfully with that call.

Measurements

Measured 2026-10-19 on a GrowthX os-workflows worker (@outputai/core 0.13.0, Temporal TS SDK 1.20.3, reuseV8Context default true, macOS footprint, workflows parked in sleep). The lazy rows come from patching the same shape by hand in node_modules.

Setup Per cached workflow Idle worker
228-workflow catalog, static entrypoint ~114 MB 1.9 GB
1-workflow catalog, static entrypoint ~4.6 MB 0.76 GB
228-workflow catalog, lazy entrypoint ~6.5 MB 1.54 GB

Prod pods reached 7-8 GB at 25 cached workflows and the fleet autoscaled to its 20-pod max. bundlerOptions.preloadModules did not help; preloading @outputai/core breaks activations ("Cannot assign to read only property 'lastIndex' of object '[object RegExp]'"). With the hand patch, a parked workflow and a real workflow with 5 activities and 3 nested child workflows completed across forced worker restarts with zero nondeterminism or module errors.

Follow-up local A/B benchmark

Repeated 2026-10-30 after rebasing against main, using Node 24.15.0, Temporal TS SDK 1.24.0, reuseV8Context: true, and the 72-workflow test_workflows catalog. Each variant started from a stopped Compose environment. Memory values are medians of three docker stats samples before and after starting 100 unique sleep workflows parked on their first timer. Worker telemetry confirmed 101 cached workflows, including the catalog, with no in-flight work.

Metric Static entrypoint Lazy entrypoint Impact
Idle worker 397.6 MiB 400.2 MiB +2.6 MiB
Loaded worker, 100 cached executions 764.9 MiB 563.0 MiB -201.9 MiB (-26.4%)
Incremental memory, 100 executions 367.3 MiB 162.8 MiB -204.5 MiB (-55.7%)
Incremental memory per execution 3.673 MiB 1.628 MiB -2.045 MiB (-55.7%)
Workflow bundle size 4.51 MB 4.53 MB +0.02 MB
Workflow bundle compilation 992 ms 986-999 ms No meaningful change
Compose start to API ready 25.823 s 26.425-27.795 s No meaningful change

This smaller catalog produces smaller absolute savings than the production catalog, but independently confirms the same effect on the current Temporal SDK and rebased implementation.

Not tested: restart during an in-flight activity; a prod-sized mixed cache.

🤖 Generated with Claude Code

Comment thread sdk/core/src/worker/loader/workflows.js Resolved
@github-actions

This comment was marked as outdated.

derencius and others added 3 commits October 30, 2026 16:53
A static re-export of every workflow makes each cached workflow evaluate the whole catalog in Temporal's reusable VM. Each workflow export is now an async wrapper that dynamic-imports its module in eager mode (same bundle); the catalog workflow stays static.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@szanata
szanata force-pushed the perf/lazy-workflow-entrypoint branch from 02b3c9b to d2672e2 Compare October 30, 2026 20:44
Comment thread sdk/core/src/worker/loader/workflows.js Resolved
@github-actions

This comment was marked as outdated.

@github-actions

Copy link
Copy Markdown
Contributor

PR review

Verdict

✅ PASS

Findings

None.

Categories

  • Design: ✅ PASS
  • Quality: ✅ PASS
  • Correctness: ✅ PASS
  • Documentation: ✅ PASS
  • Changeset: ✅ PASS
  • Tests: ✅ PASS
  • Security: ✅ PASS
  • Compatibility: ✅ PASS

@szanata
szanata merged commit 606278c into main Oct 30, 2026
7 checks passed
@szanata
szanata deleted the perf/lazy-workflow-entrypoint branch October 30, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants