fix(google-adk-agents): Node workflow loading, OTel composition, and telemetry replay-safety tests - #2276
Open
DABH wants to merge 9 commits into
Open
Conversation
The workflow bundle crashed at load on Node (Bun masked both crashes by exposing web globals): - @google/genai's web build dereferences ReadableStream at module load, before user imports could evaluate the polyfill barrel. Prepend the load-polyfills module to workflowInterceptorModules so it evaluates per workflow, before the user's workflow module, regardless of import order. - @google/adk's utils/client_labels.js runs new AsyncLocalStorage() from node:async_hooks at module load. Redirect async_hooks to a shim re-exporting the sandbox-injected AsyncLocalStorage global.
@opentelemetry/sdk-trace-base and @opentelemetry/resources were aliased to empty modules for the whole workflow bundle, so composing this plugin with @temporalio/interceptors-opentelemetry failed every workflow task with 'tracing.BasicTracerProvider is not a constructor' — disabling the SDK's replay-gated workflow span path (the only egress for ADK's gcp.vertex.agent spans) for all workflows on the worker. Both packages are pure JS and load-safe in the sandbox; keep them real.
Run a two-turn agent workflow composed with OpenTelemetryPlugin and the workflow cache disabled (every workflow task replays the whole history): the workflow must complete and export exactly one gcp.vertex.agent span per real operation — replays add none. Also pin that without the OpenTelemetry plugin, sandbox spans never reach a process-global tracer provider.
- Fix README default-case wording: it's the absent tracer provider, not an impossibility of running an OTel SDK in the sandbox, that drops ADK spans. - Add README caution that workflow task retries (unlike replays) re-emit spans, so export is at-least-once. - Pin in plugin unit tests that the pure-JS OTel packages stay unstubbed and that load-polyfills is prepended to workflowInterceptorModules. - Document the converter-modules-evaluate-first gap in the bundler recipe.
Un-stubbing @opentelemetry/sdk-trace-base exposed @opentelemetry/core's browser build, which dereferences the performance global at module load. ESM workflow files dodge that chain through harmony-import pruning, but tsc-compiled CommonJS workflow files — including the published lib/ artifacts and converter modules importing @google/adk — evaluate it eagerly, failing every workflow task with 'ReferenceError: performance is not defined'. Install a deterministic performance shim (mapped onto the sandbox-patched Date.now, mirroring interceptors-opentelemetry's workflow runtime shim) in load-polyfills, and add E2E regression tests for the compiled-CJS workflow layout and the documented converter-module workaround.
@google/adk pins an exact @opentelemetry/api version, so the Workflow bundle can contain two api copies. ADK's telemetry/tracing.js caches trace.getTracer() at module load; when a user interceptor or converter module evaluates @google/adk before the OTel interceptor factories register the sandbox tracer provider, that tracer binds the other copy's never-delegated provider and every ADK span is silently dropped while the interceptor's own spans keep exporting. Rewrite all bundle requests for @opentelemetry/api to ADK's own resolution in the sandbox-compat webpack plugin, pin the rewrite with a unit contract test, and add an E2E test that evaluates ADK from a user workflowModules entry and asserts exact span counts.
Move the converter-module workaround into the README's telemetry cautions (the PR description already pointed there), note the polyfill loader now includes the performance shim, document the single @opentelemetry/api copy pinning, and extend the google-adk-agents changelog entry with the replay-safe span-export composition.
A workflow task retry re-executes its segment and legitimately re-emits spans through the replay-gated sink, so exact-count assertions could flake on slow runners; degrade to a lower bound when the history shows WorkflowTaskTimedOut/Failed events. Also make the performance-shim comment precise about the interceptor package's unconditional shim.
DABH
force-pushed
the
adk-telemetry-replay-safety
branch
from
August 1, 2026 09:02
becf512 to
c38dfd4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2120 (targets
maplexu/google-adk-agents-contrib) — fixes issues found during a telemetry replay-safety assessment of the ADK integration.What was changed
Fix Node workflow-load crashes (currently failing all 8 Node CI jobs on feat(google-adk-agents): add @temporalio/google-adk-agents package #2120; Bun passes only because it exposes web streams natively).
BundleOptions.workflowInterceptorModulesinconfigureBundler, so it evaluates per-workflow after the activator is set and before user workflow code — regardless of user import order. An entry-preload approach was deliberately avoided: entry code evaluates before any activator exists, and in reusable-V8-context mode (the default) that no-op evaluation would be cached permanently, so polyfills would never install.node:async_hooksis shimmed to the sandbox's own workflow-scopedAsyncLocalStorage(same approach ascontrib/langsmith), fixing@google/adk's top-levelnew AsyncLocalStorage(). This is strictly stronger than ADK's own browser fallback shim (context survives awaits) and is deterministic with sandbox-managed lifetime.Stop stubbing
@opentelemetry/sdk-trace-baseand@opentelemetry/resourcesbundle-wide. Both are pure-JS and sandbox-safe; stubbing them broke composition with@temporalio/interceptors-opentelemetry/OpenTelemetryPlugin— the SDK's only replay-safe workflow span path — failing every workflow task withtracing.BasicTracerProvider is not a constructor, despite the README advising users to compose observability plugins with this one. The node-only OTel packages (exporters,sdk-trace-node,sdk-metrics,sdk-logs, resource detectors) remain stubbed, so ADK'stelemetry/setup.jsstill cannot create an un-gated egress inside the sandbox.Telemetry regression tests (
telemetry.test.ts+plugin.test.tscontract pins):adkSpansExportOncePerOperationUnderReplay: two-turn agent workflow with[OpenTelemetryPlugin, GoogleAdkPlugin]andmaxCachedWorkflows: 0; asserts the history contains ≥3 workflow tasks (replays actually happened) and the in-memory exporter holds exactly 2×call_llm, 2×invocation, 2×invoke_agent assistantfor scopegcp.vertex.agent— one span per real operation, none added by replays. This pins the property that ADK's workflow-side spans export replay-safely through the sink gate, and fails if anything regresses toward the N+1 over-counting we found (and fixed) in the Go and Python ADK adapters (contrib/googleadk: add replay-safe OpenTelemetry provider wrappers sdk-go#2514, Add replay-safe OpenTelemetry meter provider; warn in Google ADK plugin sdk-python#1710).adkSpansDoNotLeakToProcessGlobalProvider: a worker-process global tracer provider observes zerogcp.vertex.agentspans — pins the isolate boundary the replay-safety rests on.README: Telemetry & observability section. ADK's
gen_aispans (includinggen_ai.usage.*token attributes) are created inside the workflow sandbox and are dropped by default; installing the OpenTelemetry interceptors exports them replay-safely (first-execution-only). Explicit cautions: don't register custom telemetry sinks withcallDuringReplay: true(reintroduces over-counting), and span export is at-least-once (workflow task retries — unlike replays — can re-emit). Documents the known gap that custom payload/failure converter modules evaluate before interceptor modules, with the workaround.Why
ADK records telemetry via the global OpenTelemetry API from code that runs workflow-side. In the Go and Python SDK adapters this caused verified replay over-counting (1 real execution + N replays → N+1 observations of token usage/latency). The TypeScript architecture is immune to that by construction — the isolate-local OTel API plus replay-gated sinks yield exactly one export per real operation — but as written the integration instead lost all agent-loop telemetry silently, and hard-broke the one supported way to get it out. This PR fixes the composition, makes the Node CI failures go away, and pins the replay-safety property with tests so it stays true.
Testing
Full contrib suite on Node v26.5.0 against a real dev server: 51 tests passed (previously the E2E files crashed at workflow load with
ReadableStream is not defined). Causality verified by stashing the plugin fixes and reproducing the exact CI failure. ESLint and Prettier clean.