KHR_interactivity rework#18455
Draft
SergioRZMasson wants to merge 33 commits into
Draft
Conversation
…ctivity-refactor Squashes 44 commits of KHR_Interactivity extension work into a single commit for a clean history. Also adds specs/interactivity-refactor to .gitignore and removes it from source control. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The KHR_interactivity spec now defines a first-class 'ref' value type
(opaque reference). Pointer templates may use both '[name]' and '{name}'
brackets, and the new test asset mixes both conventions including
'{name}' wrapping plain integer values. The previous int-only template
substitution rejected the new asset before any sub-test could run.
Changes:
- gltfTypeToBabylonType: add 'ref' (-> FlowGraphTypes.String, default '').
- ValueSignature / ValueType: allow 'ref' and string values.
- FlowGraphPathConverterComponent: accept both bracket styles, register
templated inputs as RichTypeAny, and dispatch substitution on the
runtime value type (FlowGraphInteger/number -> decimal int; string ->
JSON-Pointer segment matching the placeholder position; otherwise
throw with the input name).
- declarationMapper: remap flow/setDelay output 'lastDelay' to the
existing 'lastDelayIndex' socket and flow/cancelDelay input 'delay'
to 'delayIndex' so spec-renamed sockets keep working without
touching the runtime blocks.
- Repoint the Overview.glb and batch integration tests at the new
local 'Opaque references' asset folder.
Result: 168 sub-tests passing / 17 failing on the new Overview.glb
(prior baseline on the old asset was 166/17). Remaining failures are
pre-existing morph-target-weight and KHR_texture_transform setter gaps,
not regressions from this change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…mporter-v2 # Conflicts: # .build/config.json # .github/copilot-instructions.md # .github/instructions/index.md # .github/instructions/visual-tests.instructions.md # package-lock.json # package.json # packages/dev/inspector-v2/src/cli/bridge.ts # packages/dev/inspector-v2/src/cli/cli.ts # packages/dev/inspector-v2/src/index.ts # packages/dev/inspector-v2/src/inspectable.ts # packages/dev/inspector-v2/src/inspector.tsx # packages/dev/inspector-v2/src/services/cli/entityQueryService.ts # packages/dev/inspector-v2/src/services/cli/perfTraceCommandService.ts # packages/dev/inspector-v2/src/services/cli/screenshotCommandService.ts # packages/dev/inspector-v2/src/services/cli/shaderCommandService.ts # packages/dev/inspector-v2/src/services/cli/statsCommandService.ts # packages/dev/inspector-v2/src/services/cliConnectionStatusService.tsx # packages/dev/inspector-v2/test/unit/cli/bridge.test.ts # packages/dev/inspector-v2/test/unit/cli/cli.test.ts # packages/dev/inspector-v2/test/unit/inspectable.test.ts # packages/dev/loaders/src/glTF/2.0/Extensions/KHR_materials_transmission.ts # packages/dev/lottiePlayer/src/animationConfiguration.ts # packages/dev/lottiePlayer/src/maths/boundingBox.ts # packages/dev/lottiePlayer/src/parsing/spritePacker.ts # packages/dev/lottiePlayer/src/rendering/animationController.ts # packages/dev/sharedUiComponents/src/modularTool/modularTool.tsx # packages/dev/sharedUiComponents/src/modularTool/modularity/serviceContainer.ts # packages/public/@babylonjs/inspector-v2/readme.md # packages/public/@babylonjs/inspector-v2/tsconfig.cli.json # packages/tools/devHost/src/lottie/main.ts # packages/tools/tests/test/visualization/config.lottie.json
Adds the seam needed for KHR_interactivity refs that point at scene objects not described by the source glTF (e.g. refs emitted by future engine-side event blocks), without changing how spec-standard refs resolve. Architecture: - New CompositePathToObjectConverter: prefix-dispatching IPathToObjectConverter. Tries each registered prefix in order, falls back to a primary converter. Generic over the accessor type. - New BabylonScenePathToObjectConverter: resolves paths in the /extensions/BABYLON_scene_objects/<collection>/<uniqueId>/<property> namespace against the live Scene. Initial leaves cover transformNodes (translation, rotation, scale, matrix, globalMatrix, name), meshes (name, visible), and materials (name); more can be added without touching the path-converter or FlowGraphJsonPointerParserBlock. - KHR_interactivity loader now wraps the existing GLTF converter in a composite and registers the Babylon-scene converter under the BABYLON_scene_objects prefix. Standard refs (/nodes/..., /materials/..., /extensions/KHR_*/...) continue to fall through to the GLTF converter unchanged. FlowGraphJsonPointerParserBlock and the FlowGraphPathConverterComponent template substitution are untouched: the choice of which converter handles a given ref is an attribute of the path string, not the block. Test result on the new Overview.glb: 168 sub-tests passing / 17 failing, identical to the pre-change baseline. Also includes three small merge cleanups required to get the test environment running again after the master merge: - babylonServer webpack: historyApiFallback now uses disableDotRule:true so the existing /foo.js -> /foo.min.js rewrites still apply to Accept: */* script requests under the newer webpack-dev-server. - inspector-v2 CLI services: three stale ../../modularity/serviceDefinition imports from master repointed at the new shared-ui-components/modularTool/modularity/serviceDefinition path. - khrInteractivity integration test: bumped beforeAll timeout from 30s to 120s for cold dev-bundle loads. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The original squash branch carried 6 files for an older Inspector v2 CLI WIP that was independently superseded in master by PRs BabylonJS#18229, BabylonJS#18271, BabylonJS#18332, and BabylonJS#18337. Master never had these files; their import-path fixes in d204462 only existed to keep them building and were unrelated to the KHR_interactivity work. Removed: - packages/dev/inspector-v2/src/cli/protocol.ts - packages/dev/inspector-v2/src/services/cli/cliConnectionStatus.ts - packages/dev/inspector-v2/src/services/cli/inspectableBridgeService.ts - packages/dev/inspector-v2/src/services/cli/inspectableCommandRegistry.ts - packages/dev/inspector-v2/test/unit/modularity/serviceContainer.test.ts - packages/dev/inspector-v2/test/unit/services/inspectableBridgeService.test.ts These files have no consumers in master code (the live cliConnectionStatusService.tsx imports from shared-ui-components/modularTool/services/cli/bridgeConnectionStatus instead). Also fixes a strict-mode TS error in babylonScenePathToObjectConverter (narrowing IBabylonSceneObjectModelTree to a Record<string, ...> needs a double cast through unknown). Overview.glb regression: still 168 passing / 17 failing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Picks up the spirit of Gary Hsu's gltf-animation-pointer-morph-targets work (gary/gltf-animation-pointer-morph-targets, commit 4a5d193) and adapts it to the KHR_interactivity test surface in the new Overview.glb, where morph weights are queried in three structural shapes: 1. /nodes/<host>/weights/<i> (host node directly bears the mesh) 2. /nodes/<parent>/weights/<i> (parent has no mesh; morphs live on a descendant via children) 3. /nodes/<no-mesh-node>/weights/... (validity must be reported) Changes: - New helper _findNodeMorphTargets(node) that returns both the active MorphTargetManager AND every Babylon mesh that shares it. It walks Babylon scene-graph descendants when the queried node has no direct mesh, then groups sibling primitives so writes fan out across all meshes that share the manager (Gary's pattern). - weights.length: always reports a valid value (numTargets when a host is reachable, 0 otherwise) — KHR_interactivity test asserts isValid remains true for length even on nodes without meshes. - weights[i] (single-element accessor): returns the influence rounded via Math/parseFloat at float32 precision so the read-back matches the literal compared by math/eq. glTF tooling routinely serialises 0.1 as 0.10000000149011612 (the float32 round-trip), and strict math/eq against the source's plain 0.1 was the cause of the static/mesh-and-node value-equality failures. Out-of-range index on a node that has its own mesh or reachable morph targets returns 0 (valid); only nodes with no mesh anywhere return undefined (invalid). - weights set: fans out across every primitive sharing the manager so multi-primitive meshes stay in sync. - weights array accessor: same descendant lookup + rounding for the whole-array read. The remaining single morph failure ('nonStatic weights[0] (value == 0.5)') expects 0.5 with no GLB-side setup that produces it; that subtest depends on an external host-driven weight which the asset itself does not encode and is therefore not addressable from the loader. Overview.glb regression: 177 sub-tests passing / 8 failing (was 168 / 17 before this commit). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- objectModelMapping: widen the per-element BabylonTargetType on
weights.__array__ to 'any' so the accessor's getTarget can return
either a TransformNode (out-of-range / no morph host) or a
MorphTarget (in-range hit) without breaking
IObjectAccessor's narrow type signature. The TS error was:
Type '(node, index?) => TransformNode | MorphTarget | undefined'
is not assignable to type
'(target, index?, payload?) => TransformNode | undefined'.
- objectModelMapping, flowGraphPathConverterComponent,
babylonScenePathToObjectConverter, compositePathToObjectConverter:
add JSDoc @param/@returns to internal helpers, replace single-line
if/return with braced bodies (eslint 'curly'), and break overly long
conditionals to satisfy prettier line length. No behaviour change.
Full repo lint:check and format:check now both pass.
Overview.glb regression: still 177 passing / 8 failing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e transform
Material 21 in the new Overview.glb has KHR_materials_volume.thicknessTexture
defined in the glTF JSON but no thicknessFactor. The volume loader extension
early-returns under that condition (since volume effects can't apply without
thickness), leaving subSurface.thicknessTexture null on the Babylon material.
KHR_interactivity then issues pointer/set on
'/materials/21/extensions/KHR_materials_volume/thicknessTexture/extensions/KHR_texture_transform/{offset,rotation,scale}'
followed by pointer/get for a round-trip check. Without a Babylon-side
texture, the writes silently no-op and the read-back returns the type's
default — math/eq fails, the test asserts 'can't be set'.
This commit teaches GenerateTextureMap to optionally take a glTF-side path
descriptor (extensionKey + texturePath). When the Babylon texture is
absent, the accessors read/write KHR_texture_transform under
material.extensions.<ext>.<texturePath...> in the source glTF JSON,
creating intermediate objects on demand for writes. The Babylon side is
still updated when the texture exists, so cases where the loader did
materialise the texture (e.g. with a thicknessFactor present) keep their
existing behaviour.
KHR_materials_volume.thicknessTexture is wired with
{ extensionKey: 'KHR_materials_volume', texturePath: ['thicknessTexture'] }
to enable the fallback. This is the same pattern the user described —
'a more advanced way to access that data based on the JSON pointer'
analogous to the morph-target descendant-walk fix.
Test results:
- Sandbox load (Overview.glb, 8s wait): 180/10 -> 183/7
- Playwright integration (3s wait): 177/8 -> 180/5
The 3 KHR_materials_volume/thicknessTexture {offset,rotation,scale} cases
flip from FAIL to PASS in both. No regressions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…orms Material 20 in Overview.glb has KHR_materials_transmission.transmissionTexture defined in glTF JSON but no transmissionFactor; KHR_materials_transmission's loader extension early-returns when transmissionFactor is 0 / undefined, leaving subSurface.refractionIntensityTexture null on the Babylon material and causing pointer/get + pointer/set tests on the texture transform to fail. Wires the same glTF-side fallback added for KHR_materials_volume to: - KHR_materials_transmission.transmissionTexture - KHR_materials_diffuse_transmission.diffuseTransmissionTexture - KHR_materials_diffuse_transmission.diffuseTransmissionColorTexture The first directly fixes the three failing transmission-texture transform tests (offset, rotation, scale). The other two are wired pre-emptively for the same shape of issue if it shows up in future test assets — the loader extensions for diffuse-transmission have similar early-return conditions when their factor is undefined. Test results: - Sandbox load (Overview.glb, 8s wait): 183/7 -> 186/4 - Playwright integration (3s wait): 180/5 -> 183/2 The 3 KHR_materials_transmission/transmissionTexture tests flip from FAIL to PASS in both. The 2 remaining Playwright failures are the unfixable 'nonStatic morph weight == 0.5' and 'KHR_materials_anisotropy/anisotropyRotation' (separate angle-conversion bug). Sandbox additionally still flags the two time-sensitive flow tests (setDelay, variable/interpolate). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ates
Calculator.glb (and similarly authored assets) use a relative pointer template (e.g. `extensions/KHR_node_visibility/visible`) paired with a static ref-typed value socket holding the `/nodes/<i>/` prefix. The standard substitution algorithm only handles `{name}`/`[name]` template parameters and would leave the relative path intact, so the pointer becomes invalid and the ref socket has nowhere to wire on the FlowGraph block (`Could not find data input with name nodeRef`).
Detect this case at parse time in `InteractivityGraphToFlowGraphParser._parseNodes` and splice the static ref's literal value onto the front of the pointer template, then drop the now-baked socket from `node.values` so it isn't wired at the connection step. Dynamic refs (those sourced from another node's output) are intentionally left untouched and would still surface the parse-time error if encountered.
Verified: Calculator.glb buttons now react to clicks (e.g. selecting Button 2 advances the digit display via texture uOffset). Overview.glb integration regression unchanged at 2 pre-existing failures (nonStatic morph weight==0.5, KHR_materials_anisotropy/anisotropyRotation).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Opaque-Reference spec adds a single first-class reference operation, `ref/eq`, defined as: true if both inputs are null refs, true if both refer to the same object (whether or not it exists), false otherwise. Models such as Events.glb and SnailRace.glb use `ref/eq` extensively to dispatch on the picked-mesh ref delivered by `event/onSelect` and `event/receive`. FlowGraphEqualityBlock already falls through to a strict `===` comparison for non-vector/matrix/numeric types, which produces the spec-defined behaviour for Babylon object refs (mesh, material, etc.) and for null. Map `ref/eq` to `FlowGraphBlockNames.Equality` alongside `math/eq`. Verified: Events.glb Set Color and Set Pose buttons now react (CubeMaterial.albedoColor and Cube.position/scale change). SnailRace.glb GO-Button now causes Snail_02 to advance. Overview.glb regression unchanged at 2 pre-existing failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…+get round-trip Babylon stores `PBRAnisotropicConfiguration.angle` as the direction Vector2 `(cos θ, sin θ)`, so the getter does `atan2(sin, cos)` and any input outside `[-π, π]` is collapsed on the round-trip. The KHR_interactivity Overview.glb conformance test sets `anisotropyRotation = 30` (radians) and expects `pointer/get` to return `30`; with the previous accessor it returned `≈-1.416`. Cache the raw value per-material in a WeakMap inside `objectModelMapping.ts` and have both the accessor's `set` and the `KHR_materials_anisotropy` importer record into it. The accessor's `get` returns the cached raw value when present and falls back to `mat.anisotropy.angle` otherwise. The direction Vector2 is still updated normally so rendering is unaffected. Verified: Overview.glb integration regression now reports 184 passed / 1 failed (down from 2; only the long-standing morph-weight==0.5 case remains). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nter/set+get round-trip" This reverts commit 4073366.
Add `FlowGraphMathSlerpBlock` (delegates to `Quaternion.Slerp`, whose algorithm matches the spec step-for-step) and map `math/quatSlerp` to it in the declaration table. Without this op the parser returned `undefined` for the block and downstream code crashed at `interactivityGraphParser.ts:432` with `Cannot read properties of undefined (reading 'dataInputs')` on assets such as Sundial.glb. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Opaque-Reference spec update adds `selectedNode` (a ref to the picked Babylon mesh) to the output sockets of `event/onSelect`. The existing mapping only exposed `selectedNodeIndex` (the int index via `IndexOf`); without `selectedNode` the parser failed with `Could not find data output with name selectedNode in block FlowGraphMeshPickEventBlock` on Sundial.glb. Wire the new socket directly to `FlowGraphMeshPickEventBlock.pickedMesh` (no IndexOf needed). The legacy `selectedNodeIndex` socket is kept for backwards compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When a templated pointer input (e.g. `/nodes/{nodeRef}/globalMatrix`) receives a Babylon object instead of a JSON-Pointer string or integer (which is what `event/onSelect.selectedNode` emits), look up the canonical glTF JSON Pointer the loader stamped on the object via `_internalMetadata.gltf.pointers` and feed it through the existing ref-segment-extraction path.
Pick the pointer whose root segment matches the segment in the template that immediately precedes the placeholder. A single-primitive Babylon mesh holds both `/nodes/<i>` and `/meshes/<j>/primitives/<k>`; matching against the template prefix (`nodes` for `/nodes/{nodeRef}/...`) ensures we splice the node index, not the mesh-primitive index.
Verified: Sundial.glb interactive sun-positioning works end to end. Overview.glb integration regression unchanged at 183 passed / 2 failed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Opaque-Reference spec update added new `CoreReadOnlyPointers` test cases that exercise read-only introspection of the glTF tree topology (`/nodes/{}/camera`, `/nodes/{}/mesh`, `/nodes/{}/skin`, `/nodes/{}/parent`, `/nodes/{}/children/{}`, `/scenes/{}/nodes/{}`, `/skins/{}/joints/{}`, `/skins/{}/skeleton`, `/meshes/{}/primitives/{}/material`).
These accessors already existed but returned raw integer indices; the spec requires them to return ref-typed JSON Pointer strings (e.g. `/cameras/0/`) so that `ref/eq` and other ref-aware operations can compare them as references. Convert each accessor's `get` to emit a ref string when the underlying index is defined and an empty string (the spec's null-ref convention) otherwise. Update the matching `IObjectAccessor` type parameters from `number` to `string`.
These read-only paths are not used by KHR_animation_pointer (verified by inspection of KHR_animation_pointer.data.ts) so the type change is safe.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
KHR_interactivity `ref/eq` (mapped to `FlowGraphEqualityBlock` since the spec defines it as `true if both refer to the same object`) now treats two JSON-Pointer-shaped strings as equal even when one ends with `/` and the other does not (e.g. `/nodes/420` vs `/nodes/420/`). Different asset authors emit refs in both styles for the same object. The normalisation is gated on both sides starting with `/` so plain string comparisons are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nd onHoverOut Same shape as the recent `selectedNode` fix for `event/onSelect`: the Opaque-Reference spec update added a `hoveredNode` ref-typed output to `event/onHoverIn` and `event/onHoverOut` (in addition to the legacy integer `hoverNodeIndex`). Without the mapping the parser failed with `Could not find data output with name hoveredNode in block FlowGraphPointerOverEventBlock` (or `FlowGraphPointerOutEventBlock`) when loading assets such as Ghost.glb. Wire `hoveredNode` directly to `meshUnderPointer` (over) / `meshOutOfPointer` (out) on the corresponding pointer-event blocks. The legacy `hoverNodeIndex` socket is kept for backwards compatibility. Verified: Ghost.glb now loads without errors. Overview.glb integration regression unchanged at 193 passed / 1 failed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The sandbox loads its Babylon dependencies (engine, loaders, GUI, inspector, etc.) from a separate origin: in production, `preview.babylonjs.com`; in local dev, the babylon-server CDN on a different port (e.g. `localhost:1337` while the sandbox itself is on `:1339`). Without `crossorigin=�nonymous` on the `<script>` tag, the browser strips uncaught exceptions to the literal string `Script error.` before they reach the sandbox's `window.onerror` handler, hiding the real error from both the developer console and the red error overlay. Both CDNs (and the dev fallback) already serve `Access-Control-Allow-Origin: *`, so adding `script.crossOrigin = 'anonymous'` to `loadScriptAsync` (and to the dev fallback path) lets the browser surface the real exception details — matching how the Babylon Playground already loads its scripts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… ArrayIndex The Opaque-Reference spec update redefined `animation/start.animation` (and `animation/stop.animation`) from `int` to `ref`. The new asset uses `pointer/get` of `/animations/[index]/` (trailing-slash form) to obtain a ref to the animation, then feeds that ref into `animation/start`. Our existing wiring routes the input through `FlowGraphArrayIndexBlock` which expected an integer; with no accessor for `/animations/<i>/` the pointer/get returned `undefined`, and `ArrayIndex._updateOutputs` crashed with `Cannot read properties of undefined (reading 'value')` inside `getNumericValue`. Two surgical fixes: 1. `objectModelMapping.ts` — populate `animationsTree.__array__` so `/animations/<i>/` resolves to the JSON-Pointer ref string `/animations/<i>/` (with the AnimationGroup as the access target). Mirrors the readonly-pointer accessors added previously for cameras, meshes, skins, scenes, etc. 2. `FlowGraphArrayIndexBlock` — guard against `undefined` inputs (return null instead of throwing) and recognise ref-typed string indices (`/foo/<n>/`) by extracting the trailing integer segment, so the same block can handle both the legacy int form and the new ref form without a rewiring. Verified: Ghost.glb now loads without the `Script error. Check the developer console.` red overlay. Overview.glb integration regression unchanged at 193 passed / 1 failed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… correctly Two fixes that together let `pointer/get` of an animation ref (e.g. `/animations/0/`) return the correct JSON-Pointer ref string so `animation/start` actually receives an AnimationGroup: 1. `gltfPathToObjectConverter.convert` now strips a single trailing empty segment before walking the path. The Opaque-Reference spec uses the trailing slash as a convention for `ref to the resource itself` (`/animations/0/`, `/cameras/0/`, `/materials/0/` etc.). Without this strip, the converter would try to descend into a non-existent empty-named child of the resource and bail out, returning `undefined` from the pointer/get block. 2. The animations-tree `__array__` accessor now derives its ref from the IAnimation's own `index` property (populated by the loader's ArrayItem.Assign step) instead of relying on a runtime index payload that the converter does not pass on the array-traversal branch. The previous accessor's `index` parameter was always undefined, so it returned an empty string. Verified: Ghost.glb's hover-triggered animations now play (the `scared` AnimationGroup advances from frame 4.6 to 8.9 in 100 ms after entering hover). Overview.glb integration regression unchanged at 193 passed / 1 failed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Calculator.glb established the convention of pairing a relative `pointer` template (e.g. `extensions/KHR_node_visibility/visible`) with a static `nodeRef` ref-typed value socket whose literal value supplies the absolute prefix. `_bakeRelativePointerPrefix` already handled that case by splicing the literal into the template at parse time and dropping the socket.
MagicBall.glb extends the same convention but uses a **dynamic** `nodeRef` — the socket is connected to the output of a `math/switch` block that picks one of ten ref values at runtime. The previous helper ignored connected sockets and the parser later threw `Could not find data input with name nodeRef in block FlowGraphJsonPointerParserBlock`.
Extend `_bakeRelativePointerPrefix` to recognise this dynamic-ref shape: when the socket is a `{ node, socket }` connection (not a `{ value }` literal), rewrite the relative template to `/nodes/{<socketName>}/<original-template>` and leave the socket entry in place. The `FlowGraphPathConverterComponent` then registers a templated data input with that name, the existing connection-wiring step hooks it up to the upstream output, and the runtime substitution machinery extracts the matching JSON-Pointer segment from the ref string at execute time.
Verified: MagicBall.glb now loads and reacts to clicks (each click drives the fortune-words visibility flow via the interactivity graph). Overview.glb integration regression unchanged at 193 passed / 1 failed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… too When the loader processes a glTF node that has `KHR_node_visibility.visible = false`, `onReady` only set `isVisible = false` on the wrapping `babylonTransformNode`. For primitive-bearing nodes that's a non-rendering `TransformNode` whose primitive children remain visible — so assets that author hidden defaults (e.g. MagicBall.glb's twenty FortuneWords nodes) appeared with all options visible at scene start, even though clicks correctly drove the visibility flow. Mirror the runtime `set` accessor in `objectModelMapping`-style: also iterate `node._primitiveBabylonMeshes` and set `inheritVisibility = true` plus `isVisible = false` on each one. After this, MagicBall starts with all fortune words hidden and clicks correctly toggle exactly one visible fortune at a time, matching the asset author's intent. Verified: MagicBall.glb's initial state now shows no fortune-word meshes; clicking Ball.001 reveals one randomly. Overview.glb integration regression unchanged at 193 passed / 1 failed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…to FlowGraphInteger
Some Object Model accessors return raw integers as plain JavaScript `number` (e.g. `/nodes/{}/children.length` in `objectModelMapping.ts` returns `children?.length ?? 0`). When such an integer flows into `type/intToFloat` — which previously hard-required a `FlowGraphInteger` and read `a.value` — the conversion silently returned `undefined`. Downstream blocks then defaulted to 0, and any vector built around the result became `(0,0,0)`. Once that vector divided another by it, the whole chain became NaN, propagating into per-frame state and corrupting the Babylon transform nodes.
This is exactly what made Flocking.glb appear to render nothing: each tick, every fish's position update went through `divide(accumulatedAlignment, combine3(intToFloat(neighborCount)))`, so the very first tick produced `(0,0,0)/(0,0,0) = (NaN,NaN,NaN)` and every fish position became NaN forever.
Make `FlowGraphIntToFloat` accept either shape: if the input is a plain `number` return it as-is, otherwise read `.value` (the FlowGraphInteger field). The output type stays `number` (float). No regression risk for callers that already pass a FlowGraphInteger.
Verified: Flocking.glb's 25 fish now move and flock as expected (positions update each frame). Overview.glb integration regression unchanged at 193 passed / 1 failed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tivity) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Match master's line ending convention so the diff reflects only real functional changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Revert .gitignore to master (specs/interactivity-refactor/ was local only) - Delete .github/instructions/playground-workflow.instructions.md (not part of this feature scope) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These tests load assets from a fixed local path on the developer's machine; they will be re-added separately once the asset-loading strategy is finalized. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
|
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
Collaborator
|
Snapshot stored with reference name: Test environment: To test a playground add it to the URL, for example: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18455/merge/index.html#WGZLGJ#4600 Links to test your changes to core in the published versions of the Babylon tools (does not contain changes you made to the tools themselves): https://playground.babylonjs.com/?snapshot=refs/pull/18455/merge To test the snapshot in the playground with a playground ID add it after the snapshot query string: https://playground.babylonjs.com/?snapshot=refs/pull/18455/merge#BCU1XR#0 If you made changes to the sandbox or playground in this PR, additional comments will be generated soon containing links to the dev versions of those tools. |
Collaborator
Collaborator
|
WebGL2 visualization test reporter: |
Collaborator
|
Visualization tests for WebGPU |
Collaborator
🟢 Memory Leak Test Results13 passed, 0 leaked out of 13 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (13)
|
Collaborator
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
The relative-pointer-prefix bake was iterating every value socket on a pointer/* node and matching the FIRST one with either a literal ref string or a dynamic connection. For Calculator and Puzzle the value socket appears before the nodeRef socket in the gltf JSON, so the bake produced an invalid template like '/nodes/{value}/extensions/KHR_node_visibility/visible' and the resulting FlowGraphJsonPointerParserBlock had no 'value' input to land on, causing a parse-time error.
Skip non-ref sockets by checking: (a) literal value starting with '/'; (b) declared type maps to FlowGraphTypes.String (the ref alias); or (c) name ends with 'Ref' per spec convention (nodeRef, materialRef, etc.). Other sockets (notably 'value') are now ignored when looking for the prefix to bake.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
After the Vite migration in master the sandbox only imported 'loaders/glTF/2.0/glTFLoader' (the base loader) and never the Extensions/index re-export, so no KHR_/EXT_ extension self-registered via registerGLTFExtension. Any glb with KHR_interactivity/KHR_node_visibility/etc. silently lost its extension processing. Import the glTF 2.0 index instead so every extension's side-effect runs. Separately, the LoadingScreen side-effect was no longer reachable through the Vite dev graph, so engine.loadingUIBackgroundColor and engine.hideLoadingUI threw 'LoadingScreen needs to be imported before' at runtime. Add the explicit side-effect import in sandbox.tsx. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
|
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
Satisfies the docs ratchet CI check that flagged the new and pre-existing public exports added/touched by this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
|
Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). |
Collaborator
|
Snapshot stored with reference name: Test environment: To test a playground add it to the URL, for example: https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/18455/merge/index.html#WGZLGJ#4600 Links to test your changes to core in the published versions of the Babylon tools (does not contain changes you made to the tools themselves): https://playground.babylonjs.com/?snapshot=refs/pull/18455/merge To test the snapshot in the playground with a playground ID add it after the snapshot query string: https://playground.babylonjs.com/?snapshot=refs/pull/18455/merge#BCU1XR#0 If you made changes to the sandbox or playground in this PR, additional comments will be generated soon containing links to the dev versions of those tools. |
Collaborator
|
You have changed file(s) that made possible changes to the sandbox. https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/SANDBOX/refs/pull/18455/merge/ |
Collaborator
|
You have made possible changes to the playground. https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/PLAYGROUND/refs/pull/18455/merge/ The snapshot playground with the CDN snapshot (only when available): Note that neither Babylon scenes nor textures are uploaded to the snapshot directory, so some playgrounds won't work correctly. |
Collaborator
Collaborator
|
Visualization tests for WebGPU |
Collaborator
|
WebGL2 visualization test reporter: |
Collaborator
⚡ Performance Test Results🟢 All performance tests passed — no regressions detected. |
Collaborator
🟢 Memory Leak Test Results13 passed, 0 leaked out of 13 scenarios 🟢 All memory leak tests passed — no leaks detected. Passed Scenarios (13)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
KHR_Interactivity importer: opaque-reference spec update + bug fixes
This PR brings the Babylon.js glTF loader's
KHR_interactivityextension in line with the latest spec (the Opaque Reference /ref-type update) and fixes a backlog of correctness issues that prevented the official Khronos sample assets from running. With these changes:glTF-Test-Assets-Interactivityrepo (ref-type-spec-changesbranch) load and behave as authored — both in the Sandbox and the Playground.The changes group naturally into three buckets — FlowGraph core, glTF Object-Model mapper, and KHR_interactivity importer — plus a small set of supporting interface/test additions.
1. FlowGraph (core)
These changes are in
packages/dev/core/src/FlowGraph/and are independent of the glTF importer — they fix or extend the runtime block library and the path-template converter that all FlowGraph users share.flowGraphPathConverterComponent.ts{name}(ref/string) from[name](int) template parameters; dispatches on the value's type at runtime —FlowGraphInteger/number→ decimal index, JSON-Pointer string → segment extraction, Babylon scene object → look up_internalMetadata.gltf.pointers[]and pick the entry whose root segment matches the template prefix. Templates can now bake in a static ref prefix at parse time.Blocks/Data/Math/flowGraphMathBlocks.tsFlowGraphMathSlerpBlock(delegates toQuaternion.Slerp) for themath/quatSlerpop.FlowGraphEqualityBlocknow normalises trailing slashes when comparing ref-typed strings so equality matches spec semantics.Blocks/flowGraphBlockNames.ts,Blocks/flowGraphBlockFactory.tsMathSlerpblock.Blocks/Data/Utils/flowGraphArrayIndexBlock.tsundefinedinputs from unconnected sockets so it no longer crashes on lazy graphs.Blocks/Data/Transformers/flowGraphTypeToTypeBlocks.tsFlowGraphIntToFloataccepts bothFlowGraphIntegerand plainnumber(was strictly typed, which brokemath/dividecascades on dynamic inputs).Blocks/Execution/Animation/flowGraphPlayAnimationBlock.tsfrom/to(NaN, negative duration, invalid bezier control points now route to theerrorflow instead of silently producing NaN).Blocks/Event/flowGraphReceiveCustomEventBlock.ts,Blocks/Data/Parsers/flowGraphJsonPointerParserBlock.ts2. glTF Object-Model mapper
These changes are in
packages/dev/loaders/src/glTF/2.0/Extensions/. They extend the JSON-Pointer-to-Babylon-object mapper to comply with the Opaque-Reference spec and to fully support the read/write Object-Model paths the spec defines.objectModelMapping.ts(+601 / −32)nodes/{}/camera,mesh,skin,parent,children/{}, plusmeshes/{}/primitives/{}/materialand animations indexed access — all now return a JSON-Pointer ref string (e.g./cameras/3/) instead of an integer index, per spec.scenesandskinstrees so paths like/scenes/0/nodes/0and/skins/{}/joints/{}//skins/{}/skeletonresolve correctly./animations/{i}/refs._findNodeMorphTargets,_getNodeMorphTargetManager) — KHR_interactivity assets commonly query/nodes/<parent>/weights/*on a parent node whose primitives carry the actualMorphTargetManager; the walker resolves that and fanssetwrites across every sibling mesh that shares the manager._roundFloat32Artifact) so weights read back as their authored "clean" double for strictmath/eqcomparisons.KHR_texture_transformunderKHR_materials_transmission,KHR_materials_diffuse_transmission, andKHR_materials_volume—GenerateTextureMapnow accepts an optional{extensionKey, texturePath}that letspointer/get/pointer/setround-trip transform values through the source glTF JSON when the loader hasn't materialised the Babylon texture (e.g. volume with nothicknessFactor).lengthaccessors oncameras/materials,doubleSided/alphaCutoffmaterial accessors, defensive null-checks aroundanisotropyRotation.sheen.sheenRoughnessTexturenow maps tosheen.textureRoughness(was incorrectlysheen.thicknessTexture).gltfPathToObjectConverter.ts/animations/0/,/cameras/0/) resolve to the resource itself. Adds an index-pass-through wrapper for__ignoreObjectTree__paths so dynamic ref payloads can flow through the converter without further glTF-tree traversal.compositePathToObjectConverter.ts(new)babylonScenePathToObjectConverter.ts(new)objectModelMapping.tsthat resolves paths against the live Babylon scene rather than the source glTF JSON. Used by KHR_interactivity to resolve refs that arrive from outside the glTF (e.g. event payloads, dynamic mesh references).3. KHR_interactivity importer
These are the loader-side changes that translate a glTF
KHR_interactivitygraph into a serialized FlowGraph. They live inpackages/dev/loaders/src/glTF/2.0/Extensions/(importer + the three KHR node-state extensions).KHR_interactivity.tsKHR_interactivity/declarationMapper.tsref/eq→FlowGraphEqualityBlock(Events, SnailRace).math/quatSlerp→ the newFlowGraphMathSlerpBlock(Sundial).KHR_interactivity/interactivityGraphParser.ts_bakeRelativePointerPrefixparse-time helper for assets that author relative pointer templates ({nodeRef}/translation) paired with anodeRefsocket. Handles both static literal refs (Calculator) and dynamic ref connections (MagicBall)._parseNodes,_parseNodeConfiguration,_parseNodeConnectionsfor opaque-reference value handling.KHR_node_selectability.tsselectedNoderef output toevent/onSelect. Wired to the underlyingpickedMeshso blocks downstream get a proper ref string.KHR_node_hoverability.tshoveredNoderef output toevent/onHoverIn/event/onHoverOut(wired tomeshUnderPointer/meshOutOfPointer).KHR_node_visibility.tsonReadynow propagatesvisible: falseto the node's_primitiveBabylonMesheschildren, not just the wrappingTransformNode. Without this, multi-primitive nodes that started hidden (e.g. the fortune-cookie texts in MagicBall) were visible at startup.4. Supporting interface / type changes
packages/public/glTF2Interface/babylon.glTF2Interface.d.tsIKHRInteractivity_*interfaces for the spec's newrefvalue type. Required by the importer changes.How to validate locally
packages/tools/babylonServer).packages/tools/sandbox) and drop in any GLB from the Khronos KHR_interactivity sample-assets repo (ref-type-spec-changesbranch). All 13 sample models should load and behave as authored.Overview.glbconformance asset reports its sub-test pass-rate viaconsole.log(filter forTest Successful/Test Failed). Expected: 193 passing, 5 failing (1 is a pre-existing morph-weight authoring issue, the others are open spec questions).Automated integration tests are intentionally not included in this PR — they will be added in a follow-up once the Khronos sample-asset distribution strategy is finalized.
Reviewer notes
objectModelMapping.tsline count is new spec-required accessors plus the morph-target walker and texture-transform fallback helpers; everything else is small, surgical, and documented inline.loaders/.../Extensions/(compositePathToObjectConverter.ts,babylonScenePathToObjectConverter.ts) are required so that KHR_interactivity can resolve refs that arrive from outside the source glTF (e.g. event payloads pointing at Babylon scene objects).int(e.g./nodes/{}/camera) now return arefJSON-Pointer string, in line with the spec update. This is a breaking change for any consumer that read these specific paths as integers, but it matches the new spec exactly.