Skip to content

Route Runtime callers through the stable Gateway Service - #16362

Open
agentydragon-agent wants to merge 7 commits into
agentydragon:develfrom
agentydragon-agent:gateway-service-routing
Open

agentydragon-agent wants to merge 7 commits into
agentydragon:develfrom
agentydragon-agent:gateway-service-routing

Conversation

@agentydragon-agent

@agentydragon-agent agentydragon-agent commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Merge only after #16345's runner-disk image is pinned, older live staging Runtimes are cycled, AND the independent-observer Gateway receiver image is pinned and running on every replica.

Move App, Notification, Sandbox Controller, Dispatcher and central sampler Runtime calls from <sandbox>-control to the stable Runner Gateway Service. All existing target settings keep their keys and use the same Sandbox identity metadata and authorization; no settings key is added. Every destination replica must already support independent zero-epoch Runtime observation.

This is stacked on #16361; review only the routing commit 62deef6 and recovery-fixture commit 9b3f848. GitHub native stacks cannot represent fork-to-upstream chains, so both PRs target upstream devel as requested; rebase away the receiver commit after it lands. Gateway route-record publication and Controller control-Service creation remain intact until all old caller Pods have rolled, allowing rollback throughout this step.

Rollout and rollback

Confirm the receiver Gateway image on every live Gateway Pod; then apply these existing-setting value changes. Check rollout completion and mounted/decoded settings for App, Notification, Sandbox Controller, Gateway/Dispatcher and central sampler, verifying each target is runner-gateway.{namespace}.svc:8080 (expanded for the Sandbox namespace). Hold route-publication removal until that inventory has no old target. A current ConfigMap alone does not prove every caller process restarted.

Rollback by reverting this PR, restoring its caller image pins where needed, and waiting for Flux and all caller Deployments to roll back to <sandbox>-control while the receiver Gateway still publishes that route. Check Gateway route recovery, Controller connected/bootstrapped readiness, idle suspend/resume, quiesce, retained History and a subsequent Session input. Do not roll back to the old exclusive Gateway until callers again use its legacy route. Reconcile uncertain calls with the original command ID; no automatic replay.

Validation

The browser failures were caused by the scripted Runtime rejecting zero-epoch WatchState requests. The fixture now accepts the receiver protocol and retains Sandbox/incarnation and peer authentication checks. The latest 40 indexed devel results for each browser target and the Gateway budget suite passed (raw GetTargetHistory, the query behind bbapi target history; CLI unavailable in this image); that index includes cache reuse.

Focused RBE invocation 7c298eca-e3b6-4abf-8ad3-4e10ec4f7125 passed //agentplane/app:test_thread_browser (all 16 shards), //agentplane/app:test_thread_window_browser, //agentplane/runner_gateway/internal/gateway:gateway_test and :queue_runner_integration_test, including the changed fixture library lint. The budget failure was a premature continuity assertion: the recovery probe had yielded a running code-mode cell while the sampler channel reconnected. The scenario now follows that same cell with wait, retaining native-session continuity, exact-once marker and next-input assertions. Focused invocation 8124457c-97ce-404e-8d3f-97d436a824c9 passed //agentplane/system_tests/suites/mcp:test_gateway_budget (348.6s) and the changed scenario library lint. No mutation replay, flake rerun, skipped test or timeout increase. These focused runs used the pre-rebase heads on 9a955bd3. After those fixes, both browser targets and the Gateway budget suite also passed in PR CI 2dbea6e6-752f-599b-b129-0270f8c35424 on the pre-rebase head. Full CI after rebasing onto 6e591f38 is pending; the range-diff confirms no patch changes. pre-commit run --from-ref origin/devel --to-ref HEAD passed after the rebase.

Required k3s merge gates are //agentplane/system_tests/suites:test_pod, :test_session_deletion, :test_firecracker, covering bootstrap (test_a_preset_launches_an_initialized_editable_codex_thread), idle suspend (test_operator_resumes_an_idle_pod_without_granting_its_identity_lifecycle_control), quiesce/History/next input (test_two_harness_threads_keep_their_context_after_sandbox_suspend_resume) and deletion fencing (test_session_command_cannot_cross_deletion). Central sampler coverage uses //agentplane/system_tests/suites/mcp:test_services, :test_gateway_crash, :test_history_outage, :test_sampler_roll and :test_gateway_budget. Real mTLS and existing service authorization remain in the k3s fixtures. Staging Flux/Cilium objects have no system test; staging is their first cluster.

Deliberately left out

Removal of legacy Lease/epoch allocation, control Service/EndpointSlice publication, Lease/EndpointSlice RBAC, and epoch proto fields; Runtime fencing changes; Dispatcher ownership Leases; EnvironmentService extraction. No Runtime-dials-Gateway mode, new auth allowlists, Identity admission, Viper, Policy error mapping or TLS-setting changes.

Questions for Claude

The Runtime/live-process gate was verified by Claude on staging at 08:25 UTC on 2026-11-14. Please confirm that the receiver Gateway image is running on every replica before merge. After applying this PR, confirm all caller processes use the stable Service before the next PR removes legacy publication. The receiver PR lists backend-specific live Runtime inventory checks; Firecracker RAM restore does not upgrade its Runtime process.

@agentydragon-agent
agentydragon-agent force-pushed the gateway-service-routing branch 5 times, most recently from d7caa0d to e6e3e96 Compare November 14, 2026 07:41
agentydragon added a commit that referenced this pull request Nov 14, 2026
<!-- claude-projects-attribution: {"github_login":"agentydragon"} -->
_Requested by **Rai** · [project
thread](https://claude.ai/code/project/chan_01RbvEzrs8Yi8YhNnHoSLtMy?thread=cmsg_01RbvEzrs8Yi8YhNnHoSLtMy2SHUu9XsCD6ej526Bhqzxa)_

Doc-only update of `agentplane/plans/task_dag.md` (and one link in
`plans/environment_rules.md`) for the last day's plans and decisions.

**Added**
- **Released 2026-11-14:** each track the operator released today, with
the quoted release, its node and its PRs' state at 07:40 UTC.
- **Sandbox-free central Sessions:** orders the existing sections 5, 13
and 14 nodes for a central Session with no Sandbox attached, from the
three #15436 decisions through the zero-Environment path and its
follow-ons. New decision node `CODE_MODE_HOST_DESIGN` for whether to
keep Codex's pinned JavaScript host or own a V8 host; it is recorded as
not a prerequisite, since the host runs in the sampler Pod and needs no
executor.
- **Section 17, two binaries and the Go layout:** the agreed
Environments/Sessions split with its decided cards (automatic wake,
directories, identity re-check, no budget for explicit resume), a stage
table mapping the gh-pages Go plan onto DAG nodes, and nodes for the
stage 3 moves, keep-alive holds, stage 2 layout fixes and stage 4 local
extractions.

**Updated**
- 2026-10-30 sprint priorities marked stale; no new priorities are
stated.
- `NOTIFICATION_SERVICE_SPLIT` carries the operator's 01:11–01:12
direction (inbox service plus GitHub actor; subscription seam open).
- `SESSION_TOOL_SURFACE_CONVERGENCE_DESIGN`: direction agreed at 01:10;
interface review open.
- `DISPATCHER_PERSISTENCE_MODEL_DESIGN`: topology selected; delivery
ownership and the Dispatcher's ownership Lease remain open.
- `RUNNER_GATEWAY_RENAME`: the Environment Gateway name is dropped; the
node now covers deployed names after the split.
- `PRIVILEGED_BACKEND_AUTH_DESIGN`, `APP_AGENT_TOKEN_DECISION`,
`ENVIRONMENT_LIFECYCLE_RULES`, `ENVIRONMENT_IDENTITY_BIND_SESSION` and
`REFACTOR_COHERENCE_AUDIT` carry their merged and in-review PRs.

**Deleted as finished:** `ENVIRONMENT_RULE_TYPE_DESIGN` (selected
04:58), `ENVIRONMENT_IDENTITY_RESUME` (#16346 merged),
`TRANSPORT_PACKAGE_RENAME` (the package lives only at `net/peertls`).

**Left out on purpose**
- Section 12 is untouched: the open Lease stack (#16361, #16362, #16365)
edits it. Section 17 refers only to `RUNNER_CONNECTION_LEASE_CLEANUP`,
which exists in both versions; whichever lands second rebases.
- `plans/environment_rules.md` status line: #16351 edits it.
- No priority changes and no new dispatch.

**Checks:** a local script parsed every Mermaid block (no unknown IDs or
cycles introduced), node headings (no duplicates) and local anchors; the
remaining findings, such as links to `central_session_contract.md` from
open #15436, predate this PR. `pre-commit run --files` on both files
passes.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01AYUnw2srgM3L5WxRM4C2Kt)_

Co-authored-by: Claude <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Visual review for 9b3f8482

Request a manual rerun from the CI run page.

Warning

12 targets used the latest devel baseline (a68e0de5) because the PR base published none. Diffs there may include devel drift, not just this PR's changes.

No visual changes among the 12 affected Bazel test targets. Open visual review.

@agentydragon-agent
agentydragon-agent force-pushed the gateway-service-routing branch 2 times, most recently from bcbd06c to 4cd31d3 Compare November 14, 2026 08:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant