[pi-plugin] pi-web embedded multi-session: field data + proposal for a public Host Adapter API (coordination with #350)

[LZHcode1986]

Summary

We run Magic Context with pi-web, which hosts multiple independent AgentSessions inside one Next.js Node process. Current Magic Context partially assumes "one Pi process = one primary session", which breaks the second and later sessions.

We have a working compatibility layer on top of v0.38.1 and would like to propose upstreaming its core: a small public Host Adapter API in packages/pi-plugin, so external embedded hosts can declare their hosting model instead of forking the plugin. This is related to #247 (closed) and overlaps in goal with the open PR #350 — we'd like to coordinate rather than compete.

Observed failures with stock Magic Context in stock pi-web

Reproduced against unmodified @cortexkit/pi-magic-context inside an unmodified production build of pi-web:

1. The process-global active latch makes every factory call after the first a no-op — the second browser session never registers Magic Context at all (same root cause class as [Pi] In-process parallel subagents can load Magic Context repeatedly and OOM in SessionManager.listAll #247, different host).
2. Historian/subagent spawning derives the Pi entry from process.argv[1], which in pi-web is the Next.js server entry, not a Pi CLI.
3. Spawned children then fail with unknown option '--print'.
4. Session cwd can resolve to the Next.js server directory instead of the actual project directory.
5. Net effect: tags/drops still run in session 1, but historian failure means no compartments are ever published and pending drops never complete.

Controlled comparison: with stock MC, session 1 gets the full ctx_* tool set and session 2 gets none. With our compat layer, both sessions register ctx_search / ctx_memory / ctx_note / ctx_expand / ctx_reduce, and both sessions' JSONL files land under the correct project cwd.

Proposal: a public MagicContextPiHostAdapter contract

Instead of encoding every host's quirks inside the plugin, let the host describe itself:

export interface MagicContextPiHostAdapter {
  readonly id: string;
  readonly mode: "standalone" | "embedded-multi-session";
  getRuntimeKey(pi: ExtensionAPI): object;
  resolveSessionCwd(pi: PiHostSessionApi): Promise<string>;
  resolvePiInvocation?(ctx: PiHostInvocationContext): PiHostInvocation | undefined;
}

Design points:

• Runtime claims are managed by MC core, keyed by the adapter's runtimeKey (the ExtensionAPI instance), replacing the process-global "second init is a no-op" latch for primary sessions. The claim registry lives on globalThis via Symbol.for, so it survives jiti moduleCache: false re-imports. In-process hidden children keep the existing lean/no-op path via MAGIC_CONTEXT_PI_SUBAGENT=1.
• Adapter loading is explicit: MAGIC_CONTEXT_PI_HOST_ADAPTER env var, or a plain Pi extension that registers only the adapter factory (no tools). A Next.js entry auto-detects as embedded-multi-session; MAGIC_CONTEXT_PI_HOST_MODE overrides.
• Scope discipline: no changes to project-identity semantics, sessionId is never used as a memory key, no core storage/schema changes, OpenCode plugin untouched, and no global mutation of process.argv / process.cwd() / PATH.
• Subagent launches accept a host-provided invocation (command = real execPath/Pi binary, prefixArgs = real CLI entry, cwd = parent session's project dir) instead of reusing process.argv[1].

Implementation status (working branch, rebased on v0.38.1)

• Split as packages/pi-plugin/src/host-adapter/ (public API + loader + runtime registry) plus a separate, out-of-tree pi-web-adapter package. For upstreaming we would scope the PR to the host-adapter API layer only; the pi-web-specific glue package stays external.
• packages/pi-plugin: 808 tests passing + typecheck clean on the v0.38.1 base; new loader/adapter tests included.
• Real-subprocess smoke test: with process.argv[1] pointing at a fake Next.js entry, PiSubagentRunner still spawns the resolved Pi CLI and the child receives --print --mode json --no-session and returns agent_end successfully.
• Dreamer timers are ref-counted per runtime claim, so closing one web session does not tear down a sibling session's project timer.
• Full end-to-end historian publish chain against a real model is the remaining validation item on our side.

Relationship to #247 and #350

• [Pi] In-process parallel subagents can load Magic Context repeatedly and OOM in SessionManager.listAll #247 diagnosed the process-global latch from the in-process-subagents angle; pi-web hits the same latch from the embedded-multi-session angle. We believe both need the same primitive: per-runtime initialization instead of per-process.
• fix(pi-plugin): support pi-web multi-session and RPC hosts #350 (open) replaces the latch with ALS child-marking inside packages/pi-plugin. Our proposal is complementary rather than mutually exclusive: fix(pi-plugin): support pi-web multi-session and RPC hosts #350 fixes which calls initialize; a Host Adapter API fixes how hosts declare their runtime topology (cwd resolution, CLI resolution, lifecycle ownership). We're happy to port our API on top of fix(pi-plugin): support pi-web multi-session and RPC hosts #350's internals if that lands first, or to split the work along that boundary.

Asks

1. Would upstream accept a public Host Adapter API in packages/pi-plugin?
2. How do the maintainers prefer to coordinate with fix(pi-plugin): support pi-web multi-session and RPC hosts #350?
3. If the direction is acceptable, we'll prepare a PR scoped to packages/pi-plugin/src/host-adapter/ (+ tests + PARITY.md notes), keeping the external adapter package out of the monorepo.

Happy to share the working branch or a diff of the API layer.

[alfonso-magic-context]

Direction settled on our side, in two parts:

Internals first. PR #350 is the plumbing any adapter needs (ALS session scoping, session-scoped lifecycle, owner-bound dreamer/RPC), and it's in final review now — one change-request round posted today, merge follows it. That work stands on its own regardless of the API question.

The public MagicContextPiHostAdapter API is deferred, deliberately. Once we publish an adapter contract, embedded hosts pin us to it — so we want it to freeze against real field data rather than a first design. Concretely: after #350 lands, the seam will exist but undocumented and subject to change; we'd rather iterate the adapter shape with you against live pi-web usage (your out-of-tree adapter staying out-of-tree is exactly the right posture for that phase) and stamp the public contract once it has survived contact. Your proposal's bones — id/mode, runtime-key selection, session-cwd resolution, optional invocation resolution, registry via Symbol.for — are a good starting frame, and the restraint in it (no storage/schema, no memory-key, no global process mutation) is noted and appreciated.

Practically: keep field data coming from the embedded runs, and once #350 is merged we'll open the design iteration on the adapter surface here on this issue.

[LZHcode1986]

Thank you for such a thorough response — this sequencing makes complete sense, and I agree the public contract should be stamped against real field data rather than a first design.

On a personal note: I really love the design of magic-context — the replay-everything cache discipline, tiered compartments with deterministic decay, memories that quietly accumulate across sessions. It's one of the most thoughtfully engineered tools I've used, and it's exactly why I'd rather build with upstream than maintain a divergent fork. Hoping magic-context keeps growing into an even better product — happy to contribute whatever I can along the way: field data from our embedded pi-web runs now, and adapter-surface iteration here once #350 lands.

Thanks again!