Browser fleet: fill-the-pane sizing, headful under Xvfb, native clipboard (text + images)

.agents/changelog/browser-fleet-improvements.md

@@ -0,0 +1,3 @@
+The `agentic-browser-fleet` skill now notes that, on resume after a human held the browser, the live view may have been resized and the page reflowed -- so every cached element number should be treated as stale and `state` re-run before acting.
+
+The `agentic-browser-fleet` and `manage-layout` skills now tell agents the browser pane is surfaced automatically when the user is watching that chat, so they shouldn't manage it by hand: never open a bare `service:browser` (rejected -- no browser bound); if a user explicitly asks to open one, use `service:browser?session=<name>`; otherwise point them at the "+ -> browser" menu.

.agents/skills/agentic-browser-fleet/SKILL.md

@@ -168,7 +168,7 @@ Every browser has exactly one controller; every command's output names the owner
   - You switch to a different browser for the rest of the task -> release the one you're leaving.
   - Driving several at once -> keep them until fully done, then release each.
   - If you forget, an idle lease auto-frees after ~90s; if a later command says you no longer hold it, just acquire it again.
-- **The human always wins.** If a human takes control, your next command comes back with status `busy_human`/`lost_control` (exit 2). You lost control: **stop, tell the user the human took the wheel, and end your turn.** Do not retry, poll, or `--reclaim` on your own. You're queued to resume first; you'll be messaged when they hand it back. On resume, **re-run `state <name>` first** (the page changed), then continue. Resume early only on an explicit "keep going": `acquire <name> --reclaim`, then `state <name>`.
+- **The human always wins.** If a human takes control, your next command comes back with status `busy_human`/`lost_control` (exit 2). You lost control: **stop, tell the user the human took the wheel, and end your turn.** Do not retry, poll, or `--reclaim` on your own. You're queued to resume first; you'll be messaged when they hand it back. On resume, **re-run `state <name>` first** (the page changed -- and the view may have been resized while they held it, reflowing the layout, so treat every element number as stale), then continue. Resume early only on an explicit "keep going": `acquire <name> --reclaim`, then `state <name>`.
 - **Agents never preempt each other.** A browser another agent holds returns (exit 3):
 
   ```text
@@ -204,7 +204,9 @@ uv run agentic-browser-fleet handoff alex-smith "solve the CAPTCHA on the sign-i
 
 ## Live view vs. your output
 
-The browser shows up live in a UI pane next to your chat so the human can watch you operate it. That pane is **viewer only** -- your actual output (the `state` listings, the `ok:`/error lines, the screenshot paths) is in your CLI output here in the chat. Read and relay the CLI output; don't tell the user to "check the tab" for results.
+The browser streams to a UI pane next to your chat. `new`/`task`/your first command surface it automatically -- **but only when the user is currently watching your chat** (it lands beside that chat on their active layout). So don't manage panes yourself: never open a bare `service:browser` (no browser bound -- the daemon rejects it). If the user explicitly asks you to open a browser that isn't showing, run `layout.py open service:browser?session=<name>` **with the name**; otherwise just tell them to open it from the workspace **+ -> browser** menu -- don't fuss over it.
+
+The pane is **viewer only** -- your real output (`state` listings, `ok:`/error lines, screenshot paths) is here in the CLI. Read and relay that; don't tell the user to "check the tab" for results.
 
 ## Multiple browsers, tabs, sub-agents
 

.agents/skills/manage-layout/SKILL.md

@@ -96,6 +96,10 @@ Targets `open` accepts:
   than creating a duplicate. This is the same terminal the chat
   panel's "Open agent terminal" button mounts.
 
+Not `browser`. The agentic-browser-fleet surfaces its own panes; a bare
+`open browser` binds to no browser and is rejected. Only open one when a
+user explicitly asks, and always name it: `open service:browser?session=<name>`.
+
 ## Refs: how every panel is addressed
 
 Every panel has a stable, type-prefixed ref returned by `inspect`:

CLAUDE.md

@@ -176,6 +176,13 @@ They are inherently flaky due to timing and useless in CI, but valuable for agen
 
 If the user talks to you about files or directories on disk, assume (unless context indicates otherwise) they mean their local disk, not the one in your sandbox -- use the `file-sharing` skill to bridge the two.
 
+# Browser is available as a tool
+
+A stealth build of Chromium designed to look like an ordinary human browser is installed in this workspace and can be used to complete browser-related tasks. 
+
+1. When the user requests any browser-related tasks to be complete or a browser to be opened, use the `agentic-browser-fleet` skill, which allows you to drive many Chromium browsers. These are collaborative browsers which all agents and human users can use, though there is a mutually-exclusive control handoff and queuing system so only one is using a browser at a time. The skill has more information. Remember to hand off control to user when help is needed in the browser, such as anti-bot detection tests, and also release control when you are finished with a task so other agents and the user can use it.
+2. If you'd like to do integration testing/small-scale web app scripting, use Playwright instead of spinning up an entire browser through the agentic-browser-fleet skill. This uses the same Chromium, just more lightweight. The user and other agents won't be able to collaborate on this; this is for quicker rendering and interaction tasks on the web.
+
 # Work delegation
 
 You can delegate larger tasks to sub-agents using the `launch-task` skill.

system/apps/browser/README.md

@@ -8,9 +8,11 @@ agent, identified by its `MNGR_AGENT_ID`, or the human).
   thread-per-connection) that owns every browser. browser_use, Playwright (async),
   and the per-browser ownership state machine run on one background asyncio event
   loop, reached from the Flask threads through a single `run_coroutine_threadsafe`
-  bridge. Each browser is a headless Chromium driven by `browser_use.BrowserSession`, observed
-  over the same CDP endpoint to stream a live view (`Page.startScreencast` ->
-  base64 JPEG frames over a WebSocket) and inject human input. Each browser is
+  bridge. Each browser is a **headful** Chromium (under an Xvfb virtual display, so
+  it has a real X11 clipboard for native copy/paste -- see `_HEADLESS` in
+  `session.py`) driven by `browser_use.BrowserSession`, observed over the same CDP
+  endpoint to stream a live view (`Page.startScreencast` -> base64 JPEG frames over
+  a WebSocket) and inject human input. Each browser is
   addressed by a random ~2-word english NAME (e.g. `alex-smith`), generated on
   demand and never reused; the fleet starts empty and there is no default
   browser.

system/apps/browser/changelog/browser-fleet-improvements.md

@@ -0,0 +1,5 @@
+The live browser now fills its viewer pane instead of a fixed 1280x800 box: the viewer reports its size and the browser grows to fill it (clamped 640x480 .. 1920x1080), frozen while an agent is driving so its element numbers don't shift mid-task, and the resolution is reported back on resume so the agent knows to re-read the page if it changed.
+
+The fleet now runs headful under an Xvfb virtual display, which gives it a real OS clipboard: copy and paste -- text and images -- between your machine and the remote browser now work natively, driven by the real copy/paste/cut events (so any keybinding, right-click, or the Edit menu works), with a non-blocking "Pasting.../Copying..." indicator for larger transfers.
+
+The browser's live pane now actually auto-opens beside your chat when an agent works a browser (`new` / `task` / any direct command): the fleet resolves which layout you're currently viewing (via `layout.py context`) and surfaces the pane there -- previously it never named a layout, so the split was silently rejected and the pane never appeared. If the layout server is unreachable (an isolated sub-agent in its own container) it skips silently; if your screen is reachable but the pane can't land (you're not viewing that layout) it offers the manual "+"-menu route instead of implying anything broke. Either way the browser is fully drivable from the CLI; the pane is only a live-view convenience.

system/apps/browser/fleet_test.py

@@ -133,24 +133,59 @@ def test_parser_accepts_direct_verbs() -> None:
 def test_pull_in_pane_opens_each_browser_in_its_own_pane(monkeypatch: pytest.MonkeyPatch) -> None:
     # A user-started agent surfaces each browser as its OWN pane (--new-group), beside
     # its own chat (--relative-to self), not tabbed into an existing browser pane. The
-    # session ref keys on the NAME.
+    # split carries the resolved --layout, and the session ref keys on the NAME.
     calls: list[tuple] = []
+    monkeypatch.setattr(fleet, "_resolve_active_layout", lambda: (True, "desktop"))
     monkeypatch.setattr(fleet, "_layout", lambda *a, **k: calls.append(a) or True)
     monkeypatch.delenv("BROWSER_FLEET_ANCHOR", raising=False)
     fleet._pull_in_pane("alex-smith")
     assert calls and "--new-group" in calls[0] and "right" in calls[0] and "self" in calls[0]
+    assert "--layout" in calls[0] and "desktop" in calls[0]
     assert any("session=alex-smith" in arg for arg in calls[0])
 
 
 def test_pull_in_pane_warns_cleanly_when_it_cant_show_a_pane(monkeypatch: pytest.MonkeyPatch) -> None:
-    # A background/sub-agent (no chat in this UI) can't land the split: we attempt it,
-    # then warn in one clean line -- never crash, never leak the raw 5s "not registered"
+    # Reachable layout server, but the split never lands (human isn't viewing that layout):
+    # we attempt it, then warn in one clean line -- never crash, never leak layout.py's raw
     # error (the browser is still running).
+    monkeypatch.setattr(fleet, "_resolve_active_layout", lambda: (True, "desktop"))
     monkeypatch.setattr(fleet, "_layout", lambda *a, **k: False)  # layout never lands
     monkeypatch.delenv("BROWSER_FLEET_ANCHOR", raising=False)
     fleet._pull_in_pane("riley-jones")  # must not raise
 
 
+def test_pull_in_pane_skips_silently_when_layout_server_unreachable(monkeypatch: pytest.MonkeyPatch) -> None:
+    # An isolated launch-task sub-agent can't reach the layout server: _resolve_active_layout
+    # returns (False, None). We must NOT attempt the split and NOT print anything.
+    attempted: list[tuple] = []
+    printed: list[str] = []
+    monkeypatch.setattr(fleet, "_resolve_active_layout", lambda: (False, None))
+    monkeypatch.setattr(fleet, "_layout", lambda *a, **k: attempted.append(a) or True)
+    monkeypatch.setattr(fleet, "_out", lambda msg: printed.append(msg))
+    fleet._pull_in_pane("riley-jones")
+    assert attempted == [] and printed == []
+
+
+def test_resolve_active_layout_prefers_client_that_messaged_this_agent(monkeypatch: pytest.MonkeyPatch) -> None:
+    # Two connected clients on different layouts; pick the one whose recent messages named
+    # OUR agent (context exposes agent_name, not id).
+    stdout = (
+        '[{"is_connected": true, "current_layout": "mobile",'
+        '  "recent_messages": [{"agent_name": "someone-else"}]},'
+        ' {"is_connected": true, "current_layout": "desktop",'
+        '  "recent_messages": [{"agent_name": "riley-jones"}]}]'
+    )
+    monkeypatch.setenv("MNGR_AGENT_NAME", "riley-jones")
+    monkeypatch.setattr(fleet.subprocess, "run", lambda *a, **k: fleet.subprocess.CompletedProcess([], 0, stdout, ""))
+    assert fleet._resolve_active_layout() == (True, "desktop")
+
+
+def test_resolve_active_layout_unreachable_returns_false(monkeypatch: pytest.MonkeyPatch) -> None:
+    # A non-zero context exit (isolated sub-agent / no daemon) -> (False, None): skip silently.
+    monkeypatch.setattr(fleet.subprocess, "run", lambda *a, **k: fleet.subprocess.CompletedProcess([], 1, "", "boom"))
+    assert fleet._resolve_active_layout() == (False, None)
+
+
 def test_cmd_new_pulls_a_pane_by_name(monkeypatch: pytest.MonkeyPatch) -> None:
     # "Open a new browser" should visibly open its pane (by the returned name), not wait
     # for the first command. The daemon returns the chosen name as `name`.

system/apps/browser/src/browser/assets/index.html

@@ -71,6 +71,12 @@
   /* While init/crashed cover the pane, hide the tab + nav chrome so a half-built
      browser never peeks through. Toggled by the `chrome-hidden` class on #app. */
   #app.chrome-hidden #tabbar, #app.chrome-hidden #navbar, #app.chrome-hidden #returnbar { display: none; }
+
+  /* Non-blocking top-center pill shown while a larger clipboard op is in flight
+     (image paste/copy, long text). pointer-events:none so it never traps the user
+     -- the paste is already server-owned, so they can navigate away freely. */
+  #cliptoast { position: absolute; top: 12px; left: 50%; transform: translateX(-50%); z-index: 30; background: rgba(14, 99, 156, 0.92); color: #fff; padding: 6px 16px; border-radius: 999px; font-size: 13px; pointer-events: none; opacity: 0; transition: opacity .15s; }
+  #cliptoast.show { opacity: 1; }
 </style>
 </head>
 <body>
@@ -89,6 +95,7 @@
   <div id="stage">
     <canvas id="screen" tabindex="0"></canvas>
     <div id="placeholder">Starting browser…</div>
+    <div id="cliptoast"></div>
     <div id="overlay">
       <div class="msg" id="overlaymsg">An agent has control.</div>
       <div class="sub" id="overlaysub"></div>
@@ -108,6 +115,8 @@
 (function () {
   "use strict";
   const app = document.getElementById("app");
+  const stage = document.getElementById("stage");
+  const cliptoast = document.getElementById("cliptoast");
   const canvas = document.getElementById("screen");
   const ctx = canvas.getContext("2d");
   const placeholder = document.getElementById("placeholder");
@@ -325,6 +334,18 @@
 
   function sendCast(obj) { if (castWs && castWs.readyState === 1) castWs.send(JSON.stringify(obj)); }
 
+  // Report our pane size so the server grows the browser to fill it (it clamps to
+  // [1280x800 .. 1920x1080] and ignores this while an agent drives). Debounced -- a
+  // drag fires a burst and each server resize re-attaches the screencast.
+  let resizeTimer = null;
+  function reportSize() {
+    clearTimeout(resizeTimer);
+    resizeTimer = setTimeout(() => {
+      sendCast({ type: "resize", width: Math.round(stage.clientWidth), height: Math.round(stage.clientHeight) });
+    }, 150);
+  }
+  new ResizeObserver(reportSize).observe(stage);
+
   // --- coordinate scaling: canvas CSS px -> frame device px ---
   function scaled(e) {
     const r = canvas.getBoundingClientRect();
@@ -364,8 +385,83 @@
     sendCast({ type: "mouse", event: { type: "mouseWheel", x: p.x, y: p.y, button: "none", deltaX: e.deltaX, deltaY: e.deltaY, modifiers: modifiers(e) } });
   }, { passive: false });
 
+  // --- clipboard bridge -------------------------------------------------------
+  // The remote browser runs headful with a real OS clipboard; these sync it with
+  // the USER's clipboard over HTTP (not the cast socket -- images exceed its cap).
+  // Driven by the real copy/paste/cut EVENTS (below), so it's OS- and keymap-
+  // agnostic: fires on Ctrl+C, Cmd+C, Ctrl+Insert, the right-click menu, or the Edit
+  // menu alike -- never tied to guessing specific keystrokes.
+  function clipUrl() { return new URL("browsers/" + browserId + "/clipboard", document.baseURI).toString(); }
+
+  // Non-blocking progress pill for larger clipboard ops. Delayed so instant text
+  // ops don't flicker; returns a done() that clears it. pointer-events:none (CSS)
+  // means it never traps the user -- the paste is server-owned once uploaded, so
+  // they can navigate away and it still lands.
+  function withClipToast(label) {
+    const t = setTimeout(() => { cliptoast.textContent = label; cliptoast.classList.add("show"); }, 200);
+    return () => { clearTimeout(t); cliptoast.classList.remove("show"); };
+  }
+
+  function b64ToBlob(b64, mime) {
+    const bin = atob(b64), arr = new Uint8Array(bin.length);
+    for (let i = 0; i < bin.length; i++) arr[i] = bin.charCodeAt(i);
+    return new Blob([arr], { type: mime });
+  }
+
+  function clipActive() { return controlOwner !== "agent" && hasFrame && castWs && castWs.readyState === 1; }
+
+  // PASTE: the event's own clipboardData carries text + images with no permission
+  // prompt (it's a user-initiated paste gesture). Send the raw bytes to the remote.
+  async function sendPaste(body, mime) {
+    const done = withClipToast("Pasting…");
+    try { await fetch(clipUrl(), { method: "POST", headers: { "Content-Type": mime }, body }); }
+    catch (_) { /* not controlling / network: ignore */ }
+    finally { done(); }
+  }
+  document.addEventListener("paste", (e) => {
+    if (!clipActive()) return;
+    e.preventDefault();
+    for (const it of e.clipboardData.items) {
+      if (it.type.indexOf("image/") === 0) {
+        const blob = it.getAsFile();
+        if (blob) { sendPaste(blob, it.type); return; }
+      }
+    }
+    const text = e.clipboardData.getData("text/plain");
+    if (text) sendPaste(new Blob([text], { type: "text/plain" }), "text/plain");
+  });
+
+  // COPY/CUT: pull the remote browser's selection and write it to the user's real
+  // clipboard (text via writeText, image via ClipboardItem).
+  async function copyFromRemote(cut) {
+    if (!navigator.clipboard) return;  // needs a secure context (https/localhost)
+    const done = withClipToast("Copying…");
+    try {
+      const r = await fetch(clipUrl() + (cut ? "?cut=1" : ""));
+      const j = await r.json();
+      if (j.ok && j.mime) {
+        if (j.mime.indexOf("text/") === 0) { if (j.text) await navigator.clipboard.writeText(j.text); }
+        else if (j.data) await navigator.clipboard.write([new ClipboardItem({ [j.mime]: b64ToBlob(j.data, j.mime) })]);
+      }
+    } catch (_) { /* denied / nothing selected: ignore */ }
+    finally { done(); }
+  }
+  document.addEventListener("copy", (e) => { if (clipActive()) { e.preventDefault(); copyFromRemote(false); } });
+  document.addEventListener("cut", (e) => { if (clipActive()) { e.preventDefault(); copyFromRemote(true); } });
+
   function key(type, e) {
     if (controlOwner === "agent" || !hasFrame) return;
+    // Clipboard shortcuts are handled by the copy/paste/cut EVENT listeners above.
+    // Let the browser's DEFAULT fire those events -- so DON'T preventDefault and
+    // DON'T forward, or (a) the paste/copy event never fires and (b) the remote
+    // would paste/cut a second time. Covers Ctrl/Cmd+C/V/X plus the legacy
+    // Ctrl/Shift+Insert and Shift+Delete bindings; every other key passes through.
+    const kl = e.key.toLowerCase();
+    if (((e.ctrlKey || e.metaKey) && !e.altKey && ["c", "v", "x"].includes(kl))
+        || (e.key === "Insert" && (e.ctrlKey || e.shiftKey))
+        || (e.key === "Delete" && e.shiftKey)) {
+      return;
+    }
     e.preventDefault();
     const isChar = type === "keyDown" && e.key.length === 1;
     sendCast({ type: "key", event: {
@@ -407,6 +503,7 @@
   function connect(id) {
     browserId = id;
     castWs = new WebSocket(wsUrl("browsers/" + id + "/cast"));
+    castWs.onopen = () => reportSize();  // send our pane size once the socket is live
     castWs.onmessage = (ev) => {
       let m; try { m = JSON.parse(ev.data); } catch (_) { return; }
       if (m.type === "crashed") showCrashed();