Browser fleet: fill-the-pane sizing, headful under Xvfb, native clipboard (text + images) - #315
Merged
Merged
Conversation
…esume The fleet rendered every browser at a fixed 1280x800. Now the viewer reports its pane size (debounced ResizeObserver) and the browser grows to fill it, clamped to [1280x800 .. 1920x1080]. The size is frozen while an agent drives (the resize is gated on the same input-enabled flag as human input, so an agent's cached `state` element indices can't shift mid-task), and reported back on resume/handback -- with a re-`state` nudge when it changed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Headless Chromium has no reachable clipboard, which blocks native copy/paste (and images entirely). Give it a real X11 session instead: deferred-install now apt-installs xvfb + xclip (own marker), a new [program:xvfb] runs an Xvfb server at :99 sized to the max render (1920x1080), and the browser service exports DISPLAY=:99 so Chromium launches headful into it. _HEADLESS now defaults to headful when a DISPLAY is present and headless otherwise, so tests and bare dev boxes still run without a display. Browser readiness also waits on the xvfb marker when headful. This is prerequisite infra for native clipboard (next commits); anti-bot fidelity is incidental and marginal (CDP, not the window, is the dominant signal). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The browser now runs headful with a real X11 clipboard, so copy/paste is bridged natively via xclip -- no page-origin Async Clipboard API, no https/activation constraints, and text or images the same way. - session.py: clipboard_paste (xclip -i then a native "paste" editing command) and clipboard_copy/cut (native "copy"/"cut" then xclip -o), gated on human control. Server side is mime-agnostic -- images already work here. - runner.py: GET/POST /browsers/<id>/clipboard (blobs over HTTP, since the cast socket has a ~1 MiB cap); MAX_CONTENT_LENGTH bounds a paste at 32 MB. - viewer: Ctrl OR Cmd + C/V/X read/write the user's real navigator.clipboard (Mac + Linux), bridged over the new endpoint instead of forwarded as keystrokes. - IframePanel: allow=clipboard-read/write so the embedded viewer can reach it. This wires text end-to-end; the viewer's image capture + a paste-progress indicator are the next commits (the server already handles image bytes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the viewer's clipboard bridge to images and adds the non-blocking progress pill the server side was already ready for. - Paste reads the user's clipboard with navigator.clipboard.read() (image, else text) and POSTs the raw bytes; copy/cut writes an image ClipboardItem (or text) back to the user's clipboard from the server's base64 payload. - A top-center pill (Pasting…/Copying…) appears after a 200ms delay -- so instant text ops don't flicker, but image/long transfers show progress. pointer-events: none, and paste is server-owned once uploaded, so the user can navigate away and it still lands. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous cut watched for Ctrl/Cmd+C/V/X keystrokes -- which missed alternative bindings (Ctrl/Shift+Insert, Shift+Delete), right-click menu, and the Edit menu, so it wasn't OS/keymap-agnostic. Switch to document copy/paste/cut EVENT listeners, which fire regardless of how the user triggered them. - paste reads the event's clipboardData directly (text + images) -- and this drops the navigator.clipboard.readText path, so no clipboard-read permission prompt. - copy/cut pull the remote selection and write the user's clipboard via navigator.clipboard (writeText / ClipboardItem). - key() no longer drives the clipboard; it just stops forwarding the clipboard shortcuts to the remote (and lets the browser's default fire the events) so nothing pastes/cuts twice. Contract note: relies on document-level clipboard events firing over the canvas viewer (the Excalidraw/Figma pattern) -- solid in Chromium; needs a cross-browser live check. A hidden focusable capture element is the fallback if a browser balks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 1280x800 floor was larger than typical panels, so sub-1280 panes pinned to it and never changed. Lower it to 640x480 so real panels track their size, and log every resize request (raw, clamped, applied, headless flag) + when the input gate drops it -- to pin down whether the stuck-at-1280x800 report is the floor, the human-control gate, or device-metrics not driving the frame under headful. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two fixes so a browser pane never orphans and its absence never reads as an error:
- server.py: the layout open/split broadcast handler now rejects a bare
service:browser (no ?session=<name>) with a 400 that guides the caller to a
real browser name -- stopping an agent from spawning the dead 'Open a browser
from the + menu' placeholder pane. The fleet's own pane-pull always carries a
session, so it's unaffected (new is_sessionless_browser_ref helper + tests).
- fleet.py: when _pull_in_pane can't surface the pane (background/sub-agent with
no chat in view), report it as a neutral, optional nudge ('browser X is ready;
open it from the + menu to watch') instead of 'I couldn't open it' -- the
browser is up and fully drivable regardless; the pane is just a convenience.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Changelog entries for all four touched projects (agents, browser, dev, system_interface), documenting the fill-the-pane sizing, headful+Xvfb native clipboard, the session-less service:browser gate, and the neutral pane-pull fallback. - Move two trailing comments to their own lines (server_test.py, session.py) so the trailing-comments ratchet stops firing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_pull_in_pane never passed the --layout that split has required since named layouts landed, so the split was always rejected and the pane never appeared -- every browser command silently fell through to the '+' menu message. Now it runs 'layout.py context' (read-only) to resolve the layout the requester's client is viewing and passes --layout to the split. Fully optimistic: - unreachable layout server (isolated launch-task sub-agent) -> skip silently - reachable but the split can't land (human not viewing that layout) -> the neutral '+' menu nudge Same-container agents (primary, + New agent, native Task subagents that share the parent's identity) surface the pane reliably when their chat is on-screen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolves conflicts from PR #298 (Fortress) landing on main: - deferred_install.sh: main()) runs _install_fortress (engine, from main) + my _install_xvfb. - session.py: use the Fortress marker/executable (from main) and keep my Xvfb marker; deferred_install_ready gates on both; _build_bu_session launches the Fortress binary headful under Xvfb. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…laywright) Adds a 'Browser is available as a tool' section to the workspace agent instructions: use the agentic-browser-fleet skill for collaborative, human-shareable browsing (with the control handoff/queue), or Playwright directly for lightweight integration testing on the same stealth Chromium. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An agent was seen running 'layout.py open browser --layout desktop' -- the exact orphan-pane anti-pattern (no session -> dead placeholder, now gated). The pane already auto-opens on new/task/first-command with a graceful fallback line, so: - agentic-browser-fleet skill: explicit 'the pane is automatic; never open it yourself with layout.py; relay the fallback line instead of retrying'. - manage-layout skill: the generic 'open <target>' row now excludes 'browser' (the fleet auto-surfaces its own panes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Correct + condense the earlier note: the pane auto-opens only when the user is watching that chat; if they explicitly ask to open one, use service:browser?session=<name>; otherwise point them at the '+ -> browser' menu. One tight paragraph in agentic-browser-fleet, two sentences in manage-layout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… path A bare playwright.chromium.launch() (no executable_path) looks in Playwright's own browser cache, which the Fortress swap leaves empty (install-deps, not install -> no managed Chromium downloaded), so ad-hoc Playwright calls error 'Chromium not present'. Symlink Fortress into that expected path -- resolved from Playwright itself so it tracks the pinned version's revision/layout -- so the default resolves to the same one engine. Chromium finds its resources via /proc/self/exe (the real Fortress dir), so symlinking the binary suffices. Not end-to-end tested with Fortress here (no Fortress on this box); the path resolution is verified. Wants a workspace smoke test of a bare chromium.launch(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Superseded by #328 (browser-live-view-v2). This branch grew directly into browser-live-view-v2 (confirmed: browser-fleet-improvements is a git ancestor of it), so #328 contains all of this work — fill-the-pane sizing, headful-under-Xvfb, native clipboard — plus the full live-view-v2 rework (striped H.264/JPEG streaming, audio, the poll-driven viewer, etc.). Closing in favor of #328. |
…ments # Conflicts: # system/apps/browser/changelog/browser-fleet-improvements.md # system/apps/browser/src/browser/session.py # system/apps/system_interface/changelog/browser-fleet-improvements.md # system/changelog/browser-fleet-improvements.md # system/scripts/env.d/1000-playwright-fortress.sh # system/supervisord.conf
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
lgtm once tests pass there are some more complexities to deal with here later, but I think this is a fine place to start with testing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Browser-fleet improvements: