Skip to content

Latest commit

 

History

History
320 lines (268 loc) · 17.1 KB

chat-ui.md

File metadata and controls

320 lines (268 loc) · 17.1 KB

Chat UI, driven headlessly

The ui group of control-omb drives the real React renderer — the same <App/> the desktop shell loads, mounted by scripts/testing/threads-preview.tsx — in a headless Chrome through the agent-browser binary the harness pins (server/browser-engine-release.ts, implementation in scripts/testing/control-omb-ui.ts). Everything it touches is disposable: the fake-engine fixture from launch, a Vite preview of the app, and one browser session whose HOME is the fixture's data directory. The user's app on port 8799 and ~/.openmausbot are never involved.

Launch

node --experimental-strip-types scripts/control-omb.ts ui launch \
  --tool-calls '[{"name":"Bash","input":{"command":"echo hi"},"ok":true}]'

Run it in the foreground so Ctrl-C reaches it. On first use it downloads the pinned agent-browser release (size and SHA-256 verified) and its Chrome for Testing into .omb-scratch/verify-tools (gitignored); later launches reuse them. OMB_AGENT_BROWSER_PATH and AGENT_BROWSER_EXECUTABLE_PATH take precedence when set. The launcher then starts the fixture, pins its language to English (PATCH /api/config), creates Pepper through the same new-bot path as Chat turns, mounts the preview, opens it in a headless session named omb-ui-<port>, and prints a handle:

{
  "ok": true,
  "ui": "/tmp/openmausbot-verify-data-XXXXXX/ui.json",
  "url": "http://127.0.0.1:PORT",
  "previewUrl": "http://127.0.0.1:5178/__threads.html",
  "botId": "…", "dataDir": "…", "logPath": "…"
}

--tool-calls and --mode script the fake engine (FAKE_CLAUDE_TOOL_CALLS and FAKE_CLAUDE_MODE in server/testing/fake-claude-cli.ts). Pass ui.json to every other verb as --ui; there is no discovery, so a recipe cannot drive a browser it did not launch.

Drive

H=/tmp/openmausbot-verify-data-XXXXXX/ui.json
pnpm control:omb ui flag --ui $H --set features.showToolCalls=true --dry-run
pnpm control:omb ui flag --ui $H --set features.showToolCalls=true
pnpm control:omb ui snapshot --ui $H --interactive
pnpm control:omb ui type --ui $H --name "Message Pepper" --text hello
pnpm control:omb ui press --ui $H --keys Enter
pnpm control:omb ui wait-settle --ui $H --timeout 60
pnpm control:omb ui snapshot --ui $H
pnpm control:omb ui click --ui $H --name "Save as skill"
pnpm control:omb ui eval --ui $H --js "document.querySelector('textarea[aria-label=\"Message Pepper\"]').value"

snapshot returns the accessibility tree with @eN refs and a refs table of accessible names and roles. click and type take --ref @eN or the exact --name, and refuse an ambiguous name by listing the candidates. flag patches server feature flags (PATCH /api/config with features); the renderer picks the change up over SSE. wait-settle succeeds only when the shared wait tool reports the seeded bot settled, no bot in GET /api/bots is busy, the transcript shows the newest server message (its data-mid row) and the browser reports network idle; on timeout it exits non-zero with the last state it saw.

Expected: the interactive snapshot has exactly one textbox "Message Pepper". After the turn, the log "Conversation with Pepper" landmark contains StaticText "hello", a StaticText "Bash" tool chip (the scripted call) and StaticText "hello from fake claude" (the fake engine's reply). The chip is present only because showToolCalls is on; the sidebar row previews the reply too, so read the transcript landmark, not the whole tree.

Evidence

pnpm control:omb ui screenshot --ui $H --out .omb-scratch/verify-evidence/chat-ui.png
pnpm control:omb ui console --ui $H
pnpm control:omb ui eval --ui $H --js "document.title"

Keep the wait-settle JSON, both snapshots, the screenshot and the printed server log path. The console output must contain no error entries.

What the screenshot looks like when the recipe passes (evidence/chat-ui/chat-ui.png):

The isolated app after the recipe: Pepper's greeting, the sent "hello", a passed Bash chip and the fake reply

The permanent form of this recipe is scripts/testing/control-omb-ui.e2e.test.ts:

The fixture also hit-tests the rendered empty band beside This run. It must target the conversation rather than the transparent dock, while points inside the card and composer still target those controls. This checks browser pointer targeting, not native OS wheel delivery.

OMB_UI_E2E=1 pnpm exec vitest run scripts/testing/control-omb-ui.e2e.test.ts

The asserted recipe now also covers the floating This run card with a successful, failed, and dry-run command. The card records every shell command the bot ran in the current ask — the commands after the person's last message — and marks the ones that went through the control CLI verified; reads (cat, git log, gh pr view, a curl that only fetches) and computer-use are left out, and a run of one unverified command shows no card. Those tool outcomes are simulated provider events, not executions of the commands written in the chips; the card's summary reads 3 steps · 3 verified · 1 failed · 1 dry run. The recipe then presses Save as skill, which fills the composer in one of two shapes. A run with a verified step opens with the trigger phrase (Create a verification skill from the run below.), then Goal: hello (the person's request), the rule not to re-run, and one line per step with the verified ones tagged (verified). A run with no verified step asks in plain words instead — Save the steps below as a reusable skill for my review., then Goal: <request>, then "Keep the exact commands and note the failed ones as gotchas. Do not re-run anything.", then the step lines; the server (server/skill-learn.ts) expands a turn that opens with that sentence into the same skill-authoring turn as /learn, so nobody sees or types a slash command. In both shapes the caret is in the composer and no new user message was sent — the transcript still holds exactly one StaticText "hello". The person adds any notes and sends as usual; the card never sends on its own. Separately, the recipe runs a real fixture health check and verifies that clicking a deliberately missing control fails. The card remains collapsible; the old execution timeline is no longer shown above chat.

For activity detail, click More → Inspector → Run Log. It shows the selected conversation's recorded commands, statuses and timestamps; command previews may be shortened. Events and Raw retain the underlying technical views. The recipe checks tab switching and saves run-log.png alongside chat-ui.png. Copy redacted run log copies only the displayed activity (up to 200 entries), not chat text or raw protocol data. Review copied logs before sharing: automatic redaction is best effort. Neither this log nor a successful command proves an unasserted user outcome.

scripts/testing/usage-details-ui.e2e.test.ts checks the header's More menu, usage breakdown, clipboard success and refusal, and responsive geometry at 390, 800, 1100 and 1600 px. It checks that opening bot settings folds the sidebar without changing its saved density and closing settings restores it. Evidence includes screenshots and a .usage-details.json next to the fixture log. These are renderer checks, not native Windows caption-button verification.

It runs when an agent-browser binary resolves and is skipped with a printed reason otherwise; OMB_UI_E2E=1 forces the verified download. The ui-smoke job in .github/workflows/ci.yml runs it on Ubuntu 24.04 and uploads the screenshot; it is not one of the required checks.

Mobile text and code blocks

OMB_UI_E2E=1 pnpm exec vitest run scripts/testing/mobile-code-blocks-ui.e2e.test.ts
pnpm exec vitest run src/components/ChatMarkdown.block-actions.test.ts src/components/ChatMarkdown.test.ts src/lib/copy-text.test.ts

The browser recipe sends a synthetic reply through a disposable fake-engine server and renders it in the real chat. Its five blocks cover plaintext, highlighted code, an unknown language, an omitted language, and indented code. Copy preserves only the selected block's text, including whitespace; Wrap and Unwrap affect only that block and leave the copied source unchanged.

At narrow widths and on coarse-pointer or no-hover devices, the labeled actions occupy their own row with at least 44px touch targets. Check 320px and 390px portrait, landscape and tablet widths, and a desktop with a fine pointer. Long tokens must wrap within the bubble when Wrap is selected; unwrapped code can scroll horizontally inside its block without widening the page. Copy keeps a stable mobile label while its icon shows success or failure; a persistent live region exposes the feedback to assistive technology. Keyboard focus has an outline.

A hosted Linux runner has no mouse, so its headless Chrome reports no hover. The recipe starts Chrome with a fine, hovering pointer and runs the desktop pass before touch emulation starts: once touch emulation is switched on or off, Chrome goes back to the host's own pointer.

This is mobile web renderer coverage in an isolated Chromium profile. It does not verify the separately maintained native iOS or Android clients, physical devices, or Safari. Clipboard permissions are confined to the fixture origin; the refusal case replaces only that disposable page's clipboard writer.

Before: small icon-only actions After: labeled touch controls
Before at 390px After at 390px

Thinking timer across thread switches

scripts/testing/thinking-timer-ui.e2e.test.ts holds the working row's elapsed readout to the server's turnStartedAt stamp. It launches the full app with the fake engine in hang mode, so a sent turn stays officially in flight and the bot stays busy with no reply arriving. A second thread is created through the same POST /api/bots/:id/tasks the sidebar's New thread dispatches, the bot's thread list is expanded through its chevron, and the test switches to the idle thread and back mid-turn. The readout before the switch, the readout after the return, and the stamp on the wire are compared: the count must resume from the stamp (13+ seconds in), never from the moment of re-selection, and the stamp itself must not move while the turn runs. A screenshot of the anchored readout is kept as evidence.

Groups never showed the readout at all — their turns run on the group's busy slot, not on a member's task, so there was no stamp to count from. The second test in the same file gives groups their own: a one-member group is created through the same POST /api/groups the sidebar's group creation dispatches (with setup completed, so the composer is live at once), a message routes to the default responder, and the group's claim of its speaker — the busyBotId transition the server stamps as turnStartedAt on the group, cleared again when the group goes idle — must appear in the readout. The test switches to the member's 1:1 thread and back mid-turn and holds the resumed readout to the same claim stamp, keeping a second screenshot.

OMB_UI_E2E=1 pnpm exec vitest run scripts/testing/thinking-timer-ui.e2e.test.ts

In-flight reply text

The desktop shows a reply once it is finished; while a turn runs, the chat shows the bot's busy state. The renderer keeps no streamed text: runtimeFrameAction in src/state/store.tsx passes only the model-variant frames on, and src/state/store.test.ts checks that a reply or reasoning delta leaves the store state unchanged. src/components/ChatView.follow.test.ts checks that the finished reply still moves a following transcript to its end.

pnpm exec vitest run src/state/store.test.ts src/components/ChatView.follow.test.ts

What a bot is doing right now

While a step runs, the working line names it ("Running scrape.py", "Searching the web for “Acme construction leads”", "Reading linkedin.com"), with the whole command on hover. Once the step has run past the quiet notice it adds how it is doing: the time since its last output, or what its own processes show ("waiting on the network", "busy, no output yet", "idle, no output yet"). A click on the line, or Enter on it, unfolds the Now card: the current ask's steps, a running command's last lines of output and Open activity, which opens the Activity panel at this run. A second click or Escape folds it. A working bot's sidebar row shows the same step in place of its last message.

A scripted call with hold stays running until FAKE_CLAUDE_RELEASE exists; hold: "busy", "idle" or "network" also keeps a stand-in process that burns CPU, sleeps or holds a loopback connection, as the call's command would. The quiet-status knobs cross into the fixture, so the sign of life shows in seconds:

OMB_ACP_QUIET_NOTICE_MS=10000 OMB_ACP_QUIET_TICK_MS=2000 FAKE_CLAUDE_RELEASE=/tmp/release \
node --experimental-strip-types scripts/control-omb.ts ui launch --tool-calls \
  '[{"name":"WebSearch","input":{"query":"Acme construction leads"},"ok":true},
    {"name":"Bash","input":{"command":"cd ~/acme && python3 scrape.py --pages 40"},"hold":"busy"}]'

Expected after a sent message: Working · 15s · Running scrape.py · busy, no output yet, the button What Pepper is doing with aria-expanded="false", and, once clicked, a card listing Searched the web for “Acme construction leads” and Running scrape.py. Claude Code prints a command's output only when it ends, so streamed output needs an engine that sends it: --engine codex adds the fake Codex app-server, and FAKE_CODEX_COMMAND, FAKE_CODEX_COMMAND_OUTPUT (JSON chunks), FAKE_CODEX_COMMAND_OUTPUT_MS and FAKE_CODEX_COMMAND_GATE script its command. A bot moved to the codex instance then shows last output just now and the command's last two lines in the card.

The before-and-after screenshots were removed because they showed real example data; run the scripts/control-omb.ts ui launch command above to see the working line and the Now card.

The checks are server/step-summary.test.ts (the words and their redaction), server/harness/live-steps.test.ts and server/harness/bus.test.ts (one path for every engine, the sign of life, the output's last lines), server/bot-visibility.test.ts (a step reaches only someone who may see the chat), src/components/TurnPresence.test.ts, src/components/ChatView.live-status.test.ts (the line and the card) and src/components/Sidebar.simple-mode.test.ts (the sidebar row).

Bot setup and MCP access recipe

scripts/testing/bot-tools-ui.e2e.test.ts uses the same full-app launcher and optional OMB_UI_E2E=1 gate. It verifies profile-only role creation, closing and reopening the dialog during a slow creation without duplicate submissions, recovery when the preset PATCH fails after creation, the composer’s Tools shortcut, optional setup ideas, a pasted config added from Add MCP server that stays off when it doesn't answer, refreshed per-bot MCP switches and saved opt-outs, and modal Tab/Escape containment. The MCP command is an inert fixture command; no real accounts are connected. The advisory renderer job runs both recipes. Runtime mounting, direct/channel turns, busy-state rejection and revoked-session imports are separately exercised by server/mcp-selection.e2e.test.ts against disposable fake-engine servers.

Cleanup

Interrupt ui launch with Ctrl-C. It closes the browser session (waiting until agent-browser no longer lists it), then the preview, then the fixture, and removes only its data directory; the server log stays at the printed path and the tools directory keeps the downloads. Every verb refuses a handle whose launch has stopped.

Queued edits and Claude update recovery

The queued-message Edit action must remove the server's held send before returning its text to the same thread's draft, ahead of any existing text. To exercise the desktop Claude update card without an actual provider, launch an isolated UI with FAKE_CLAUDE_MODE=api-error and FAKE_CLAUDE_API_ERROR="API Error: 400 Claude Code 2.1.268 does not support this model; version 2.1.280 or newer is required.". Send one short message, then click Update Claude for me. The fake updater must report its synthetic version and the card must offer Retry, even when a digest follows the error. This proves the update request and recovery UI, not a real Claude installation or a successful provider retry.

What this proves, and what it does not

Proven: the real composer sends a turn on Enter, the fixture runs the scripted fake-engine turn, the transcript renders the sent text, the tool chip and the reply, a server-side feature flag reaches the renderer live, and Save as skill fills the composer without sending — all in a Chromium page, through accessibility names, with no mouse coordinates.

Not proven: the Electron shell (menus, preload bridge, screen capture, dictation), a real provider, Settings, sidebar drag-and-drop, the VM modal, the browser panel and updater UI, and anything Show threads gates (that toggle is renderer localStorage, outside ui flag). AGENT_BROWSER_HEADLESS=1 is set for consistency with the harness, but agent-browser 0.37.0 is headless by default and reads AGENT_BROWSER_HEADED to opt out.