feat - Remove pi-mcp-adapter extension as dependencies for pi coding agent

[tan-yong-sheng]

Problem

Installing too much pi extension is sometimes hard to manage like version compatibility, etc, so we can keep uninstall the unnecessary layer to reduce maintenance overhead ...

For example, codebase-memory-mcp is also mcp tool, but using a single .ts file, we can easily translate those stdio mcp function into native tool via pi extension: https://github.com/DeusData/codebase-memory-mcp as shown below.

Proposed solution

Example code for codebase-memory-mcp pi extension that you may take reference with:

File name: ~/.pi/agent/extensions/cbmem.ts

// codebase-memory-mcp:start
// Generated by codebase-memory-mcp for pi.
// Regenerated on install/update from the MCP tool registry, so it cannot
// drift from the tools the server actually serves. Edits inside this block
// are overwritten; edit outside it, or remove the markers to take ownership.
import { spawn } from 'node:child_process';

const BIN = '/home/tys203831/.local/bin/codebase-memory-mcp';

async function call(tool, args, signal) {
  return new Promise((resolve) => {
    const child = spawn(BIN, ['cli', tool, JSON.stringify(args ?? {})], {
      stdio: ['ignore', 'pipe', 'pipe'],
      env: { ...process.env, CBM_LOG_LEVEL: 'error' },
    });
    let out = '';
    const onAbort = () => { if (!child.killed) child.kill(); };
    signal?.addEventListener('abort', onAbort, { once: true });
    child.stdout.on('data', (d) => (out += d.toString()));
    child.on('error', (e) => {
      signal?.removeEventListener('abort', onAbort);
      resolve({ error: String(e && e.message ? e.message : e) });
    });
    child.on('close', () => {
      signal?.removeEventListener('abort', onAbort);
      const lines = out.split('\n').map((l) => l.trim()).filter(Boolean);
      for (let i = lines.length - 1; i >= 0; i--) {
        try { return resolve(JSON.parse(lines[i])); } catch { /* keep scanning */ }
      }
      resolve({ error: 'no JSON response from codebase-memory-mcp' });
    });
  });
}

export default function (pi) {
  pi.registerTool({ name: 'index_repository', run: (args, ctx) => call('index_repository', args, ctx?.signal) });
  pi.registerTool({ name: 'search_graph', run: (args, ctx) => call('search_graph', args, ctx?.signal) });
  pi.registerTool({ name: 'query_graph', run: (args, ctx) => call('query_graph', args, ctx?.signal) });
  pi.registerTool({ name: 'trace_path', run: (args, ctx) => call('trace_path', args, ctx?.signal) });
  pi.registerTool({ name: 'get_code_snippet', run: (args, ctx) => call('get_code_snippet', args, ctx?.signal) });
  pi.registerTool({ name: 'get_graph_schema', run: (args, ctx) => call('get_graph_schema', args, ctx?.signal) });
  pi.registerTool({ name: 'get_architecture', run: (args, ctx) => call('get_architecture', args, ctx?.signal) });
  pi.registerTool({ name: 'search_code', run: (args, ctx) => call('search_code', args, ctx?.signal) });
  pi.registerTool({ name: 'list_projects', run: (args, ctx) => call('list_projects', args, ctx?.signal) });
  pi.registerTool({ name: 'delete_project', run: (args, ctx) => call('delete_project', args, ctx?.signal) });
  pi.registerTool({ name: 'index_status', run: (args, ctx) => call('index_status', args, ctx?.signal) });
  pi.registerTool({ name: 'check_index_coverage', run: (args, ctx) => call('check_index_coverage', args, ctx?.signal) });
  pi.registerTool({ name: 'detect_changes', run: (args, ctx) => call('detect_changes', args, ctx?.signal) });
  pi.registerTool({ name: 'manage_adr', run: (args, ctx) => call('manage_adr', args, ctx?.signal) });
  pi.registerTool({ name: 'ingest_traces', run: (args, ctx) => call('ingest_traces', args, ctx?.signal) });
}
// codebase-memory-mcp:end

This will reduce the dependencies and keep your pi extension to be as minimal ...

Alternatives considered

No response

[avifenesh]

Thanks, this is a reasonable ask. A generated pi extension that shims the CLI (like the codebase-memory-mcp example you show) fits the design here well, since every MCP tool already has a stable CLI entry point to wrap. I will look at emitting one from the tool registry on install/update so it cannot drift. If you want to take a swing at it in the meantime, a PR following your sketch is very welcome.

[tan-yong-sheng]

Hi @avifenesh , I did try to vibe code the pi extension for this, yet 2 of the mcp functions (i.e., perform_action and set_value) are failing because it requires the stateful mcp connection, which if without pi-mcp-adapter (or if we're not using modelcontextprotocol/sdk in slim-computer-use-linux.ts), it is hard to have that stateful mcp connection ... Below is half-baked scripts I vibe-coded:

File: ~/.pi/agent/extensions/slim-computer-use-linux.ts
(which only expose 1 native tool in pi extension with 18 subcommands, where 2 subcommands are failing and 1 is partial success because that setup_window_targeting function is not working in linux mint xfce and seems only supporting wayland). With this code design, the system prompts drop to around 300-400 tokens from 4k tokens (not measured properly, just rough guestimate)

// slim-computer-use-linux.ts
//
// Slim wrapper for the computer-use-linux MCP server: ONE LLM-callable
// tool, `computer_use_linux`, that dispatches to any of the underlying
// 18 MCP tools. Per-tool JSON Schemas are dropped from the system
// prompt to save ~14,500 tokens; per-tool docs live in the
// `computer-use-linux` skill.
//
// Installation path of computer-use-linux cli binary: https://github.com/agent-sh/computer-use-linux
//
// Schema design:
//   tool: required enum of the live MCP tools (sourced from
//         computer-use-linux mcp at the last refresh of this file).
//         Refresh the enum below if upstream renames/adds tools.
//   args: free-form object passed through to `tools/call`.
//         Per-tool field shape is documented in the computer-use-linux
//         skill; mistyped args fail at the MCP layer, not the schema
//         layer. Same trade-off slim-bg-task.ts and slim-cbmem.ts make.

import { spawn, execFileSync } from 'node:child_process';
import { existsSync } from 'node:fs';
import { join } from 'node:path';
import { Text } from '@earendil-works/pi-tui';

const TIMEOUT_MS = Number(process.env.COMPUTER_USE_LINUX_TIMEOUT_MS || 60_000);
const INSTALL_URL = 'https://github.com/agent-sh/computer-use-linux';

// Pulled from the live MCP `tools/list` (computer-use-linux latest at
// the time of this file's last refresh). The list may drift if
// upstream adds or renames tools; refresh this enum and reload.
const TOOLS = [
  'activate_window',
  'click',
  'doctor',
  'drag',
  'focused_window',
  'get_app_state',
  'list_apps',
  'list_windows',
  'move_window',
  'perform_action',
  'press_key',
  'resize_window',
  'screenshot',
  'scroll',
  'set_value',
  'setup_accessibility',
  'setup_window_targeting',
  'type_text',
];

const TOOL_DOC_HINT =
  'Dispatch a computer-use-linux MCP tool. `tool` selects which one ' +
  'of 18 tools to call (activate_window, click, doctor, drag, ' +
  'focused_window, get_app_state, list_apps, list_windows, ' +
  'move_window, perform_action, press_key, resize_window, ' +
  'screenshot, scroll, set_value, setup_accessibility, ' +
  'setup_window_targeting, type_text). `args` is a free-form object ' +
  'whose shape is per-tool, read the `computer-use-linux` skill ' +
  'before calling for the first time to learn each tool\'s required ' +
  'and optional fields.';

/**
 * Resolve the path to the `computer-use-linux` binary.
 *
 * Order of precedence:
 *   1. Explicit override: COMPUTER_USE_LINUX_BIN env var.
 *   2. Anything on PATH (via `which`).
 *   3. Common install prefixes: npm-global bin, cargo bin, linuxbrew,
 *      /usr/local/bin, /usr/bin, ~/.local/bin, ~/.npm-global/bin.
 *   4. Hardcoded legacy fallback.
 *
 * Inlined from computer-use-linux.ts to keep this file self-contained.
 */
function resolveBin() {
  const env = process.env.COMPUTER_USE_LINUX_BIN;
  if (env) {
    return existsSync(env) ? env : null;
  }

  const tryPaths = [];
  try {
    const w = execFileSync('which', ['computer-use-linux'], { stdio: ['ignore', 'pipe', 'ignore'] })
      .toString()
      .trim();
    if (w) tryPaths.push(w);
  } catch { /* which not available or not on PATH */ }

  const dirs = [];
  try {
    const npmRoot = execFileSync('npm', ['root', '-g'], { stdio: ['ignore', 'pipe', 'ignore'] })
      .toString()
      .trim();
    if (npmRoot) dirs.push(join(npmRoot, '..', 'bin'));
  } catch { /* npm not available */ }
  const cargoHome = process.env.CARGO_HOME || join(process.env.HOME || '', '.cargo');
  dirs.push(join(cargoHome, 'bin'));
  dirs.push(join(process.env.HOMEBREW_PREFIX || '/home/linuxbrew/.linuxbrew', 'bin'));
  dirs.push('/usr/local/bin', '/usr/bin', '/opt/computer-use-linux/bin');
  if (process.env.HOME) {
    dirs.push(join(process.env.HOME, '.local', 'bin'));
    dirs.push(join(process.env.HOME, '.npm-global', 'bin'));
  }
  for (const d of dirs) tryPaths.push(join(d, 'computer-use-linux'));

  for (const p of tryPaths) {
    if (p && existsSync(p)) return p;
  }
  return null;
}

const BIN = resolveBin();

/**
 * Spawn one `computer-use-linux mcp` child, run the JSON-RPC handshake,
 * and resolve with the parsed result for the request whose `id` matches
 * `wantId`. Returns the raw JSON-RPC message object.
 *
 * Inlined from computer-use-linux.ts; identical behavior.
 */
function mcpRpc(method, params, wantId, signal) {
  return new Promise((resolve, reject) => {
    const child = spawn(BIN, ['mcp'], { stdio: ['pipe', 'pipe', 'pipe'] });
    let buf = '';
    let err = '';
    let initDone = false;
    let settled = false;
    const onAbort = () => {
      try { if (!child.killed) child.kill(); } catch { /* ignore */ }
    };
    if (signal) {
      if (signal.aborted) {
        try { child.kill(); } catch { /* ignore */ }
        return reject(new Error('cancelled'));
      }
      signal.addEventListener('abort', onAbort, { once: true });
    }
    const timer = setTimeout(() => {
      try { if (!child.killed) child.kill('SIGKILL'); } catch { /* ignore */ }
      reject(new Error(`computer-use-linux ${method} timed out after ${TIMEOUT_MS}ms`));
    }, TIMEOUT_MS);
    const finish = (fn) => {
      if (settled) return;
      settled = true;
      clearTimeout(timer);
      if (signal) signal.removeEventListener('abort', onAbort);
      try { if (!child.killed) child.kill(); } catch { /* ignore */ }
      fn();
    };
    child.stdout.on('data', (d) => {
      buf += d.toString();
      let i;
      while ((i = buf.indexOf('\n')) !== -1) {
        const line = buf.slice(0, i).trim();
        buf = buf.slice(i + 1);
        if (!line) continue;
        let msg;
        try { msg = JSON.parse(line); } catch { continue; }
        if (!initDone && msg && msg.id === 0 && msg.result) {
          initDone = true;
          try {
            child.stdin.write(JSON.stringify({ jsonrpc: '2.0', method: 'notifications/initialized' }) + '\n');
          } catch { /* ignore */ }
          child.stdin.write(JSON.stringify({
            jsonrpc: '2.0',
            id: 1,
            method,
            params: params || {},
          }) + '\n');
        } else if (msg && (msg.id === wantId || msg.error)) {
          if (msg.error) return finish(() => reject(new Error(String(msg.error.message || msg.error))));
          finish(() => resolve(msg));
        }
      }
    });
    child.stderr.on('data', (d) => { err += d.toString(); });
    child.on('error', (e) => finish(() => reject(new Error(`failed to spawn ${BIN}: ${e && e.message ? e.message : e}`))));
    child.on('close', () => {
      if (!settled) finish(() => reject(new Error(`computer-use-linux ${method} closed with no result (stderr: ${err.trim() || 'empty'})`)));
    });
    child.stdin.write(JSON.stringify({
      jsonrpc: '2.0',
      id: 0,
      method: 'initialize',
      params: {
        protocolVersion: '2024-11-05',
        capabilities: {},
        clientInfo: { name: 'pi-slim-computer-use-linux', version: '0.1.0' },
      },
    }) + '\n');
  });
}

async function callMcpTool(name, args, signal) {
  const res = await mcpRpc('tools/call', { name, arguments: args || {} }, 1, signal);
  return res.result;
}

export default function (pi) {
  if (!BIN) {
    pi.on('session_start', async (_event, ctx) => {
      ctx.ui.notify(
        `slim-computer-use-linux: binary not found on PATH or in any known install prefix. ` +
        `Set COMPUTER_USE_LINUX_BIN, or install via "npm i -g @agent-sh/computer-use-linux". ` +
        `Docs: ${INSTALL_URL}`,
        'warning',
      );
    });
    return;
  }

  pi.registerTool({
    name: 'computer_use_linux',
    label: 'computer-use-linux',
    description: TOOL_DOC_HINT,
    parameters: {
      type: 'object',
      properties: {
        tool: { type: 'string', enum: TOOLS },
        args: { type: 'object' },
      },
      required: ['tool', 'args'],
    },

    // Custom TUI rendering so the call row reads like
    //   computer_use_linux click  x=400 y=300 button=left click_count=1
    // instead of dumping the raw { tool, args } JSON object.
    renderCall(args, theme, _context) {
      const params = (args && typeof args === 'object') ? args : {};
      const toolName = typeof params.tool === 'string' ? params.tool : '?';
      const inner = (params.args && typeof params.args === 'object') ? params.args : {};
      const parts = [];
      for (const [k, v] of Object.entries(inner)) {
        if (v === undefined || v === null || v === '') continue;
        let s;
        if (typeof v === 'string') {
          s = v.length > 60 ? v.slice(0, 57) + '…' : v;
          if (/\s|["\\]/.test(s)) s = JSON.stringify(s);
        } else if (typeof v === 'object') {
          s = JSON.stringify(v);
          if (s.length > 60) s = s.slice(0, 57) + '…';
        } else {
          s = String(v);
        }
        parts.push(`${k}=${s}`);
      }
      const line =
        theme.fg('toolTitle', theme.bold('computer_use_linux ')) +
        theme.fg('accent', toolName) +
        (parts.length ? '  ' + theme.fg('muted', parts.join(' ')) : '');
      const text = (_context.lastComponent) ?? new Text('', 0, 0);
      text.setText(line);
      return text;
    },

    renderResult(result, options, theme, context) {
      const text = (context.lastComponent) ?? new Text('', 0, 0);
      const callArgs = (context.args && typeof context.args === 'object') ? context.args : {};
      const toolName = typeof callArgs.tool === 'string' ? callArgs.tool : '?';
      const badge =
        theme.fg('muted', '[') +
        `tool=${toolName}` +
        (result.isError
          ? ' | ' + theme.fg('error', 'error')
          : ' | ' + theme.fg('success', 'ok')) +
        theme.fg('muted', ']');

      if (!options.expanded) {
        text.setText(badge);
        return text;
      }

      const body = (result.content || [])
        .filter((c) => c && c.type === 'text')
        .map((c) => c.text)
        .join('\n');
      text.setText(body ? `${badge}\n\n${body}` : badge);
      return text;
    },

    async execute(_toolCallId, params, signal) {
      const tool = params?.tool;
      const args = params?.args ?? {};
      if (typeof tool !== 'string' || !TOOLS.includes(tool)) {
        return {
          content: [{ type: 'text', text: `error: unknown tool "${tool}". Read the computer-use-linux skill for the supported list.` }],
          isError: true,
        };
      }
      try {
        const result = await callMcpTool(tool, args, signal);
        const content = result && Array.isArray(result.content) && result.content.length
          ? result.content
          : [{ type: 'text', text: JSON.stringify(result ?? {}) }];
        return { content, details: result };
      } catch (err) {
        return {
          content: [{ type: 'text', text: `error: ${err.message}` }],
          details: { error: err.message },
          isError: true,
        };
      }
    },
  });
}

[tan-yong-sheng]

The above scripts while saving tokens, yet it got 1 weakness which is the llm could not know the schema for all that 18 subcommands, so I have asked the llm to call ~/.pi/agent/skills/computer-use-linux/SKILL.md whenever it triggers that computer-use-linux native tool (which is the above script) ...

The skill folder structure I plan:

computer-use-linux
  SKILL.md
  PI_EXTENSION.md
  TROUBLESHOOTING.md

and the content as below:

[tan-yong-sheng]

1. SKILL.md

--- name: computer-use-linux description: "Drive a local Linux desktop through the computer-use-linux MCP server (https://github.com/agent-sh/computer-use-linux). Use when the user asks to click, type, screenshot, scroll, or otherwise operate a Linux GUI app, or when the agent needs the AT-SPI accessibility tree, compositor window list, or a desktop screenshot." author: local license: MIT platforms: [linux] ---

computer-use-linux

The computer-use-linux MCP server exposes 18 tools over JSON-RPC. Most coding agents (Claude Code, Codex, Cursor, Hermes, Pi) connect to it as a stdio MCP server, after which each tool appears as a normal agent tool. The per-tool field names, required fields, and gotchas are not self-evident from the tool descriptions alone; this skill is the consolidated reference.

Read this skill when you first need to call the MCP server, or when a call returns a field-name or selector error you do not recognise.

Pi adapter

When using Pi's single computer_use_linux wrapper, read PI_EXTENSION.md before the first call.
It contains the Pi-specific MCP envelope, context-budget rules, and failure signatures.

Calling the server

The MCP server takes one stdio process; each request is one JSON-RPC line on its stdin. Agents differ in how they expose the call.

Direct JSON-RPC (any MCP host):

{ "jsonrpc": "2.0", "id": 1, "method": "tools/call",
  "params": { "name": "screenshot", "arguments": { "max_bytes": 200000 } } }

The result.content is an MCP content array; the same shape the tool descriptions promise. The result.structuredContent is a parallel JSON object when the tool has one.

Pi extension shorthand (if the local Pi extension is installed): the same 18 tools are reachable through one tool registered as computer_use_linux. The body shape is the same call but flattened:

{ "tool": "screenshot", "args": { "max_bytes": 200000 } }

Top-level fields alongside tool are ignored. The two forms are equivalent; pick whichever the host surfaces.

Glossary

Four named concepts cover most of what the body otherwise has to spell out. Use them.

• target — the window selector. Pass exactly one of: window_id, pid, app_id, wm_class, title, tty, terminal_pid, terminal_command, terminal_cwd. The most specific one wins.
• snapshot — the result of one get_app_state call. The only legal source of element_index and element_identifier; valid only until the screen changes. A stale snapshot makes perform_action / set_value return no cached node.
• bound - output-size controls: max_bytes, max_width, max_height, max_depth, max_nodes, scale, quality, and include_screenshot. Use them to keep tool results under Pi's context budget.
• mutator — a tool that can change real app state: click, drag, press_key, type_text, perform_action, set_value, move_window, resize_window, activate_window. Treat like bash: confirm before destructive calls.

The 18 tools

tool	required	input modes	notes
doctor	—	—	session start. Check readiness.can_build_accessibility_tree, can_query_windows, can_send_development_input, blockers
list_apps	—	—	enumerate backends the server can see
list_windows	—	—	returns bounds; lets you click on a known window without a screenshot
focused_window	—	—	which window has keyboard focus
screenshot	-	target optional	always bound the output (max_bytes ≤ 200000); use smaller dimensions or JPEG when acceptable
get_app_state	-	target optional	produces a snapshot; always pass max_depth and max_nodes, and set include_screenshot: false unless visual data is required
activate_window	—	target	compositor must permit; some selectors fail on Wayland without the GNOME Shell extension
click	one of: x+y / element_index / name+role+text+states	coord, element, semantic	optional button, click_count, relative, target
drag	start_x, start_y, end_x, end_y	absolute pixels	field names are fixed — do NOT use x_start / x_end / from_x / to_x
press_key	key	key chord	grammar case-insensitive, hyphens ignored: Return, ctrl+l, Ctrl+Shift+T, super+up, f5. Compositor-chords like Super+Up may be eaten by GNOME — fall back to ctrl+alt+key
type_text	text	literal	prefer a target over relying on current focus
move_window	x, y	+ target	desktop pixel coords of the new frame top-left
resize_window	width, height	+ target	pixels
scroll	direction	+ optional x/y, pages, element_index, target	uinput backend may silently no-op; verify with a follow-up screenshot if the scroll matters
perform_action	snapshot	element_index / element_identifier / name+role+text+states	action defaults to the element's primary. no cached node means the snapshot is stale — re-snapshot
set_value	value	+ snapshot (same selectors as perform_action)	for text inputs the value is the new text; for sliders pass a numeric string and role: "slider"
setup_accessibility	—	—	one-shot; enables GNOME AT-SPI via gsettings
setup_window_targeting	—	—	one-shot; installs the GNOME Shell extension. On GNOME Wayland ask the user to log out and back in

Context budget

Tool results become model input in full.
A collapsed result in the Pi TUI still consumes the same context.

Use this safe observation profile unless the task requires more detail:

• get_app_state: max_depth: 8, max_nodes: 200, and include_screenshot: false.
• screenshot: max_bytes: 200000, with smaller dimensions or JPEG quality when acceptable.
• Increase one bound at a time and verify that the returned result remains usable.

Procedure

1. Run doctor when readiness is unknown.
2. If a readiness flag is false, run its matching setup_* tool and re-check.
3. Use list_windows or focused_window to identify the target before requesting an accessibility tree.
4. Request one bounded get_app_state snapshot when semantic targeting is needed.
5. Use selectors only from the fresh snapshot, then reacquire a snapshot after the UI changes.
6. Pass exactly one target selector at the top level of the tool's argument object.
7. Prefer one desktop observation or mutator at a time because desktop state is mutable.

Verification

doctor reports ready when readiness.can_register_mcp_tools, can_build_accessibility_tree, can_query_windows, can_send_development_input are all true and readiness.blockers is empty. A follow-up screenshot with max_bytes=200000 should return a non-empty image, and list_windows should return at least one window. If any of those fail, the backend is not actually wired up and the next mutating call will silently no-op.

See also

TROUBLESHOOT.md is symptom-driven: reach for it when a tool returns an error you don't recognise, when doctor reports a backend flag false, or when a mutating call appears to do nothing. This file is the happy path; that file is the recovery path.


[tan-yong-sheng]

2. PI_EXTENSION.md

# computer-use-linux: Pi adapter

This file contains only Pi-specific guidance for the computer_use_linux wrapper.
The upstream tool contract belongs in SKILL.md and the upstream repository.

MCP envelope

Pi registers one tool named computer_use_linux.
Its argument body selects the upstream tool and carries that tool's arguments:

{
  "tool": "get_app_state",
  "args": {
    "title": "Thunar",
    "max_depth": 8,
    "max_nodes": 200,
    "include_screenshot": false
  }
}

The tool name computer_use_linux is selected by the host.
The call body therefore starts with the upstream selector tool, not another nested computer_use_linux object.

Output bound

Every observation spends Pi context budget because the complete returned content is appended to the conversation.
A collapsed result in the Pi TUI is only a display choice and does not reduce the model payload.

Use this bounded observation profile:

• get_app_state: set max_depth and max_nodes on every call.
• Set include_screenshot: false when accessibility data is sufficient.
• screenshot: set max_bytes to at most 200000; use smaller dimensions or JPEG quality when visual detail permits.
• Request one observation at a time and keep only the latest snapshot in active reasoning.

An unbounded accessibility tree is a tree bomb.
In an observed session, two get_app_state results were 641,849 and 408,292 characters.
One contained a screenshot payload, while the other contained no screenshot and still reached 408,292 characters.

Snapshot discipline

A get_app_state result is a fresh snapshot.
Use its element_index or element_identifier only until the target UI changes.
After a mutator changes the screen, request a new bounded snapshot before another semantic action.

Use list_windows or focused_window to identify a target before requesting a tree.
Pass the chosen window selector directly in the inner args object.

Target shape

Target selectors are top-level fields in the inner arguments:

{
  "tool": "activate_window",
  "args": { "window_id": 75515016 }
}

Use one selector such as window_id, pid, app_id, wm_class, title, tty, terminal_pid, terminal_command, or terminal_cwd.
The adapter passes these fields through to the MCP server.

Pi-specific failure signatures

• Validation failed: tool: must have required properties tool: use the MCP envelope above.
• backend: unknown with a request for a target: move the selector out of target: {} and place it directly in args.
• No cached accessibility node ...: obtain a fresh snapshot and use its selector.
• context_length_exceeded: proactive compaction threshold reached (...): the installed Pi Cliproxy provider extension generated a synthetic compaction signal after its context estimate crossed the threshold.
  The usual trigger is an oversized observation, especially an unbounded get_app_state result.
• scroll reports a missing ydotool socket: the desktop input backend needs ydotoold; verify the result with a follow-up observation.
• setup_window_targeting reports missing GNOME commands: the current desktop is not GNOME, so use the desktop's supported X11 or compositor backend.

Compact operating loop

1. Establish readiness with doctor when needed.
2. Identify the window with list_windows or focused_window.
3. Take one bounded observation using the output-bound profile.
4. Perform one action using the fresh snapshot or an explicit target.
5. Re-observe after the UI changes.

Read TROUBLESHOOT.md when a backend or selector error persists.
Read the upstream SKILL.md for the complete tool contract.


[tan-yong-sheng]

3. TROUBLESHOOTING.md

# computer-use-linux troubleshooting

Symptom → likely cause → recovery. Read this only when something is already broken.

doctor flags a false readiness

doctor is the single source of truth. If any readiness.* flag is false or readiness.blockers is non-empty, fix that first; downstream tools will silently no-op otherwise.

symptom (from doctor)	likely cause	recovery
readiness.can_build_accessibility_tree: false	AT-SPI bus not running, or toolkit accessibility off	computer-use-linux setup_accessibility; restart the target GTK/Qt/Electron app
readiness.can_query_windows: false (GNOME Wayland)	GNOME Shell extension for window control not installed	computer-use-linux setup_window_targeting; ask the user to log out and back in if the extension was newly installed
readiness.can_send_development_input: false	no input backend is usable	see ydotool / uinput / xdotool sections below
readiness.blockers non-empty	read the blocker message; it names the missing piece	follow the upstream message

Always re-run doctor after each fix and confirm readiness.blockers == [] before resuming.

ydotool: "no connectable ydotool socket" / "ydotoold exit status 1"

ydotool is the fallback input backend used when uinput is not available (typically Wayland without permission to write /dev/uinput). The daemon (ydotoold) exposes its socket under $XDG_RUNTIME_DIR, which for a normal user session is /run/user/$UID. doctor looks for it at /run/user/$UID/.ydotool_socket and falls back to /tmp/.ydotool_socket.

If the CLI is installed but the socket is missing, the daemon is not running for the current user session.

Recovery, in order:

1. Install ydotool from your distro's package manager. The package must be >= 1.0.3; doctor rejects older CLIs. Build from source as a last resort: https://github.com/ReimuNotMoe/ydotool.
2. Enable and start the per-user systemd service:

   systemctl --user enable --now ydotoold

   This is the right unit. The socket path is set by Environment=XDG_RUNTIME_DIR=/run/user/$UID inside the unit, which is why a per-user service is required — a system-wide ydotoold would look for the socket at /run/ydotool.sock and the server would still not find it.
3. Confirm the socket exists:

   ls -la "$XDG_RUNTIME_DIR/.ydotool_socket"

4. Re-run computer-use-linux doctor | jq .readiness and confirm readiness.can_send_development_input: true.

If the socket still does not appear after step 2:

• journalctl --user -u ydotoold -n 50 — the daemon's own log usually names the missing capability (most often CAP_SYS_ADMIN not granted, or /dev/uinput not writable by the user session).
• The per-user service may need AmbientCapabilities=CAP_SYS_ADMIN CAP_DAC_OVERRIDE and a writable /dev/uinput. If the system is locked down by a security profile (nono, firejail, SELinux enforcing), loosen the profile for the user session, or pick a different input backend.

uinput: "/dev/uinput permission denied"

The user session must be able to open /dev/uinput for write. Most distros solve this with a uinput group or a udev rule that grants the active console user access.

Recovery:

• Quick check: ls -la /dev/uinput — should be writable by the user, or by a group the user is in.
• If the device exists but the group is wrong: add a udev rule under /etc/udev/rules.d/90-uinput.rules:

  KERNEL=="uinput", GROUP="input", MODE="0660"
  

  Then sudo udevadm control --reload-rules && sudo udevadm trigger and log out and back in. Add the user to the input group: sudo usermod -aG input $USER.
• On Wayland without logind console access, uinput is often denied by design. Switch to ydotool (above) or the RemoteDesktop portal.

xdotool: "xdotool missing" on X11

Install xdotool. On Debian/Ubuntu: sudo apt install xdotool. On Fedora: sudo dnf install xdotool. On Arch: sudo pacman -S xdotool. The server needs XTEST support compiled in; most distro packages include it by default. Verify with xdotool key ctrl+alt+t — a terminal should open.

"scroll did nothing"

scroll is the most failure-prone input tool. It silently no-ops when:

• the ydotool backend is selected but ydotoold is not running (see above);
• the uinput backend is selected but the session cannot write /dev/uinput;
• the targeted window is at rest and the scroll amount rounds to zero at the kernel layer.

The tool returns ok regardless. Always verify the result with a follow-up screenshot or get_app_state if the scroll matters; do not trust the return value alone.

"no cached node" on perform_action / set_value

These two tools need a recent snapshot (see SKILL.md glossary). The error means either you have not called get_app_state this turn, or the screen has changed since you did. Re-run get_app_state for the target and use the fresh element_index. element_identifier is stable across snapshots but only for the lifetime of a single process tree of the target app; if the app was restarted, re-snapshot.

"field name" errors

These four tools have field names that an agent regularly gets wrong on first contact. The correct names are fixed in the server; there is no override.

• drag wants start_x, start_y, end_x, end_y (not x_start / x_end / from_x / to_x).
• press_key wants key (not name / text).
• type_text wants text (not value / string).
• set_value wants value (not text / input).

"compositor ate the key chord"

Super+Up/Down/Left/Right and similar window-manager chords are frequently intercepted by the desktop environment before reaching the focused app. If press_key with super+up reports ok but the window does not move, fall back to ctrl+alt+key, or drive the move through move_window instead.

"tool result is too big"

get_app_state and full-screen screenshot can each dump 100-500 KB of JSON / base64 image. Two in a row is enough to overflow the host's proactive-compaction threshold. bound them: max_bytes=200000 for screenshots, max_depth=10 and max_nodes=200 for app state. The skill's Procedure section covers the happy path; this file is the recovery path.


[tan-yong-sheng]

Perhaps this change is not that suitable for this repo, this pi extension I vibe code is to try to reduce the system prompt token to the minimal (around 300-500 tokens if the coding agents doesn't use computer-use-linux native tool in a session that doesn't need this capabilities instead of always loaded) via only introducing 1 tool only while try to retain the existing functions of the pi extension with 18 tools, still in testing, not sure if it will be a success or not ... Perhaps can close this issue if don't want to introduce the maintenance effort like importing modelcontextprotocol/sdk inside your codebase, writing skill.md to support different harness like pi agent or opencode, etc ...

[avifenesh]

Shipped in v0.5.0.

Pi now installs directly with:

pi install npm:@agent-sh/computer-use-linux

No pi-mcp-adapter dependency or ~/.pi/agent/mcp.json write is required. The package starts with one small loader tool, exposes exact generated native schemas only when needed, and keeps one session-scoped MCP process so stateful calls such as perform_action and set_value work. Selected tools persist across Pi reloads, outputs are bounded, and binary/catalog drift is rejected.

I verified the published npm package from a fresh registry install, including the MCP safety contract and Pi real reload lifecycle. Thanks for the detailed prototype and especially the stateful-connection report; it directly informed the final design.

Closing as completed.

[avifenesh]

A bit more detail on what we ended up building, because it is deliberately different from the single free-form dispatcher prototype:

• Pi starts with only one small computer_use_linux_tools loader active, so sessions that do not need desktop control do not carry all Computer Use schemas in the prompt.
• When desktop control is needed, the loader enables only the selected computer_use_linux_* tools. They remain normal first-class Pi tools with their exact schemas, generated from the live MCP tools/list, rather than a free-form args object the model has to learn from documentation.
• All enabled tools share one lazily started, session-scoped MCP process. This preserves the accessibility snapshot/cache, which is the important part for perform_action and set_value after get_app_state.
• The selected tool set survives Pi reloads, but only for the current session branch.
• Cancellation and timeouts are forwarded without replaying an action whose result is uncertain. Text/images are bounded, and the extension rejects a binary if its version or tool catalog does not match the packaged schemas.
• The npm package contains the extension, MCP client bundle, and skill itself; it does not write Pi MCP configuration. If an old adapter registration is still present, it is reported for manual cleanup rather than modified automatically.

For Mint/Xfce specifically, setup_window_targeting is the optional GNOME Shell helper and is not expected to install on Xfce. Xfce/X11 window operations should go through the generic X11 backend instead; doctor and list_windows should show whether that path is available.

Could you try @agent-sh/computer-use-linux@0.5.0 in your actual Pi workflow and let me know whether this is helping? I am especially interested in:

1. the idle-session prompt/token footprint,
2. whether perform_action and set_value now work reliably after a fresh get_app_state,
3. whether the loader finds/enables the tools naturally enough, and
4. anything still confusing or broken on Mint/Xfce.

If something fails, the Pi version, desktop/session type, computer-use-linux doctor output, and exact tool error would be very useful.

[tan-yong-sheng]

Thanks a lot for the support, i just put my comment here first:

Hopefully, there is fallback auto-resolve bin path like which computer-use-linux to confirm if the cli binary is truly installed or not, for example, in my environment, it has a false negative because i am installing node via homebrew and thus when i run npm i -g computer-use-linux it will add that cli binary to /home/linuxbrew/.linuxbrew/bin/computer-use-linux

But ya, I can solve this by manually setting that COMPUTER_USE_LINUX_BIN myself

[avifenesh]

Thanks a lot for the support, i just put my comment here first:

Hopefully, there is fallback auto-resolve bin path like which computer-use-linux to confirm if the cli binary is truly installed or not, for example, in my environment, it has a false negative because i am installing node via homebrew and thus when i run npm i -g computer-use-linux it will add that cli binary to /home/linuxbrew/.linuxbrew/bin/computer-use-linux

But ya, I can solve this by manually setting that COMPUTER_USE_LINUX_BIN myself

I didnt know you can use homebrew on linux, ill reopen the issue in order to fix that