Hive attaches pluk — an
external, per-agent output-capture layer — to every agent's tmux session. Pluk
writes a structured JSONL event stream for that session, and other agents (or
operators) can read it for read-only peer awareness without attaching to the
agent's tmux socket.
This page documents what pluk is, the log format Hive's own code depends on,
and how hive-panes uses it. It does not document pluk's internals — pluk is
a separate npm package (@kubestellar/pluk), not part of this repo — only the
parts of its output that Hive code and scripts actually consume.
Pluk is installed in the container image as an npm global package
(@kubestellar/pluk, pinned by PLUK_VERSION in Dockerfile). It is not a
Hive-authored component; Hive only shells out to it.
When the agent manager creates a tmux session for an agent
(pkg/agent/manager.go), it looks up pluk on PATH and, if found, attaches
it to that session's pane with:
pluk watch <tmux-session> --cli=<backend>
via tmux pipe-pane. <backend> is the agent's configured CLI backend
(claude, codex, etc.), defaulting to claude when the backend is empty or
is an inference backend. If the pluk binary is not on PATH, attachment is
skipped silently — the agent still runs, it just has no peer-visible log.
Pluk also exposes a send subcommand (pluk send hive-<session> --text=<message> --enter), which writes through a named FIFO and confirms
delivery via a command_received event rather than relying on tmux send-keys. Hive no longer calls it: its only caller was the retired inception
workflow's fallback kick path.
- Pluk attachment is automatic for every agent tmux session Hive creates — no per-agent configuration is required.
- It requires the
plukbinary to be present onPATHin the container. It is installed unconditionally in the shipped image, but a custom or stripped image without it will run agents with no pluk logs and nohive-panesoutput. - The run directory (
/var/run/pluk—logs/andcommands/subdirectories) is created ahead of time in the image (Dockerfile) and re-created defensively before each attach byensurePlukRunDirs(pkg/agent/pluk_dirs.go). Both subdirectories are0770plus setgid, owned by thedevuser and the sharednodegroup, so any agent UID in that group can read another agent's log without world-readable permissions. - If directory setup fails, Hive logs a warning
(
"pluk run directory setup failed; pluk publisher may be degraded") and still attempts the attach — a degraded or absent log is non-fatal to the agent itself.
Each agent's events land in one file per tmux session:
/var/run/pluk/logs/<tmux-session>.jsonl
Session names follow Hive's hive-<agent-name> convention (e.g.
hive-brainstorm.jsonl for the brainstorm agent). Each line is one JSON
object, newline-delimited (JSONL), appended as the agent produces output.
The file is written by a shell redirect, not by pluk. pluk watch classifies
its stdin and prints events to stdout; it opens no file of its own. tmux pipe-pane feeds the pane into that command's stdin but does nothing with its
stdout, so the publisher invocation in pkg/agent/manager.go has to end in
>> /var/run/pluk/logs/<session>.jsonl for anything to be written at all. It
did not between #1759 — which
replaced the Go pluk-publish binary, a publisher that wrote the log itself,
with the TypeScript pluk watch — and
#4285. Throughout that
window this directory was empty and every consumer below was reading files that
did not exist.
Hive pre-creates each session log 0660 and dev:node before attaching
pipe-pane. hive-panes is run by one agent to read every other agent's log,
and agent UIDs differ, so the shared node group is what makes cross-agent
reads work — leaving creation to the shell's >> would take whatever umask the
pane shell carries and a tight one yields a log no peer can read.
Nothing rotates or caps these files. hive-panes only ever reads the last 2000
lines, so no consumer needs the history, but the files themselves grow for the life of the container.
The event envelope, as read by Hive's own code, is:
{"type": "<event-type>", "session": "<tmux-session>", "data": {...}}type is a string discriminator; data is a flat string-to-string map whose
keys depend on type. Hive's dashboard used to subscribe to this stream for
the brainstorm session (the inception watcher, retired with the inception
workflow); hive-panes.sh is the reader today. The event types:
-
raw_output— captured pane output. The text is ondata.line, and only there.hive-panes.shhas always readdata.line; the retired dashboard subscriber readdata.messageuntil #4285 corrected it, which had left its wholeraw_outputarm dead. Earlier revisions of this page advised new consumers to check both keys; that was wrong, and generalised from theerror/rate_limitshape below. pluk builds the event ascreateEvent(..., 'raw_output', { line })indist/classifier.js.raw_outputis opt-in:pluk watchdefaultsincludeRawto false, so the publisher has to pass--include-rawor no such event is ever emitted. It is the only event typehive-panes.shconsumes, so peer awareness depends on that flag. -
state_change—data.stateis"working"or"idle". -
rate_limit—data.messagedescribes a provider rate limit. -
error—data.messagedescribes an agent-side error. -
tool_call_completed— no payload; emitted after a tool call finishes.
No in-process Hive consumer tails these logs today. They exist for
hive-panes and manual/operator inspection.
hive-panes (installed by src/deploy/hive-panes.sh when the deployment
includes that script) is a read-only peer-awareness command available inside
an agent's shell:
hive-panes [lines] # default: 30
It:
- Lists every
hive-*.jsonlfile in/var/run/pluk/logs. - Skips the file for the calling agent's own session, identified by the
HIVE_PROXY_AGENTenvironment variable (so an agent never has to filter its own output out of the results). - For each remaining file, takes the last 2000 raw lines, parses each as
JSON, keeps only entries where
type == "raw_output", and extractsdata.line. - Strips ANSI escape sequences and cursor-position control codes, drops
blank lines, and prints the last
Ncleaned lines per agent under a banner naming that agent.
It never attaches to another agent's tmux socket — it only reads the JSONL
file on disk, so it cannot send input or otherwise disturb another agent's
session. If /var/run/pluk/logs does not exist, it exits with an error
rather than printing empty output, which is the fastest signal that pluk
never attached (binary missing, or the agent manager could not create the
run directory).
This repo does not implement rotation or a retention policy for
/var/run/pluk/logs/*.jsonl — there is no cron, size cap, or truncation
logic in pkg/agent or src/deploy for these files. /var/run is typically
an in-memory or ephemeral filesystem inside the container, so logs do not
outlive the agent's tmux session/container lifecycle, but within that
lifetime the files grow unbounded from Hive's side. Any rotation behavior
(e.g. size-based rollover) would come from pluk itself; check the installed
@kubestellar/pluk version's own behavior if you need to reason about disk
growth, since Hive does not manage it.
- Use
hive-panes [lines]when you need situational awareness of what other agents in the fleet are currently doing — for example, before duplicating work, or to check whether another agent already touched a file you are about to change. - Treat its output as recent raw terminal text, not structured data — it is ANSI-stripped CLI output, not a machine-readable event feed. Do not parse it for automation; if you need structured events, read the JSONL files directly.
- Avoid depending on it being present. It requires
hive-panes.shto be deployed andplukto have attached successfully; on a deployment that omits either,/var/run/pluk/logswill not exist and the command exits with an error. Code and prompts should not treat pluk logs as a guaranteed input. - It is read-only by design: it cannot be used to send input to another
agent's session. Use each agent's own kick/messaging path
(
SendKick/pluk send) for that, nothive-panes.
Separate from pluk's JSONL stream, each agent's tmux session keeps a deep terminal scrollback:
- Agent sessions are created with tmux
history-limit50000 (tmux's own default is 2000, which silently truncated long runs — #3790). Override withHIVE_TMUX_HISTORY_LIMIT(positive integer); it must be set at session creation — raising it later never deepens an existing pane. The attach-time twinHIVE_TTYD_HISTORY_LIMIT(also 50000) only affects panes created after a browser attach. - The browser terminal attaches with tmux mouse mode on, so the scroll wheel drives copy-mode scrollback; hold Shift (Option on macOS) for native browser text selection (#3722).
- Full-log view/download:
GET /api/agents/{name}/logreturns the entire retained buffer for the agent's latest run astext/plain(add?download=1for aContent-Dispositiondownload namedhive-<agent>-<timestamp>.log) — #3711. In the dashboard these are the📄 full logand⬇ logcontrols on the agent card. Normal dashboard auth applies (any authenticated role); output passes through token redaction, so it is not a redaction bypass.
The live capture is scoped to the agent's current tmux session: a restart kills and recreates the session, and a hive upgrade replaces the whole container. Per-kick log archiving (below) is what preserves run logs across those boundaries.
Hive archives each kick's terminal output to a durable file so operators can read the logs of the last several runs — not just the latest — (#4296, #4295):
- When snapshots happen. The scrollback is captured to a file as soon
as the kick's turn is observed to end (the CLI is back at its idle prompt —
the same moment the
kick turn endedlog line and the kick outcome come from), and again before it can be destroyed: when the next kick is delivered (the previous kick's output is archived and the tmux history is cleared, so each archive covers exactly one kick), beforekill-sessionon an agent restart, and on graceful shutdown (SIGTERM — pod roll or hive upgrade) for every agent with un-archived kick output. There is one archive per kick: the later snapshot of the same kick replaces the turn-end one (latest content wins), so a daily-cadence agent whose container crashes hours after its turn ended — a hard kill that never runs the shutdown snapshot — still has that run under past kicks. - Where they live.
/data/logs/kicks/<agent>/<timestamp>-<reason>.log(override the root withHIVE_KICK_LOG_DIR)./datais the persistent volume on hosted hives, so archives survive agent restarts, pod rolls, and hive image upgrades. Each file starts with a small header (agent, archive time, snapshot reason, kick start time, prompt snippet, build/backend/model,source) followed by the raw scrollback.<reason>isturn-end,kick,restart, orshutdown— what wrote the archive, not how the kick ended. - Transcript source. The scrollback is what the CLI painted, so tool
output the CLI collapsed (
(N lines hidden)) is not in it.HIVE_KICK_LOG_SOURCE=transcript(orauto) builds the archive and the live full log from the CLI's own transcript instead — Copilot CLIevents.jsonland Claude Code session JSONL — rendered as plain text: timestamped turns, every tool call with its full arguments (the shell command on a$line) and its full output. Only entries since the kick started are included. A missing, stale or unparseable transcript falls back to the scrollback with one warning in the hive log, and the header'ssource:line says which was used. A transcript overHIVE_KICK_LOG_MAX_BYTESkeeps its head and tail around a[… N bytes omitted …]marker. The default,scrollback, is unchanged. The source is also a dashboard setting: Settings → Logging → Kick log source (config keygovernor.logging.kick_log_source) offers Scrollback, CLI transcript and Auto, applies to the next kick without a restart, and the kick history page shows each archive'ssource. AHIVE_KICK_LOG_SOURCEset on the deployment wins over the saved setting; the control is then shown disabled with the effective value and a "set byHIVE_KICK_LOG_SOURCE" note. - Retention. Per agent, the newest 10 archives are kept
(
HIVE_KICK_LOG_RETENTION;0disables archiving) within a 64 MiB per-agent size cap (HIVE_KICK_LOG_MAX_BYTES); the oldest files are pruned first and the newest archive is never deleted. An agent with less history than normal (fresh install, retention just lowered) simply lists fewer entries — never an error. - Dashboard access. The
🕘 past kickscontrol on the agent card opensGET /agents/{name}/kicks, an index of the live log plus every archived kick with view/download links. Programmatic access:GET /api/agents/{name}/kicks(JSON list, newest first) andGET /api/agents/{name}/kicks/{id}(text/plain;?download=1for an attachment). Same auth rule as the full-log endpoint (any authenticated role), and archived content passes through the same token redaction. - Sandboxed agents. A sandboxed kick (
sandbox.enabled: true, Podman orruntime: job) has no tmux session. When the run ends hive archives the container's captured stdout/stderr here instead, as<kick-start>-sandbox.logwithsource: sandbox-podmanorsource: sandbox-joband an outcome block (exit code, commits, branch, PR orpr_withheldreason, agent-report path and status, error) ahead of the output, followed by one---- verify: <cmd> (<verdict>) ----section perpublish.verifycommand that ran. For these agents the📄 full log/📋 log textcontrols andGET /api/agents/{name}/logserve the newest such archive — the only durable copy, since a completed Job is deleted at once and the sandbox workspace is swept after two hours — and an agent that has not run yet gets a plain "no archived run yet" message rather than a pane error.HIVE_KICK_LOG_RETENTION=0disables these archives too, and the card then has no log to show. - Limitations. The archive holds at most the retained scrollback
(
history-limit, 50000 lines by default), so an extremely long run self-truncates from the top. A hard kill (SIGKILL, OOM, node loss) skips the shutdown snapshot, so a turn still running at that moment loses whatever it produced after the last snapshot; turns that had already ended were archived at turn end, and the previous kicks' archives on/dataare unaffected.
src/deploy/README.md—hive-panes.shinventory entry.- Agent configuration — where
hive-panesis introduced as an agent-facing capability. - Architecture — system-level mention of the read-only peer-observation workflow.