Skip to content

Latest commit

 

History

History
751 lines (642 loc) · 39.8 KB

knowledge-curator.md

File metadata and controls

751 lines (642 loc) · 39.8 KB

Knowledge promotion configuration

The knowledge.curator block now controls only promotion of existing verified facts between llm-wiki layers. The earlier merged-PR extraction pipeline was never wired into any binary and has been removed rather than leaving a config-promised feature that did not run.

knowledge:
  enabled: true
  engine: llm-wiki
  curator:
    auto_promote_threshold: 0.9
Field Current behavior
auto_promote_threshold Defaults to 0.9 when knowledge is enabled. Promoter.AutoPromoteCandidates selects facts whose llm-wiki page has status == "verified" and confidence >= threshold.

Merged-PR extraction and its old scheduling/source-list knobs are intentionally not documented as supported configuration because no scheduler, CLI command, or HTTP endpoint triggered extraction.

Remote git sources (knowledge.git_sources)

knowledge.git_sources indexes markdown from a remote git repository (or a subdirectory of one) as a knowledge source, so agents get facts from an external repo — a runbook repo, an upstream docs repo, a shared pattern library — primed into their kicks the same way wiki-layer facts are. This is implemented and live, unlike curator scheduling above: pkg/knowledge/gitsource.go does the cloning, indexing, and periodic sync; cmd/hive/main.go:2303-2347 wires configured entries at startup.

knowledge:
  enabled: true
  git_sources:
    - name: dakota-skills            # required — display name and dedup/lookup key
      url: https://github.com/projectbluefin/dakota   # required — https:// only
      branch: main                   # optional — default "main"
      subpath: docs/skills           # optional — index only this subdirectory
      layer: project                 # optional — default "project"

Config fields (GitSourceConfigYAML, pkg/config/knowledge_config.go:58-65, mirrored by the runtime type GitSourceConfig, pkg/knowledge/gitsource.go:33-39):

YAML key Required Default Notes
name Yes — Display name; also the key used to look up/remove the source via the API.
url Yes — Git remote URL. https:// only — see Auth below.
branch No main (gitsource.go:66-68) Branch to shallow-clone (--depth 1 --branch <branch>). Connecting through the dashboard/API with a blank branch resolves the remote default (git ls-remote --symref <url> HEAD) and records it in the saved config.
subpath No — (whole repo) When set, only this subdirectory is checked out (git sparse-checkout, gitsource.go:199-219,232-248) and indexed.
layer No project when set via the API (api_knowledge.go:1032-1034); required, no code default, when set via hive.yaml One of personal, project, org, community — see Layer semantics below.

What "indexed" means

On connect, the source is shallow-cloned (depth 1) into <knowledge-data-dir>/git-sources/<slug> and the clone (or its subpath) is handed to a FileStore, which indexes markdown files under it (gitsource.go:78-114). If subpath doesn't exist after cloning, connection fails with subpath "<x>" not found after clone — check the path relative to the repository root, not the branch's top-level display in GitHub's UI.

Layer semantics

layer places the source's facts in the same 4-layer precedence used everywhere else in the knowledge system (pkg/knowledge/types.go:8-30, lower number = higher precedence, i.e. overrides on conflict):

Layer Precedence Typical use for a git source
personal 1 (highest) A private runbook repo only this operator's hive should see.
project 2 A repo scoped to the project this hive manages — the default.
org 3 A shared org-wide reference repo.
community 4 (lowest) A public upstream docs repo, e.g. a framework's own documentation.

Auth for private repos

There is no auth configuration for git sources. ensureCloned invokes a plain git clone with GIT_TERMINAL_PROMPT=0 (so a credential prompt fails instead of hanging) and no token, SSH key, or credential-helper wiring (gitsource.go:183-227,503-512). In practice this means:

  • A public HTTPS repo works with just url:.
  • A private repo will fail to clone — there is no field to supply a token or deploy key, and the code does not fall back to any host git credential store. Do not attempt to embed a token in the url (e.g. https://TOKEN@github.com/...); nothing in this codebase does that pattern for git sources, and hardcoding a token in hive.yaml is explicitly against the fail-closed secret-handling used elsewhere in this repo.
  • Only https:// (and, if HIVE_ALLOW_PRIVATE_GIT_SOURCE=true, http://) URLs validate at all; git@-style SCP syntax is explicitly rejected (gitsource.go:267-268,275-281).

If your knowledge source is private, treat this as unsupported today rather than assuming a missing config knob — see Open questions.

SSRF / URL hardening (operator-relevant)

ValidateGitSourceURLContext rejects URLs whose host resolves to a loopback, private, or link-local address (including the cloud metadata IP) before cloning, and fails closed on a DNS lookup error (gitsource.go:259-351). git itself is invoked with -c http.followRedirects=false so a remote can't 302 the clone to an internal address after validation passes (gitsource.go:479-501). An in-cluster git server (e.g. an internal GitLab) needs HIVE_ALLOW_PRIVATE_GIT_SOURCE=true set as an explicit opt-in (gitsource.go:293-297,368-370).

Refresh / staleness

Once connected, a background loop pulls and reindexes every 5 minutes (gitSourceSyncInterval, gitsource.go:25, StartSyncLoop, gitsource.go:161-180). A failed sync is logged at warn and the loop keeps retrying on the next tick — it does not surface as a dashboard alert, so staleness is only visible in the hive's logs (git source sync failed). There is no operator-facing "last synced" timestamp in GitSourceInfo (gitsource.go:531-540) — only whether the source is Ready and its current page count.

Static config vs. the runtime API

GET/POST/DELETE /api/knowledge/git-sources (owner-role only, pkg/dashboard/api_knowledge.go:980-986,988-1070,1072-1114) manage sources at runtime and are the same underlying list as knowledge.git_sources in hive.yaml — not a separate system:

  • POST connects a source immediately and, if it isn't already present (matched by url+subpath), appends it to Config.Knowledge.GitSources and persists the config (api_knowledge.go:1048-1066). A POST for a source that is only in hive.yaml but not yet connected in the running process (e.g. right after editing the file without restarting) will add a duplicate config entry once reconnected, since the dedup check is by URL+subpath against what's already in Config, not against what main.go loaded at boot.
  • DELETE disconnects the live source and removes matching entries from Config.Knowledge.GitSources, then persists (api_knowledge.go:1100-1110).
  • Editing git_sources: directly in hive.yaml takes effect on the next process restart (main.go's startup loop at cmd/hive/main.go:2420-2466); it does not hot-reload while the process is running. Use the API for a live change without a restart.

Diagnosing a failed source

  • Never appears / knowledge not enabled: if knowledge.enabled is false but git_sources is non-empty, startup auto-enables a minimal knowledge API (engine: file) just to host the git sources (main.go:2421-2427) — so a git source can work even with knowledge.enabled: false. If you still get "knowledge not enabled" from the API, no source has triggered that auto-enable yet (empty git_sources list).
  • Connect fails immediately: check the hive log for failed to connect git source with the URL and error (main.go:2437) — most often an SSRF-validation rejection, a bad branch name, or (for private repos) an authentication failure from git itself.
  • Connects but subpath errors: subpath "<x>" not found after clone — the path is wrong or doesn't exist on the configured branch.
  • Facts never show up in kicks: confirm the source reached Ready: true (GET /api/knowledge/git-sources) — the primer only registers a source's FileStore for priming after it reports ready (main.go:2450-2461).

Local vaults (knowledge.vaults)

knowledge.vaults lists local file-based (Obsidian-style) vaults that main.go auto-connects at boot (cmd/hive/main.go:2381-2417):

knowledge:
  vaults:
    - name: team-notes
      path: /data/vaults/team-notes
      auto_index: true
      git_sync: true

For each entry, boot git-inits the directory if needed (InitVaultRepo, pkg/knowledge/gitsync.go:146), seeds it from /opt/hive/seed-data/wiki (SeedVaultContent, gitsync.go:162), connects it to the knowledge API, and registers its store with the kick primer at the personal layer.

  • auto_index is parsed-but-unactioned: connecting a vault always indexes it (ConnectVault → NewFileStore, pkg/knowledge/api.go:588-606); the flag is only echoed in the vault auto-connected log line. Setting it false does not skip indexing.
  • git_sync: true adds the vault to a background syncer that runs git pull and reindexes every 60 seconds (gitSyncInterval, gitsync.go:14) — the Obsidian Git integration path.

Document sources (knowledge.documents)

knowledge.documents imports standalone documents (PDF, HTML, markdown, plain text) as knowledge facts at boot (cmd/hive/main.go:2467-2494):

knowledge:
  documents:
    - name: style-guide
      url: https://example.com/style-guide.pdf   # or file_path: /data/docs/guide.pdf
      layer: project

Each entry names either a url to fetch or a local file_path. Imports run once per boot via KnowledgeAPI.ImportDocument; the parsed facts land in the vault with a doc- slug prefix (pkg/knowledge/docsource.go). Fetches are capped at 50 MB with a 30-second timeout (docsource.go:19-24) and go through the same SSRF validation as git sources. Like git_sources, a non-empty documents list auto-enables a minimal file-engine knowledge API even when knowledge.enabled: false (main.go:2468-2475). Documents can also be imported at runtime via POST /api/knowledge/documents — see api-reference.md.

Connectors (knowledge.connectors)

knowledge.connectors is the shared seam every external knowledge system plugs into (pkg/knowledge/connector). A connector lists and fetches upstream pages; the connector syncer runs each enabled connector on its interval, converts the pages to markdown facts and writes them into the vault for the connector's layer. Incremental sync, auth, SSRF hardening, size/time caps and per-connector status are shared, so individual connectors only implement listing and fetching.

knowledge:
  connectors:
    - name: eng-handbook          # lowercase letters, digits, dashes; unique
      type: git                   # connector type (see below)
      layer: org                  # personal | project | org | community
      interval: 30m               # Go duration, minimum 1m; default 15m
      enabled: true               # default true
      scope:                      # type-specific keys
        url: https://github.com/acme/handbook.git
        branch: main
        subpath: docs
    - name: style-guide
      type: document
      layer: project
      scope:
        url: https://example.com/style-guide.pdf
    - name: next-docs
      type: document
      layer: project
      scope:
        context7_id: /vercel/next.js
      auth:
        env: CONTEXT7_API_KEY     # or file: /secrets/context7-key
Field Required Meaning
name yes Unique id; part of every fact slug.
type yes Connector type registered in pkg/knowledge/connector.
layer yes Knowledge layer the facts are written to.
interval no Sync period (Go duration, at least 1m); default 15m.
enabled no false keeps the entry configured but unscheduled.
scope per type Type-specific settings (map[string]string).
auth.env / auth.file per type Where the credential lives: an environment variable name or an absolute file path. At most one.

Credentials are never accepted inline: any other key under auth (for example auth.token) and any scope key that looks like a secret (token, password, secret, api_key, private_key, credential) fail config validation with an error naming the field.

Built-in connector types

Type Scope keys Notes
git url (required), branch (default main), subpath Runs the existing git_sources clone/pull code path (same SSRF validation, redirect suppression and sparse checkout) in a private per-connector directory, then emits every indexed markdown page. Cursor: checked-out commit SHA.
document exactly one of url, file_path, context7_id Runs the existing documents fetch/parse/chunk pipeline; one fact per extracted chunk. url is SSRF-checked before the fetch. auth optionally supplies the Context7 API key. Cursor: content hash.
confluence base_url (required), spaces and/or root_page_ids, email (Cloud), deployment, include_labels, exclude_labels, include_attachments, max_pages, full_sync_every Confluence Cloud and Data Center via the REST content search API (CQL). Storage-format bodies are converted to markdown with Confluence macros (code, panels, expand, links, images) rendered; archived/trashed pages are deprecated. Cursor: newest version.when. See knowledge-connectors.md.
notion root_page_ids, database_ids, include_archived, max_pages, full_sync_every Notion API v1 with an internal integration token; pages must be shared with the integration. Blocks are converted to markdown; database rows carry their properties as attributes; archived/trashed pages are deprecated. Cursor: newest last_edited_time. See knowledge-connectors.md.
github-wiki repos (required, comma-separated owner/repo), branch (default master) Clones https://github.com/<owner>/<repo>.wiki.git through the same git clone/SSRF path as git. Emits Home first (marked root: "true"), then pages in _Sidebar.md link order (standard Markdown links to a page name, or [[text|page]] wiki links; headings and nested items become the page path), then the remaining pages alphabetically. _Sidebar and _Footer are not emitted. updated_at comes from the last commit touching each page. A repository without a wiki (404) or one that needs credentials reports a clear "not found" status error. Only public wikis are supported for now. Cursor: per-repo commit SHAs.
google-drive folder_ids and/or shared_drive_ids (comma-separated Drive IDs, at least one), include_mime (comma-separated docs, sheets, md, txt; default docs,md,txt) Lists each folder recursively ('<folder>' in parents) and each shared drive through Drive v3 files.list with pagination. Google Docs are exported as text/markdown (falling back to text/html), Sheets as CSV rendered to a markdown table capped at 200 rows, and .md/plain-text files are downloaded as-is. PDFs are not supported yet. Trashed files are emitted as archived on incremental syncs. auth must supply an OAuth2 bearer token with the drive.readonly scope (for a service account, mint the token externally and point auth.env/auth.file at it) and the folders or drives must be shared with that identity. Cursor: newest modifiedTime seen; later syncs emit only files modified after it.
sharepoint drive_ids and/or site_ids (comma-separated, at least one), folder_path, include_files (default .md,.txt,.html,.htm) Reads document libraries and OneDrive drives through Microsoft Graph. A site contributes its default library. The first sync enumerates the drive with the delta endpoint and later syncs fetch only changes; files are downloaded via /content (HTML is converted to markdown) and items deleted upstream are marked deprecated. auth (required) supplies a Graph bearer token, for example from a client-credentials app registration with the Sites.Read.All / Files.Read.All application permission, or Sites.Selected granted per site for least privilege. docx/pdf files and modern site pages are not read yet. Cursor: JSON map of per-drive delta links.
repo-wiki repos (required, comma-separated owner/repo), branch (default main), dir (default .hive/wiki) Ingests the knowledge a repository carries in its own PR-reviewed history. See Repository-carried knowledge. Cursor: per-repo commit SHAs.
knowledge:
  connectors:
    - name: project-wikis
      type: github-wiki
      layer: project
      scope:
        repos: acme/app, acme/docs
    - name: team-drive
      type: google-drive
      layer: org
      scope:
        folder_ids: 1AbCdEfGhIjKlMnOp
        include_mime: docs, sheets
      auth:
        env: GOOGLE_DRIVE_TOKEN
    - name: team-sharepoint
      type: sharepoint
      layer: org
      scope:
        site_ids: contoso.sharepoint.com,<site-collection-id>,<site-id>
        folder_path: Handbook
        include_files: .md,.txt
      auth:
        env: GRAPH_TOKEN

Repository-carried knowledge

A repository can carry its own Hive knowledge as markdown under .hive/wiki/. Because the files live in the repo, they change only through normal reviewed pull requests, and the repo-wiki connector ingests them from the configured branch (clone/pull, SSRF validation and credentials are the same as the git connector).

knowledge:
  connectors:
    - name: repo-knowledge
      type: repo-wiki
      layer: project
      scope:
        repos: acme/app, acme/docs
        branch: main          # default
        dir: .hive/wiki       # default

Contributor layout (every *.md under the directory, nested folders allowed):

.hive/wiki/
  architecture.md
  runbooks/deploy.md
---
title: Deploy runbook   # optional; defaults to the file name
status: approved        # optional: draft | approved (default) | deprecated
tags: ops, deploy       # optional
---

Body...
  • Facts: one per file, slug repo-wiki-<connector>-<owner>-<repo>-<path>, written to the connector's layer with attr_repo, attr_path, attr_branch and attr_commit provenance. attr_state is approved unless the file says draft or deprecated.
  • Removal: a file deleted from the repo is tombstoned (status: deprecated).
  • Validation: front matter must be flat key: value lines closed by --- and status must be a known value. Invalid files are skipped, the other files still sync (deleted files are still tombstoned and the cursor still advances), and the connector's status last_error lists each offending repo/path until it is fixed.

Facts written by connectors

Each page becomes one fact with the deterministic slug <type>-<name>-<source_id> (lower-cased, reduced to [a-z0-9-], with a short hash appended when that reduction loses information). Front-matter:

---
title: Deploy runbook
type: reference
layer: org
status: active            # or deprecated
tags: [connector, git, eng-handbook]
source: git               # connector type
connector: eng-handbook   # connector name
source_id: docs-deploy
source_url: https://github.com/acme/handbook.git
synthesized: 2026-10-08T12:00:00Z   # upstream last-modified when known
synced_at: 2026-10-08T12:00:00Z
---

A fact is marked status: deprecated (a tombstone; the file stays so links resolve) when the upstream page is archived, or when a page disappears from a full listing. Incremental syncs never tombstone pages they did not list. Unchanged pages are not rewritten. Page bodies are capped at 1 MiB, each sync run at 10 minutes, and connector HTTP fetches at 30 seconds and 20 MiB with Retry-After-aware backoff on 429/5xx.

Status

The syncer records, per connector: last successful sync, last attempt, pages emitted by the last sync, active and deprecated fact counts, last error, the incremental cursor and whether the last sync was truncated at its page cap. GET /api/knowledge/connectors returns it and owners can trigger a sync with POST /api/knowledge/connectors/{name}/sync — see knowledge-connectors.md.

Relationship to git_sources and documents

knowledge.git_sources and knowledge.documents keep working exactly as before: same YAML keys, same boot-time behaviour, same /api/knowledge/sources output. The git and document connector types reuse their code paths for operators who want the connector lifecycle (interval, layer-targeted facts, tombstones, status). hive starts the connector syncer at boot: facts land in /data/knowledge/connectors/<layer> (connected as vault connectors-<layer>), and cursors/status persist in /data/knowledge/connector-state/status.json. Operators manage connectors from the Settings → Knowledge → Connectors pane (see knowledge-connectors.md).

Bead synthesizer (knowledge.bead_synthesizer)

The bead synthesizer periodically scans every agent's closed beads, classifies them into wiki fact types, and writes the results into a dedicated vault so future kicks are primed with past findings (pkg/knowledge/bead_synthesizer.go, wired at cmd/hive/main.go:2508-2581).

It is on by default. enabled is a tri-state pointer that defaults to true when absent (IsEnabled, pkg/config/knowledge_config.go:51-56), and main.go auto-enables a file-based knowledge API for it even when knowledge.enabled: false (main.go:2498-2505). A hive that never mentions knowledge: in its config still runs the hourly synthesis loop and the retention manager below. bead_synthesizer.enabled: false is the only opt-out.

Synthesized facts are only primed when knowledge.enabled: true. knowledge.enabled gates exactly one thing: the kick primer. With it false the synthesizer still writes facts into its vault and they stay browsable in the dashboard and API, but no primer exists, so no kick receives them. The dashboard's knowledge panel reports this state as not primed (enabled on GET /api/knowledge/stats and GET /api/knowledge/health is primer registration, with the registered sources under primer), not as enabled.

Enabling knowledge from the dashboard (PUT /api/knowledge/enabled, handleKnowledgeToggle) builds the primer live and registers every store already connected — the bead-synth vault, ready git sources, and any configured vault connected at boot (registerKnowledgeStores, cmd/hive/boot_knowledge_primer.go) — so kicks are primed immediately. When the hive booted with knowledge.enabled: false, configured vaults: were never connected and the dashboard's knowledge API is the file-only fallback; the response's primer.restart_required and primer.restart_reason (also shown as a panel banner) name what still needs a restart: those vaults, browsing configured wiki layers: (kicks query them already), a non-file engine, and knowledge.curator settings.

knowledge:
  bead_synthesizer:
    # enabled: false                # opt out (defaults to true)
    schedule: hourly                # hourly (default) or daily
    min_confidence: 0.0             # drop facts classified below this
    target_layer: personal          # primer layer for the synth vault
    max_facts_per_cycle: 0          # 0 = unlimited
    max_api_calls_per_cycle: 40     # cap on charged PR GETs per cycle
    vault_path: /data/vaults/bead-synth-wiki
    retention_policy:
      max_beads: 5000
      archive_after_synth_days: 7
      high_priority_retain_days: 30
      preserve_with_deps: true
Field Current behavior
schedule hourly (also the default and the fallback for unrecognized values) or daily (ParseSynthSchedule, bead_synthesizer.go:929-937).
min_confidence Facts whose heuristic classification confidence (ClassifyBead, bead_synthesizer.go:254, emits 0.5–0.8) falls below this are skipped. Default 0.0 — everything classifiable is ingested.
target_layer Layer the synth vault is registered at with the primer; empty defaults to personal (main.go:2525-2528).
max_facts_per_cycle Cap per synthesis cycle after dedup; 0 means unlimited (bead_synthesizer.go:223-226).
max_api_calls_per_cycle Cap on charged GET /pulls/{n} calls per synthesis cycle; default 40. PR GETs are conditional (If-None-Match, ETag and body kept across cycles), so a 304 reuses the cached PR. The HIVE_BEAD_SYNTH_MAX_API_CALLS env var overrides this (0 = unlimited).
vault_path Defaults to /data/vaults/bead-synth-wiki; the directory is created and auto-connected as vault bead-synth-wiki (main.go:2510-2522). Only the default path is included in owner-triggered spoke backups; facts under a custom vault_path are not backed up.

Repository publish (knowledge.bead_synthesizer.publish)

Optionally the synthesizer also publishes its facts into a repository as markdown files at <path>/<slug>.md. This is off by default and separate from the connector mirror in knowledge.publish. Publishing is always one pull request on the fixed branch hive/bead-synth-knowledge, updated in place while it is open.

knowledge:
  bead_synthesizer:
    publish:
      enabled: true
      repo: ""                 # empty = project.primary_repo
      path: docs/knowledge/
      base_branch: ""          # empty = the repo's default branch
      mode: pr
      cadence: per_run
Field Current behavior
enabled Default false. The block alone does nothing until this is true.
repo Target repository (owner/name, or a name under project.org). Empty selects project.primary_repo, then the first project.repos entry.
path Directory in the repo that receives the facts. Default docs/knowledge/; use .hive/knowledge/ to keep them out of the published docs tree. Must be relative with no .. components.
base_branch Branch the PR targets. Empty selects the repo's default branch. Must not be hive/bead-synth-knowledge.
mode pr (default) or commit. PR-vs-commit policy: hives at ACMM level 4 or below publish via PR only. Direct commit is allowed only where policy permits it, and the current publisher does not honor commit: it logs a warning and still opens a PR.
cadence per_run (default, publish after each synthesis cycle that produced facts) or daily.

The local vault write is unaffected by publish failures, which are logged. Whether or not publishing is enabled, the default bead-synth-wiki vault is included in spoke backups.

retention_policy drives a bead lifecycle manager that archives old beads — it is always constructed, whether or not the block is present (main.go:2541-2553). Zero-valued fields fill in as max_beads: 5000, archive_after_synth_days: 7, high_priority_retain_days: 30 (pkg/knowledge/bead_lifecycle.go:29-31); archiving stops once counts drop to 80% of max_beads. One asymmetry to watch: preserve_with_deps (never archive a bead with open dependents) defaults to true when the whole retention_policy block is absent, but to false when the block is present without the key — set it explicitly whenever you write the block.

Runtime status and toggle: GET /api/knowledge/bead-synthesizer and PUT /api/knowledge/bead-synthesizer/enabled (owner-only) — see api-reference.md.

Kick primer size cap (knowledge.primer.max_bytes)

knowledge.primer.max_facts (default 25) bounds how many facts a kick may receive; knowledge.primer.max_bytes (default 4096, minimum 512) bounds how many bytes the rendered # Relevant Knowledge block may take (pkg/knowledge/primer_render.go). Facts are visited in the primer's rank order (type priority, then confidence) and each is kept only if its whole entry — title, confidence, source link and body — fits the remaining budget. A fact that does not fit is dropped whole, never clipped mid-body, and later, smaller facts are still considered, so one long fact cannot suppress everything ranked after it. Context7 "Available Documentation" hints follow the same rule.

When anything was dropped the block ends with a visible one-line marker, _… truncated: N facts / M bytes omitted (size cap)._ (the same … truncated: prefix the code map uses), and the scheduler logs knowledge primer: dropped facts over the byte cap with the counts and the cap, so a cap that is too tight for the vault is visible in the kick and in the log rather than silently thinning what agents are told.

knowledge:
  primer:
    max_facts: 25
    max_bytes: 4096        # 0 = default; values under 512 are raised to 512

Context7 content cap

The Context7 llms.txt body a document import or the context7 connector fetches is bounded the same way before it is stored (CapContext7Content in pkg/knowledge/context7.go). The body is split into Context7's snippet blocks (or paragraphs for plain prose), snippets are ranked by how many query/topic terms they mention, and each is kept only if it is within the per-snippet cap and fits the remaining total; dropped snippets are dropped whole and the content ends with … truncated: N snippets / M bytes omitted (size cap). Kept snippets stay in source order. The fetch logs context7: dropped snippets over the content cap naming the library.

Env var Default Meaning
HIVE_CONTEXT7_MAX_SNIPPETS 40 Snippets kept per fetch.
HIVE_CONTEXT7_MAX_SNIPPET_BYTES 4096 A single snippet larger than this is dropped.
HIVE_CONTEXT7_MAX_TOTAL_BYTES 16384 Total content kept, marker included; minimum 1024.

Unparseable or non-positive overrides fall back to the default rather than to no cap.

Repository code maps (knowledge.code_maps)

A code map is a generated knowledge page — never hand-authored — that summarizes a checked-out repository so agents get a bounded orientation primer for it. The generator lives in pkg/knowledge/codemap.go and the refresh loop in pkg/knowledge/codemap_schedule.go.

knowledge:
  code_maps:
    enabled: true                 # opt-in; absent or false does nothing
    schedule: hourly              # how often HEAD is checked (hourly or daily)
    vault_path: /data/vaults/code-maps
    layer: project
    max_bytes: 16384
    repos:
      - name: hivecommons/hive    # used as the repo:<name> scope tag
        path: /workspace/hive     # an existing checkout...
      - url: https://github.com/org/other   # ...or an https URL to clone
        branch: main

Each map has these sections, every one sorted deterministically:

Section Contents
Packages Directories with source files; Go directories show the package name and file/test counts, other languages a per-language file count.
Entry points Go main packages, cmd/ directories, scripts (*.sh, scripts/, bin/) and build/entry files (Makefile, Dockerfile, go.mod, package.json, …).
Ownership hotspots CODEOWNERS rules (repo root, .github/ or docs/) and, for git checkouts, directories ranked by file changes in the last 200 commits.
Public APIs Exported top-level Go identifiers per non-main package (first 15, then +N more). Other ecosystems get only the generic directory summary.
Test layout Test-file counts per directory and test directories (test/, tests/, e2e/, testdata/, …).
Extension points Exported Go interfaces, Register* functions, and plugin-style directories (plugins/, hooks/, connectors/, providers/, …).

Dependency and build directories (vendor/, node_modules/, dist/, target/, dot-directories other than .github/) are skipped.

Size caps. Each section keeps at most 40 entries and 4 KiB; the whole body is capped at max_bytes (default 16 KiB, minimum 1 KiB); individual entries are clipped to 240 bytes; the walk stops after 20,000 files and the map says it is partial. Dropped entries or sections are replaced by a … truncated: marker line, always at the same point for the same input.

The same whole-entry-or-nothing rule bounds what the kick primer renders — see Kick primer size cap.

Storage and scoping. Maps are written to <vault_path>/codemaps/codemap-<repo>.md as type: reference, state: approved facts tagged codemap, generated and repo:<name>, so the existing repo scoping (for example the knowledge table of contents repos filter) includes a map only for its own repository. The vault is connected as code-maps and primed like any other vault.

Freshness metadata. Frontmatter records generator: hive-codemap, generator_version, source_sha (the checkout's HEAD), content_hash, generated_at and synced (which feeds the existing freshness signal).

Regeneration. Every tick the refresher reads HEAD for each repo (URL repos are pulled first). A map is regenerated when it is missing, the generator version changed, HEAD moved, or the revision is unknown (not a git checkout). The file is rewritten only when something changed: a new HEAD whose summary is identical updates source_sha but keeps generated_at, and an identical regeneration leaves the file untouched. Edits made by hand are overwritten on the next change, so do not edit these pages.

Per-agent scopes (knowledge.agent_scopes)

Maps an agent name to the layers, repos, types, tags and include_states that agent's kick primer and agent-identified knowledge reads are restricted to. A missing agent or an empty field is unrestricted (the default). Listing draft in include_states additionally lets the agent's kicks carry the drafts it authored (author: frontmatter), marked (draft). Unknown layer or lifecycle state names fail config load. See Per-agent scopes for the intersection rules, and agent-knowledge-and-memory.md for how the scope is applied on the kick path end to end.

knowledge:
  agent_scopes:
    scanner:
      layers: [project, org]
      repos: [hivecommons/hive]

Mid-task knowledge advisor (knowledge.advisor)

The kick primer above runs once, when the kick is typed. The advisor (#17052) keeps watching after that: every poll tick it reads the scopes the agent's pane shows it working in — a directory prefix such as src/crux/agent from a Read(...)/Edit ... line, or a gh pr / gh issue / gh run command family — and the first time a scope appears that was not on screen when the kick landed, it asks the same Primer (under the same knowledge.agent_scopes restriction) for entries about it. Each hit is typed into the live session as one line, like a CI-poll nudge:

📚 Knowledge: Re-pin TRUSTED_SCRIPTS_TREE — Any src/scripts edit must … (full: https://…)

Rules, all enforced by the runtime (crux/agent/knowledge_advisor.go):

  • Off by default. knowledge.advisor: true turns it on; knowledge.enabled must also be on. Flip it once your own hive has run it for a while.
  • At most knowledge.advisor_max_per_kick lines per kick (default 3), with a two-minute cooldown between lines.
  • Never an entry the kick already carried. The primer records which entries survived the byte cap; those are excluded, and a slug is typed at most once per kick.
  • Never into a tool call, never after the turn. A line waits while a shell command or tool is running; if the agent reaches an empty prompt first, the pending advice is dropped (the next kick primes afresh). It is also held while a human is attached to the pane and while a kick is being typed.
  • Never stalls the poll tick. The adviser is called on its own goroutine, at most one call in flight per agent (#17153). Scopes seen while a call runs queue newest first, capped at the lines left in the kick's allowance. The tick waits briefly for a fast matcher so its line still lands on the tick that saw the scope. A slower result is held until a later tick passes the gates above. A result that comes back after the agent's kick changed is dropped.
  • Each scope is tagged with an action and asked about at most twice per kick. The pane line a scope comes from gives its Action: ⏺ Read(…) is read, ● Edit … / Write / Update is edit, and a $ … shell line (such as $ gh pr …) is run. The adviser is asked on the scope's first sighting, and once more on the first edit in a scope first seen without one, or on the next command in a gh family. A third sighting is ignored, and elapsed time never makes a scope eligible again.
  • No pane text unless the adviser asks. An adviser whose policy sets WantsPaneTail receives the last PaneLines (default 80) scrollback lines, passed through logscrub.RedactPaneTokens first. Phase 1's matcher never asks, so no pane text leaves the runtime.
  • Per-agent opt-out: knowledge.advisor_exclude_agents: [scanner] matches an agent name or its replica family base name.
  • Measurable. Each delivery is recorded in the dashboard prompt history with trigger knowledge-advisor and logged as knowledge advisor: delivered with the agent and slug.
knowledge:
  enabled: true
  advisor: true
  advisor_max_per_kick: 3
  advisor_exclude_agents: [scanner]

Model-ranked advisor (knowledge.advisor_llm)

knowledge.advisor_llm configures the model-ranked variant (#17054): a small model picks one of the matcher's candidates or nothing (#17155). It is ignored unless knowledge.advisor is on. The pipeline, modes, budgets, decision log line and rollout steps are in agent-knowledge-and-memory.md.

knowledge:
  advisor: true
  advisor_llm:
    mode: off                    # off | shadow | rank
    model: "<your-small-model>"  # required for shadow/rank
    endpoint: ""                 # default: governor.litellm endpoint
    api_key_env: ""              # or api_key_file; default: governor.litellm key
    candidates: 5
    pane_lines: 80
    timeout_s: 15
    max_calls_per_agent_hour: 6  # hard ceiling without budget_override
    max_calls_per_day: 300       # hard ceiling without budget_override
    budget_override: false
    holdout_percent: 0           # 0..100
    on_error: silence            # silence | matcher
  • model, endpoint and the key resolve like the governor reviewer: explicit value, then governor.litellm, then unset. With no endpoint or key from either place the block is unavailable.
  • shadow or rank with no model resolves to off and logs a warning; it never stops the hive from loading.
  • max_calls_per_agent_hour above 6 or max_calls_per_day above 300 is a load error naming the key unless budget_override: true.
  • holdout_percent outside 0..100, an unknown mode or on_error, and a negative candidates, pane_lines or timeout_s are load errors.

Without knowledge.advisor_llm (or with mode: off) the advisor is the pure phase-1 matcher with no model in the loop. The advisor is wired only when the agent runtime runs in-process; the crux serve sidecar has no adviser in this phase.

Open questions

  • #4944 (the issue behind this section) suggested documenting auth for private repos as if it exists. It does not: see Auth for private repos above. If private-repo support is added later, this section should be updated with the actual credential-config shape rather than assumed ahead of the code.