Skip to content
jmarcenoPublic

About

Simple local agent memory solution using DuckDB + BM25 + Vector Search + Graphs. Daemon gets session logs from time to time, index it and agents get a skill to query, simple and effective.

Topics

Resources

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

duck-mem

duck-mem

Give your coding agents a memory they can actually search.
Every Codex, Claude, Cursor, Muse, OpenCode, DeepSeek Harness, ZCode and Antigravity session lands in one DuckDB file — conversations only, no tool noise — and becomes a search index your agent can query from the terminal.

Open a pull request for your favorite agent if it is not here.

License: MIT Go 1.24 Platform: Linux Database: DuckDB

What it does · Keeping sessions private · Install · Usage · How search works · Tray · Agent skill · Contributing


What it does

Your agents forget everything between sessions, and the raw logs they forget with are full of tool calls, environment dumps and half-finished reasoning. duck-mem turns those logs into something searchable:

  • One store, eight agents. Codex, Claude, Cursor, Muse, OpenCode, DeepSeek Harness, ZCode and Antigravity sessions are parsed into a single DuckDB file. Nothing is copied out of the original agent directories, and the stores are read-only.
  • Conversations, not transcripts. Tool calls, tool output, reasoning blocks and metadata are dropped. You keep the prompts, the answers and the text that actually explained the work.
  • Search that returns the conversation. query prints whole sessions — the prompt that opened them, the matching messages, the project — not a wall of message IDs for the agent to go and read.
  • A topic graph, not just keywords. A duckpgq property graph over the vocabulary finds what a project talks about together, so a query phrased differently still reaches the right session.
  • It keeps itself current. A systemd user service re-indexes every few minutes and skips unchanged files; a tray icon shows status and runs a full re-index with one click.
  • Privacy at the prompt. Start a session with private, noingest or noduck and duck-mem keeps that session out of memory entirely.
  • No LLM anywhere. Ingestion, indexing and search are deterministic. No message text ever leaves your machine, and no API key is needed.

Keeping sessions private

Open a session by starting the first prompt with one of these tags:

tag example first prompt
private private the payroll merge details
noingest noingest: demo of the client login flow
noduck noduck read my passwords

On the next ingest cycle a tagged session is never stored: if an older parser already put it in memory, duck-mem removes its messages, its topic-graph contribution and its copy-detection fingerprint. The session is then permanently marked in the no_ingest table — the mark outlives the source log, so the session stays out of memory even if you edit the tag away afterwards.

The match is on the first word of the first user message, case-insensitively and ignoring surrounding punctuation, so Private: blocks and privateer the ship does not. Marked files are checkpointed and never re-parsed.

Install

Requires Linux with Go 1.24+ and systemd --user (any GNOME, KDE, COSMIC or Hyprland session). Extensions are installed on first run.

git clone https://github.com/jmarceno/duck-mem.git
cd duck-mem
go build -o duck-mem ./cmd/duck-mem
./duck-mem --install

--install copies the binary to ~/.local/bin, adds that directory to your PATH if needed, and enables and starts duck-mem.service (the indexer) and duck-mem-tray.service (the tray icon). Open a new login shell to use duck-mem by name.

To remove it again:

duck-mem --uninstall                 # asks whether to keep the database
duck-mem --uninstall --keep-data     # noninteractive, keeps the database
duck-mem --uninstall --purge-data    # also deletes duck-mem data and config

Uninstalling never touches your Codex, Claude, Cursor, Muse, OpenCode, DeepSeek Harness, ZCode or Antigravity session stores.

Usage

duck-mem ingest [--db PATH] [ROOT...]   index new messages from session logs
duck-mem query [--db PATH] [--project P] [--source S] [--limit N] [--hits N] <text...>
                                        best-matching sessions, with their hits
duck-mem show [--db PATH] [--context N] [--from A] [--to B] [--full] [--system] <session>[#seq]
                                        read the messages around a hit
duck-mem related [--db PATH] [--project P] [--depth 1|2] [--limit N] <term>
                                        what a project discusses together with <term>
duck-mem index [--db PATH] [--min-df N] [--full]
                                        incremental (or full) topic-graph index
duck-mem daemon [--db PATH] [--interval 5m] [ROOT...]
                                        the ingest loop the service runs
duck-mem repack [--db PATH] [--out PATH]
                                        rebuild the file and reclaim unused space;
                                        without --out it swaps in place
duck-mem relocate [--db PATH] <new-path>
                                        copy the database to a new location, verify
                                        the copy with a smoke run, then optionally move
                                        the old file to the trash
duck-mem projects [--db PATH]
                                        list stored projects, session counts, aliases
                                        and captured git identity

Flags may come before or after the text. The database defaults to ~/.local/share/duck-mem/memory.duckdb; because it is a plain DuckDB file you can also open it with the DuckDB CLI or any DuckDB client.

Moving the database

duck-mem relocate <new-path> chooses where the database lives — every command and the installed services follow the recorded location, no --db needed. On a machine with no database yet it only records the location; the file is created there on the first ingest. When a database already exists it is copied to the new path, the copy is verified table by table and opened for a smoke run, and only then are you asked whether the old file should go to the trash — never a permanent deletion, and n (the default) keeps it where it was. If the copy or the smoke run fails, the old database stays exactly where it was. The indexer is stopped for the move and restarted on the new location.

Search

$ duck-mem query "flaky test on darwin arm64" --project duck-mem --limit 2
2 sessions for "flaky test on darwin arm64" in projects matching "duck-mem", best first. Messages are numbered #seq.

[1] 0199f1c4-2b7a-7c31-9f0e-5d2a8b41c7de  claude · 2026-08-14 → 2026-08-14 · 412 msgs · /home/you/Projects/duck-mem
    opened with: "the arm64 build times out on the test runner, can you look at it?"
    #12 user 14:02:11: the arm64 runner times out on ingest_test.go:184, flaky?
    #41 assistant 14:19:03: TestRetryExponentialBackoff failed 3 of 5 runs on darwin-arm64
    #58 assistant 14:41:57: the timeout is the -race flag, not the backoff; fixed in helper_test.go:22
    ended with:
      #410 user 15:02:40: commit it
      #411 assistant 15:03:12: committed as 4f2a9c1; CI is green on darwin-arm64

[2] 0199f2d8-77aa-70c2-9d13-6b2f0aa9e410  codex · 2026-08-15 · 96 msgs · /home/you/Projects/duck-mem
    (no query words here; found through the topic-graph neighbours below)
    opened with: "re-run the CI matrix and report what broke"
    #7 assistant 09:11:20: darwin-arm64 job failed again: flaky, retrying

Topic graph: often discussed with these words: race (4), timeout (3), retry (2), matrix (1)
  Add one to the query to widen it, or map the area: duck-mem related --project duck-mem runner
Read a hit with the messages around it: duck-mem show 0199f1c4-2b7a-7c31-9f0e-5d2a8b41c7de#41

Each result is a session: what it was, when, which project, how it started, the messages that matched (placed where the query terms are closest together) and how it ended: the last prompt, the last reply and, when the last turn got no reply, a note with the reason (usage limit, interrupted) and the log path. Sessions that match only through the topic graph are marked as such and appear after the direct hits, so the graph widens a search without reordering it; the topic-graph line is printed only when it explains such a session or the query left slots empty. A session ID (or unique prefix) in the query puts that session first, which is how an agent picks up another agent's thread.

show is the second step — it prints the messages around a hit so you can read the reasoning in context. Host context (system prompts, AGENTS.md dumps, IDE state) is collapsed to one line; --system prints it:

$ duck-mem show 0199f1c4#41 --context 2
session 0199f1c4-2b7a-7c31-9f0e-5d2a8b41c7de · claude · /home/you/Projects/duck-mem
2026-08-14 · 412 messages (#0–#411) · log /home/you/.claude/projects/<project>/<session>.jsonl
showing #39–#43; earlier: --from 0 --to 38; later: --from 44 --to 64; whole messages: --full

Unique ID prefixes are enough, and --project/--source narrow either command.

Projects and identity

Every session carries a project value: the directory (or short name) the harness reported. Values are normalized at ingest — trailing separators, duplicate separators, ~ and backslash paths are canonicalized — so one checkout cannot become two projects. --project accepts more than the exact stored string. Resolution tries, in order: the exact stored value or path, a name the project was renamed from, the directory basename (an agent's usual repo-directory name), and a shared git identity — worktrees and clones of one remote are the same project. Only then does the old substring filter apply.

--project is advisory: resolving a name narrows a read; it never merges or rewrites data. Only duck-mem rename changes stored values, and it records an alias so later queries under the old name keep resolving.

When a project cannot be resolved, the output says so instead of searching everything and offers up to three similar stored names with their session count, sources and date range:

$ duck-mem query --project celestial flaky test
no project matches "celestial"
similar names: /home/you/Projects/celeste (36 sessions, claude+codex+cursor, 2026-10-06 → 2026-10-19)
Similar name only, not verified — may be a different project, or an older name for the same one.

A filter that names more than one project says so and lists each with its session count, so Generic and generic are never silently blended:

$ duck-mem query --project generic ledger
--project "generic" matches 2 projects: /home/you/Projects/generic (79 sessions), /home/you/Projects/Generic (60 sessions)

Git identity is captured lazily while changed sessions are re-parsed; there is no backfill migration in the code. A previous data repair was done by hand once, before this existed, and the alias map keeps it from reverting. duck-mem projects shows everything the store knows:

$ duck-mem projects
60 stored projects, 4 with a captured git identity

/home/you/Projects/organizer               925 sessions  claude,codex,cursor  …
                                           git github.com/you/organizer · /home/you/Projects/organizer
                                           aliases: /home/you/projects/organizer-tiptap, …
/home/you/Projects/celeste                  36 sessions  claude,codex,cursor,dsh,opencode  …

Topic relationships

$ duck-mem related runner --depth 2
matrix (14, via timeout)
race (11, via flaky)
flaky (9)
backoff (replaces: retry -> delay, 3)

related answers "what is discussed together with X" by traversing the topic graph. Nodes are Porter stems, so enemy and enemies are the same node and your query words find their node directly. Co-mention edges are undirected and weighted by how many messages mention both terms; typed relations recovered from possessive and replacement phrasing (bastion --owns--> turret) keep their direction and quote the sentence they came from. --depth 2 follows neighbours-of-neighbours, expanding only the strongest direct links.

How search works

Two rankings are fused with reciprocal rank fusion, and the topic graph takes part in every query.

  • BM25 over a posting list of Porter-stemmed terms, which gives recall: word forms match, and a long message that mentions the terms still ranks. Postings are written in the same transaction as their messages.
  • Vector similarity over hashed term and phrase vectors (384 dimensions, exact cosine scan in DuckDB). It rewards messages that reuse the query's adjacent word pairs, and it only reorders the BM25 candidates, so a hash collision between two unrelated short messages cannot promote one of them.
  • The topic graph contributes one hop of co-mention neighbours to the query, at a fraction of the weight of a direct term. If your words match fewer sessions than you asked for, those neighbours are searched again to fill the empty slots, and the results say so.

Ingestion and indexing are incremental by default: per-file checkpoints skip logs that have not changed, and appended messages are read from the byte offset where the previous version left off. Rewritten or truncated sessions are reconciled in place instead of rebuilt. Because incremental threshold decisions use counts as of each run, a periodic full rebuild heals the small drift:

0 4 * * 0 duck-mem index --full   # weekly rebuild; daily ingest stays incremental
0 5 1 * * duck-mem repack         # monthly repack reclaims space from rewrites

DuckDB allows one read-write process at a time. The daemon is that process: it opens the database at startup and keeps it. query, show, related, ingest, and index send their work to the daemon and wait in one queue, so a sync and a search no longer fail each other with a file-lock error. Time in the queue is not limited. Once a query, show, or related starts, it has 30 seconds to finish; ingest, index, and checkpoint are not capped. With the daemon stopped, those commands open the database themselves. repack also works while the daemon runs: it closes the database for the duration, rebuilds the file, and reopens.

A failed transaction (Failed to commit, Conflict on update) leaves DuckDB's handle unable to commit again. The owner reopens the file after such an error, and pauses writes when it happens twice without a clean write in between, so a damaged file stops cleanly with the real error instead of failing every cycle with a misleading conflict. A successful repack, or a restart after repair, resumes the writes.

Vector search deliberately uses no approximate index. The vss HNSW index that earlier versions persisted kept every checkpoint's index blocks, so the file grew by roughly 100 MB per changed cycle with almost no new data. An exact scan of ~60k embeddings costs ~40 ms warm and finds every true neighbor (the HNSW index returned about 60–75% of them). Opening an older database drops that index; its blocks become reusable but the file only shrinks after a repack.

duck-mem repack rebuilds the database file to reclaim space left by rewrites and dropped data, and drops a legacy HNSW index if one is still present. It verifies every table's row count and complete-row hash, preserves sequence positions, and rebuilds indexes before the new file becomes visible. With no flags it replaces the live file in place; the source stays untouched when the repack fails. --out NEW_FILE writes a separate copy instead and leaves the live file alone; the destination must not exist. Repack does not need the services stopped, and the tray menu offers the same in-place repack.

What gets indexed

Source Read from Project from
Codex ~/.codex/sessions/**/*.jsonl, ~/.codex/archived_sessions/*.jsonl session metadata cwd
Claude ~/.claude/projects/*/*.jsonl the record's cwd
Cursor ~/.cursor/projects/*/agent-transcripts/*/*.jsonl, ~/.cursor/plans/*.plan.md workspace storage workspace.json
Muse ~/.local/share/muse/sessions/**/session.jsonl route facts / workspace root
OpenCode ~/.local/share/opencode/opencode.db (read-only, including WAL) session_v2.directory, else the worktree
ZCode ~/.zcode/cli/db/db.sqlite (read-only, including WAL) session.directory
DeepSeek Harness ~/.dsh/sessions/*/*/session.v4.jsonl.zstd (zstd, read-only) session header cwd
Antigravity ~/.gemini/antigravity/brain/*/.system_generated/logs/transcript_full.jsonl conversation_summaries.db workspace_uris

Project values are normalized when they are parsed (filepath.Clean-style: trailing and repeated separators, ~, backslash paths), compared case-insensitively, and stored with the source's casing.

Kept: user prompts, assistant text, system and developer text. Dropped everywhere: tool calls, tool outputs, reasoning and thinking blocks, environment snapshots, queue and metadata records. A Cursor plan file is imported whole as a single note, and OpenCode child sessions are imported alongside their parents. ZCode subagent sessions are imported the same way: their reports reach the main conversation only as tool output, which is dropped, so the subagent rows are the only place that content is kept. Cursor transcripts whose project cannot be resolved stay under cursor:<slug> rather than being given an invented path.

DeepSeek Harness session files are zstd-compressed event logs, decompressed as they are read. The one source of system text it does not contribute is its own system prompt: the harness writes that, plus the workspace instructions, runtime state and skill catalog it injects, as user-role records carrying a source kind other than user, and only records the user actually sent are indexed. An unknown session format version is reported as an unexpected format instead of being guessed at.

Antigravity conversations are append-only JSONL step logs. Each prompt is stored as the text inside the <USER_REQUEST> wrapper, without the metadata blocks the app appends for the model (local time, model-selection notices), and only user input the user actually sent and the model's visible replies are kept: tool output records and reasoning are dropped. transcript.jsonl beside each log is the same conversation with long content clipped, and chunks/ holds partitions of both, so only the untruncated log is read and one conversation can never be imported twice. The model id carries its reasoning level (gemini-3.8-flash-high), which is split into the model and effort columns; suffixes that are part of the model name (-tiered, -n, -lite) stay in the model and leave the effort Unknown. A conversation outside any project keeps an empty project rather than an invented path.

Three Antigravity behaviours are documented as temporary limitations, kept until there is data to design against:

  • Failed and aborted turns leave no note. Every step observed reported DONE, so the shape a failure or a user cancellation writes is unknown. A turn that ended without a reply looks like it stopped, unlike Codex and Cursor, which record a "turn failed" or "turn aborted" note.
  • Subagent conversations are not linked to their parent. The summaries index carries parent_conversation_id and nesting_depth, but no subagent brain directory has been observed, so it is unknown whether a subagent gets one of its own. A subagent that does is ingested as an ordinary unlinked session.
  • A conversation stored only as chunks is skipped. The full log and its chunk partitions have existed side by side in every sample. If a long conversation ever ships chunks without the full log, nothing is ingested rather than half-importing it or duplicating the conversation.

Every session also records the model it ran on, the reasoning effort it ran at, and its source dates: created and last-updated, both taken from the harness's own data (message timestamps, model-selection records, SQLite update columns), never from ingest time. A model or effort switch mid-session keeps the final value. Where a harness reports neither field the session stores Unknown instead of an empty value; Cursor transcripts and plan files carry neither dates nor a model, so they take the OS file birth and modification times as dates and stay Unknown otherwise. A re-ingest that re-reads identical content refreshes the metadata but never moves the dates, and neither does a project rename: only genuinely new data does.

Imported cross-harness copies are suppressed when their entire conversation exactly matches the beginning of another retained session in the same project. Matching removes known timestamp/query wrappers, host context, and import markers; it preserves wording, internal whitespace, order, and failure notes. It requires at least three messages, 500 bytes, and a substantive assistant reply. Uncertain matches and shorter conversations remain imported.

Suppression removes the copy's messages, embeddings, search postings, and graph contributions. Its identity, source path, fingerprints, and file checkpoint remain. Changed copies are fully reparsed: distinct content restores the full conversation as a separate session, including its shared history. Rewriting an owner first restores any copies it would no longer cover; unavailable copy logs prevent that rewrite rather than losing history. show accepts suppressed IDs and resolves them to the retained session. Existing imports are checked automatically on the first sync after upgrading; no parser-version bump or full re-index is needed.

Schema

sessions(session_id TEXT PRIMARY KEY, source, project, started_at, path,
         model, effort, updated_at)
messages(session_id, seq, source, project, role, text, created_at,
         indexed, embedding FLOAT[384], PRIMARY KEY(session_id, seq))
msg_terms(session_id, seq, term, tf)     -- BM25 postings, Porter-stemmed
term_df(project, term, msgs)             -- per-project document frequencies
topics(topic_id, project, term, label)   -- graph nodes
topic_edges(src_id, dst_id, term_a, term_b, weight, kind,
            from_term, to_term, evidence)
msg_edges(session_id, seq, term_a, term_b, ...) -- per-message edge log
file_checkpoints(path, size, mtime_ns, tail_hash, ends_line, session_id, …)
session_copies(session_id, hashes, eligible, imported, duplicate_of, seen_at)
project_aliases(old_name TEXT PRIMARY KEY, new_name)   -- rename map
project_identities(project, repo_root, remote_url, common_dir, updated_at)

project is a value, not a column: DuckDB columns are schema, so one column per project would need DDL for every new project and break every query. It is an indexed TEXT on both tables; reads resolve a name to exact stored values (see Projects and identity) and filter with WHERE project IN (...), with the old substring match only as a fallback.

Tray

duck-mem-tray.service registers a StatusNotifierItem (and DBusMenu) item using the project logo. The menu shows the indexer's state, the last completed sync, and how many sessions are registered, and can start or stop the indexer, sync immediately instead of waiting for the next cycle, force a full re-index, rename a project, or repack the database to reclaim space. A dot appears over the icon while an action is running or when the indexer is down.

Sync history is retained as monthly JSONL files in ~/.local/state/duck-mem/logs/. Each cycle records the harness, time, session and extracted message counts (including zero when unchanged), and success or failure. Failures include Unexpected format, Database error, or Other and available error details; cycle-wide errors use harness all. No session text is stored in these logs. Logs are retained without automatic deletion.

Open database folder opens ~/.local/share/duck-mem/ the same way. View logs opens this directory in the default file manager and clears the indicator for existing failures. Failures include the affected file path and show a red dot drawn directly into the tray icon. The dot also clears when a later cycle finishes without an error and without skipping a file, because a skipped file is not checkpointed and is still missing. A later failure sets the dot again. Busy and stopped states use amber and grey dots respectively. The tray refreshes every ten seconds.

The tray is optional — duck-mem ingest, query and the rest work without it.

Agent skill

A read-only agent skill is included at .agents/skills/duck-mem-query. Point an agent at it and it will search your history before guessing, so it can answer "we did this last month, here's the session" instead of reinventing it.

How it differs

  • From grep over the logs: search returns the session that matters, with the surrounding conversation, instead of matching lines you have to reassemble.
  • From vector-only tools: BM25 keeps recall for names, error text and word forms; the graph supplies the vocabulary a plain embedding misses.
  • From hosted memory services: everything stays in one local file. No account, no API key, no text leaving the machine.

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for the build, test and commit conventions, and please include a fixture with any parser change — the tests are the fastest way to agree on what a session log should produce.

License

MIT © 2026 Jardel G Marceno

About

Simple local agent memory solution using DuckDB + BM25 + Vector Search + Graphs. Daemon gets session logs from time to time, index it and agents get a skill to query, simple and effective.

Topics

Resources

Contributing

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Contributors

Languages