watch._rebuild_code stamps the whole corpus into the manifest on code-only rebuilds, making --update silently skip new docs

[ConnorCloze]

Summary

On a code-only auto-rebuild (the AST-only path in graphify/watch.py::_rebuild_code), graphify calls save_manifest(detected["files"]) with a full-corpus detect result. Because save_manifest rewrites the whole manifest from whatever file dict it's handed, this stamps every doc/paper/image in the corpus as "seen" — including files that were never semantically extracted. The next graphify --update then diffs against this poisoned manifest, sees those docs as unchanged, and skips them forever. The same method also unconditionally deletes the needs_update flag, cancelling the pending-docs nudge that a prior doc change had raised.

Net effect: in a mixed code+docs corpus under --watch (or the post-commit hook path), any doc added/changed between full --update runs can become permanently invisible to the graph while appearing "processed."

Impact (real-world)

In a ~4,800-file personal knowledge vault under --watch, 819 files had zero extracted nodes while the manifest recorded them as processed — an entire month of notes, plus specific decision docs, invisible to every graph query. detect_incremental reported "nothing to update." The graph looked healthy (node count high, recent-dated sources present) because unrelated files were flowing in, which masked the hole.

Root cause

graphify/watch.py, _rebuild_code(...):

try:
    from graphify.detect import save_manifest
    save_manifest(detected["files"])   # <-- detected = full detect(), includes docs/papers/images
except Exception:
    pass

_rebuild_code only extracts code files (AST). But detected["files"] is the full corpus. save_manifest (graphify/detect.py) stamps mtime+hash for every path it's given, with no check that the file was actually extracted this run. So docs get a "processed" stamp they never earned.

Compounding it, later in the same method:

flag = out / "needs_update"
if flag.exists():
    flag.unlink()   # <-- clears the doc-change nudge, but this path never handled docs

The needs_update flag is only ever written for doc/paper/image changes (which require the LLM path). A code-only rebuild clearing it drops the one signal telling the user to run --update for pending docs.

Suggested fix

Stamp only the files actually re-extracted in this run, merged over the existing manifest, and leave needs_update for the supervised --update flow to clear:

try:
    from graphify.detect import load_manifest, _md5_file, _MANIFEST_PATH
    manifest = load_manifest()
    for f in code_files:                 # only what AST actually processed
        p = Path(f)
        try:
            manifest[str(p)] = {"mtime": p.stat().st_mtime, "hash": _md5_file(p)}
        except OSError:
            pass
    mp = Path(_MANIFEST_PATH)
    mp.parent.mkdir(parents=True, exist_ok=True)
    mp.write_text(json.dumps(manifest, indent=2), encoding="utf-8")
except Exception:
    pass

And drop the needs_update unlink from _rebuild_code (a code-only rebuild hasn't cleared the doc backlog, so it shouldn't clear the flag).

More broadly: save_manifest could take an optional "extracted files only" contract, or callers should never pass files they didn't process. The manifest is the source of truth for --update coverage, so anything that stamps un-extracted files is a correctness bug, not just a cache-warming shortcut.

Repro sketch

1. Corpus with both code and docs, run a full build so a manifest exists.
2. python -m graphify.watch <corpus> (or install the post-commit hook).
3. Add a new .md doc and touch a .py file, so the watcher takes the code-only rebuild path.
4. graphify --update → the new doc is reported as unchanged and never extracted; graph.json never gains its nodes. Manifest shows it stamped.

Environment

• graphify (pip package graphifyy), graphify.serve/graphify.watch in use
• Python 3.14, macOS
• Trigger path: --watch on a mixed code+docs corpus (also reachable via the post-commit hook)

[safishamsi]

Fixed in v0.9.21: the update runbook now stamps only files that actually produced output (mirroring the library extract path), so a failed or omitted chunk is re-queued on the next update instead of being marked done and lost. https://github.com/Graphify-Labs/graphify/releases/tag/v0.9.21