Skip to content

Repo: embed before writing, batch the first index, one lock across processes - #11

Merged
ashish-bagri merged 3 commits into
mainfrom
repo-embed-before-transaction
Nov 19, 2026
Merged

ashish-bagri merged 3 commits into
mainfrom
repo-embed-before-transaction

Conversation

@sudopower

@sudopower sudopower commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Why

Indexing glassflow/tares took 108 minutes, and 105 of them were embedding. All of it ran inside one transaction, and new vectors reached embedding_cache only after the commit. If the index Job hit its time limit or crashed, everything rolled back, so the retry started from zero.

What changes

Every index run

  • Embed first, then write. Files are chunked and embedded before a write transaction opens. A transaction now lasts seconds, not the whole run.
  • The cache is saved during the run. Every 256 new vectors are written to embedding_cache on a second connection, and the rest are written at the end, even if the run fails. A retry finds them and skips that embedding.

A repo's first index (no indexed_commit yet)

  • Commits in batches. Every LORE_FIRST_INDEX_BATCH files (default 200) commit in their own transaction, so the repo is searchable within minutes. indexed_commit is set only by the last transaction.
  • Resumes after a failure. A run that dies keeps its batches. The retry skips files already stored with the same content hash.
  • Shows as partial. The MCP list_sources tool returns partial: true until the last batch commits. The admin page shows "indexing (partial)".

Later indexes

  • Stay all-or-nothing. They still commit in one transaction, so search never mixes two commits.
  • Bounded memory. At most LORE_MAX_HELD_CHUNKS (default 50k, about 100 MB) prepared chunks wait for that transaction. Files past it are embedded first, which warms the cache, then prepared again from the cache inside the transaction.

One lock across processes

  • Server pushes and syncs now take the per-source advisory lock that lore index already held.
  • Embedding now happens before the row lock, so without this a server and a Job would embed the same repo at once.
  • A push that meets a running index gets 409 index_in_progress. The poller skips that repo quietly and doesn't record an error.
  • The lock connects with the pool's own settings, so it's always in the database being written to. Advisory locks are per database.

Behaviour changes

  • Deleting during embedding: a source or tenant delete no longer waits for a running index to finish embedding. The run finds its source gone and writes nothing.
  • Push during a Job: a push during an index Job is refused with 409 index_in_progress instead of queueing behind it.
  • Cache after a failure: a failed run's vectors stay in the cache. A cache entry is the embedder's vector for an exact text, valid whether or not the run commits.

Review

A second model reviewed the change independently. Its findings that this PR fixes:

  • The server's lock was taken in the database from LORE_DATABASE_URL, not the pool's database. This would have failed CI, which runs Postgres on 5432.
  • The source was read before taking the lock, so a run could act on a stale indexed_commit.
  • A later index held all its prepared chunks in memory with no limit.
  • A sync that met a running Job recorded a sync error on every poll.

Known and not fixed here:

  • Progress: it counts files as done when they're embedded, before they're written.
  • One extra connection per active run: each run holds a lock connection besides the pool's 10. This only matters with many repos indexing at once.

Testing

cargo fmt --check, cargo clippy --all-targets -- -D warnings and cargo test --all-targets pass against Postgres. New tests cover:

  • partial search during a batched first index
  • resuming after a failed first index
  • a later index staying all-or-nothing
  • the held-chunk limit
  • a push getting 409 while another process holds the lock

The cli_index tests of a push queued behind a Job now expect 409. They used to pass only because the two locks were in different databases.

🤖 Generated with Claude Code

sudopower and others added 3 commits November 18, 2026 17:25
…cache

A push used to chunk and embed every changed file inside one transaction that held the
source's row lock, and new vectors were saved to the embedding cache only after it committed.
If a job was killed partway (job time limit, crash), all of that embedding work was lost, so
the retry embedded everything again.

- Every changed file is now chunked and embedded first, with no transaction open. The
  transaction then only writes the rows and moves indexed_commit, so search still switches
  to the new commit all at once.
- Every 256 new vectors are written to embedding_cache on a second connection while the
  push runs. A retry finds them there and does not embed those chunks again. With no free
  connection (a one-connection pool), they are written after the commit, as before.
- A file that looked unchanged during the embedding phase is checked again under the row
  lock, and prepared again if another writer changed it in between.
- Deleting a source or tenant no longer waits for a running push to finish embedding. The
  push then finds its source gone and writes nothing. The two tests that expected the delete
  to wait now test this instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ery index run

- A repo's first index commits every LORE_FIRST_INDEX_BATCH files (default 200) in its own
  transaction. The repo is searchable as it goes, and a run that dies keeps its batches: the
  retry finds those files stored with the same hash and skips them. indexed_commit is set
  only by the last transaction. Later indexes stay one transaction, so search never mixes
  two commits.
- list_sources (MCP) gains `partial`: a repo with documents but no indexed commit yet. The
  admin page shows such a repo as "indexing (partial)".
- Server pushes and syncs now take the per-source advisory lock that `lore index` already
  held. Embedding now runs before the row lock, so without it a server and a Job would embed
  the same repo at once. A push meeting a running index gets 409 index_in_progress; the
  poller skips the repo quietly.
- Each write transaction locks the source row and checks the indexed commit (lock_source),
  for a process without the advisory lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fresh source

- The advisory lock connects with the pool's own options, not LORE_DATABASE_URL. Advisory
  locks are per database: the server's lock landed in another database than the one it wrote
  to (tests passed locally only because both URLs pointed at the same server; CI would fail).
  Its connect and lock query are bounded by the query timeout.
- Push takes the advisory lock before reading the source; sync re-reads the source once it
  holds the lock and stands down (Busy) if another process indexed meanwhile. Busy is not
  recorded as a sync error.
- A later index holds at most LORE_MAX_HELD_CHUNKS (50k, about 100 MB) prepared chunks for its
  one transaction; files past it are embedded first (warming the cache) and prepared again in
  the transaction.
- Fresh vectors are published to the cache after a failed run too: they are what the retry
  needs.
- The first-index and later-index paths are explicit branches; Job::delete_doc is gone.
- The cli_index tests of a push queued behind a Job now expect 409 index_in_progress: both
  hold the same advisory lock now. Their old pass relied on the lock landing in another
  database.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sudopower sudopower changed the title Repo: embed before the push transaction and checkpoint the embedding cache Repo: embed before writing, batch the first index, one lock across processes Nov 18, 2026
@ashish-bagri
ashish-bagri merged commit ebd9f68 into main Nov 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants