Skip to content

TR-1495: files_in_index in lore's sync report and repo list - #12

Merged
ashish-bagri merged 1 commit into
mainfrom
factory/TR-1495
Nov 19, 2026
Merged

ashish-bagri merged 1 commit into
mainfrom
factory/TR-1495

Conversation

@ashish-bagri

@ashish-bagri ashish-bagri commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

TR-1495: https://linear.app/glassflow/issue/TR-1495/lore-a-sync-report-and-the-repo-list-say-how-many-files-the-repos

What changed

  • IndexReport has a new files_in_index: how many files the repo's index holds after the run (indexed plus unchanged, never skipped or deleted ones). It is set by the first index, the bundle push and the incremental sync, and by an up to date sync, which indexes nothing and still says what the index holds. lore index prints it too.
  • The name is files_in_index, not files_total, because /progress and /admin/pushes already return files_total for the files a run will index or delete (Ashish chose the rename, D402). Those two are unchanged.
  • In index_commit_staging (after Repo: embed before writing, batch the first index, one lock across processes #11) the count is taken in the last transaction of the run, the one that sets indexed_commit. A first index commits in batches; by then they are all committed, so the count holds them plus the last transaction's own writes.
  • Each RepoItem of GET /v1/repos (and the POST /v1/repos answer) carries files_in_index. The list is counted with one query for all repo sources of the tenant.
  • The count is count(*) over the source's documents rows in Postgres. It never walks the git tree. No migration.
  • Older servers: files_in_index is #[serde(default)] where the CLI reads a report or a repo item back, so a newer CLI against an older server still works (it reads 0).
  • docs/usage.md documents the field and says it is not the files_total of the progress route.

Release note (v0.8.x patch)

The index report, the sync report and GET /v1/repos now carry files_in_index, the number of files the repo's index holds after the run.

Checks

Tests in tests/repo_files_in_index.rs (first index in 3 batches of 100; ticket checks 1 to 3 through sync and bundle push; skipped files not counted; the list counts each repo and only the caller's; POST /v1/repos of an existing repo) and in tests/cli_repo.rs (the CLI push and its up-to-date path read 6). Each was broken and restored:

  • CLI up-to-date report reading 0: push_indexes_full_then_incremental_then_nothing fails (0 vs 6).
  • POST /v1/repos answering 0 for an existing repo: a_sync_and_the_repo_list_count_the_files_the_index_holds fails (0 vs 301).
  • Count taken after the first batch of a first index and indexing stopped there: cli_repo fails (0 vs 6).
  • Count taken only inside the first-index batch loop, none on the incremental path: both ticket-check tests fail (0 vs 301).
  • Earlier rounds: total from files_indexed + files_unchanged (2 vs 301), skipped files added (5 vs 3), list ignoring totals (0 for every repo).

Run in rust:1.95-trixie against pgvector/pgvector:pg17 (the crew machine has no Rust toolchain):

  • cargo fmt --all and cargo clippy --all-targets -- -D warnings: clean.
  • cargo test --all-targets --no-fail-fast on the rebased tree: 744 passed, 3 failed. Two are cli_search tests (onnxruntime prints a cpuid warning on stderr in the arm64 container; they fail the same on clean origin/main). One was cli_repo::push_of_a_repo_without_commits_fails with PoolTimedOut (test database connection); it passes on rerun (13 of 13). After the last small edits (a batch size in the test setup, fmt) repo_files_in_index and cli_repo pass again.

Notes

@ashish-bagri

Copy link
Copy Markdown
Member Author

factory/review: pass at cd31cff (TR-1495)

Reviewer: ashish-crew-reviewer

Verified

  • Unbroken, in Docker (rust:1.95-trixie, -j 2): cargo test --no-fail-fast --test repo_files_total --test cli_repo --test cli_index --test sync gave 46 passed, 0 failed. --test index_runs gave 4 passed.
  • I broke each counting path and checked that a test fails:
    • Counting from the run's counters: 2 failed.
    • The sync up-to-date path returning 0: 1 failed.
    • Counting outside the push tx: 1 failed.
  • The premise holds: one documents row per indexed file, skipped files are deleted, and repo_file_totals filters by tenant and kind='repo'. The count in the tx runs under the per-source lock. IndexReport is never stored, so there is nothing to migrate.
  • The builder's full suite gave 742 passed and 2 failed. The 2 are the known arm64 cli_search failures, which fail on main too. fmt and clippy are clean. CI passes.

Findings (none blocking)

  • F1 (P3), src/repo/mod.rs:90: files_total now means "files in the index" in the push and sync report and in the repo list. It already means "files this run will index or delete" (null until known) in /v1/repos/{id}/progress and /v1/admin/pushes. Consider a distinct name, or at least say in docs/usage.md that the two differ.
  • F2 (P3), src/http/repos.rs:107 and src/cli/repo.rs:142: two paths are untested. POST /v1/repos for an existing repo and the CLI's up-to-date push both still pass every test when they return 0. Also, #[serde(default)] on the CLI RepoItem reads an older server's missing field as 0.

Not verified

  • I did not re-run the full suite, fmt or clippy; I relied on the builder's run and CI.
  • I did not measure the cost of the list query on a large tenant, and I did not test with Tares as the consumer.

@ashish-bagri

Copy link
Copy Markdown
Member Author

factory/review: changes (round 2, head bce20b5, reviewer ashish-crew-reviewer)

Blocking

  • F3 P2, src/repo/mod.rs:521: the PR conflicts with main. Repo: embed before writing, batch the first index, one lock across processes #11 (ebd9f68) rewrote index_commit into index_commit_staging. A first index now commits in batches (main mod.rs ~511-524), and only a later index publishes in one transaction (~528-551). This PR counts once, "inside the transaction", just before set_indexed_commit. After the rebase, the count must come after the last batch of a first index, in the same transaction as set_indexed_commit, and the comment must match. Per A43, rebase onto main, rerun repo_files_in_index and cli_repo, and ask for a review again.

Earlier findings

  • F1, fixed: the field is renamed to files_in_index, and docs/usage.md says it is not the progress route's files_total.
  • F2, fixed: a create of an existing repo now asserts 301, and the CLI's up-to-date push asserts 6. Making the CLI field an Option instead of #[serde(default)] is still a side note, not blocking.

Verified

  • I read the delta cd31cff..bce20b5 in full; the rename is complete.
  • git merge-tree --write-tree HEAD origin/main conflicts in src/repo/mod.rs.

Not verified

  • I ran no cargo at bce20b5. The head cannot merge, and the resolution rewrites the counted path, so the suite runs on the rebased head.

…/repos

The number of files the repo's index holds after a run (indexed plus unchanged,
never skipped or deleted ones), counted from the document rows in Postgres.
Named files_in_index because /progress and /admin/pushes already return
files_total for the files a run indexes or deletes.
@ashish-bagri ashish-bagri changed the title TR-1495: files_total in lore's sync report and repo list TR-1495: files_in_index in lore's sync report and repo list Nov 19, 2026
@ashish-bagri

Copy link
Copy Markdown
Member Author

factory/review: pass at a00ffae (round 3, TR-1495)

F3 is fixed. files_in_index is now counted with count_source_files inside the final transaction, after the first-index batches and delete_vanished, just before set_indexed_commit. The up-to-date sync path counts from the pool. (F1 and F2 were fixed in round 2.)

What I verified (fresh worktree, Docker rust:1.95, lore-builder Postgres)

  • cargo fmt --all -- --check: clean.
  • cargo clippy -j 2 --all-targets -- -D warnings: clean.
  • cargo test --test repo_files_in_index --test cli_repo --test index_runs --test repo_atomic: 4, 13, 17 and 26 passed, 0 failed.
  • I broke the code, ran the tests and restored it each time. These breaks are caught:
    • Counting before the deleted-paths loop: 2 fail (302 != 301).
    • Counting from the pool outside the transaction: 2 fail (300 != 301).
    • The up-to-date sync reporting 0: 1 fails (0 != 301).

Findings (non-blocking)

  • F4 (P3), tests/repo_files_in_index.rs: two breaks survive the new tests.
    • Counting before delete_vanished: no test runs a full-mode sync in which a file vanished.
    • Dropping the tenant filter in repo_file_totals: no test lists two tenants' repos on one server.
    • One case each would pin them.

Not verified

  • The full cargo test --all-targets suite. My run failed only on "No space left on device", because the Docker VM disk was full (environment, not this PR). CI on this head is green.
  • A live push against a real server.

Ashish merges lore by hand (D401).

— ashish-crew-reviewer

@ashish-bagri
ashish-bagri merged commit dcb838e into main Nov 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant