Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,50 @@ Writes to `benchmarks/results/speed-<sha12>.json`.

</details>

<details>
<summary>Cold index phase profiler</summary>

Profile fresh `create_index_from_path` calls without enabling production telemetry:

```bash
uv run python -m benchmarks.profile_cold_index \
--corpus-path /path/to/source-tree \
--model-path /path/to/model \
--repetitions 5 \
--label experiment-name \
--revision revision-id \
--output /path/to/profile.json
```

The model is loaded before measurement; every repetition passes `previous=None`. Cold means fresh index state, not a
fresh process or cold parser/filesystem caches. `records` contains raw per-run nanosecond timings, counts, call-size
arrays, index shape/vector bytes, and process peak RSS. `summary` recursively reports median/min/max for numeric fields
and `embedding_invariants` states whether each duplicate-pass check held across every repetition.

`total.instrumented_wall_ns` and `total.instrumented_process_cpu_ns` surround `create_index_from_path`, including all
wrapper and recorder overhead. They must not be compared with an uninstrumented benchmark; measure that total
separately. Phase `wall_ns` is inclusive, `exclusive_wall_ns` removes nested measured boundaries, and
`overlap.nested_wall_ns` names each parent/child overlap. Inclusive parent and child times must not be summed.

For each phase, `calls` counts boundary entries and `items` counts outputs appropriate to that boundary.
`file_walk_iterator` includes traversal work and its internal filesystem checks. `file_status_checks` includes file
eligibility stats and sub-128-byte emptiness probes; its `items` count is checked files, not individual stat calls.
`source_reads` covers only the subsequent production source read;
`counts.source_read_bytes` is the UTF-8 size of its returned text and is counted after the timer. Direct manifest mtime
stats are unattributed. `call_sizes` plus count/total/median/min/max are emitted for BM25 updates, `embed_chunks`,
`StaticModel.encode`, and Model2Vec tokenizer batches. `index` reports files, chunks, dimensions, and vector bytes.

`static_model_encode` is the exact `StaticModel.encode` API boundary, not pure native model time.
`model2vec_tokenization` is nested within it; `model2vec_lookup_mean_stack_normalization` is the derived remainder
(`encode - tokenize`) and includes Python overhead. `unattributed_including_profiler_overhead` is the root exclusive
remainder: unwrapped index orchestration, vector/BM25 finalization, manifest mtime stats, and profiler bookkeeping.

`memory.process_peak_rss_bytes` is a monotonic process-lifetime high-water mark. It includes the preloaded model and
all earlier repetitions, so it is not a per-repetition allocation delta. The recorder is single-thread-only; a future
parallel indexer requires thread-local or synchronized instrumentation.

</details>

<details>
<summary>Ablations</summary>

Expand Down
Loading