Skip to content

perf(weave): avoid deep-copying table rows on table_create#7190

Draft
gtarpenning wants to merge 1 commit into
masterfrom
griffin/table-create-shallow-copy
Draft

perf(weave): avoid deep-copying table rows on table_create#7190
gtarpenning wants to merge 1 commit into
masterfrom
griffin/table-create-shallow-copy

Conversation

@gtarpenning

@gtarpenning gtarpenning commented Jun 11, 2026

Copy link
Copy Markdown
Member

Summary

  • table_create deep-copied every row (req.model_copy(deep=True)) just to rewrite project_id. Replaced with a shallow copy of the request wrapper + table that shares the rows list; the ref converter is already copy-on-write for rows, so the caller's request stays unmutated.
  • Drops one full recursive copy + ~2x peak memory of the payload on the hot table-write path. No behavior change: digests, stored values, and ref conversion are unchanged.

Context: prod POST /table/create is slow on multi-MB payloads (Datadog us5, env:prod) - 27.6s/10MB inserted, 37.2s rejected. This deep copy is a pure-waste CPU/memory pass per request under concurrent large uploads.

Performance

Local ClickHouse, end-to-end server.table_create, median of 12 runs with fresh rows (no CH dedup). Isolated, the deep-copy step alone is ~46ms / 7MB peak on a 10MB payload; the shallow copy is ~0.03ms / ~0MB.

payload median ms p90 ms peak MB peak reduction
2 MB 135 -> 128 149 -> 136 7.86 -> 6.78 -14%
2 MB (20% ref rows) 140 -> 143 191 -> 184 7.33 -> 6.52 -11%
10 MB 328 -> 266 354 -> 276 23.3 -> 17.9 -23%
10 MB (20% ref rows) 303 -> 268 315 -> 287 22.8 -> 18.7 -18%

Testing

added a functional test asserting the caller's rows are not mutated and embedded refs still convert; existing table_create tests pass.

table_create deep-copied every row just to rewrite project_id. Replaced with
a shallow copy of the request wrapper + table that shares the rows list; the
ref converter is already copy-on-write for rows, so the caller's request
stays unmutated and no per-row deep copy is made.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wandbot-3000

wandbot-3000 Bot commented Jun 11, 2026

Copy link
Copy Markdown

@gtarpenning gtarpenning changed the title perf(trace_server): avoid deep-copying table rows on table_create perf(weave): avoid deep-copying table rows on table_create Jun 11, 2026
@codecov

codecov Bot commented Jun 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant