table: Add ranged dumps for batched export - #2755
Conversation
8154244 to
04a80ac
Compare
There was a problem hiding this comment.
I'd think add a limit is not a good solution.
In this case, I think we can have a async, batch export to support a large data table. @madcodelife
Thanks review, that makes sense. A total export limit would reject legitimate large tables rather than support them. A possible direction would be an asynchronous batched export: prompt for the destination first, materialize only one fixed-size row batch at a time through the foreground GPUI context, and pass each batch through a bounded channel to a background CSV writer. This would avoid building the complete table or CSV in memory, while still exporting every row. The batch size would control peak materialization and scheduling rather than impose a total data limit. The existing eager, unbounded I’ll hold off on further changes to the current limit-based approach and wait for your discussion with @madcodelife on the preferred design and ownership. @madcodelife, if you would like me to continue with this change, would you prefer a public row-range batching primitive on |
|
Agreed, in fact in our application dump is already called from an async context, so this direction matches real usage. For your question: prefer a public row-range API on The story's |
|
Thanks, that clarifies the intended boundary. I’ll rework this PR to add a I’ll keep progress UI, cancellation, remote pagination, and delegate changes |
04a80ac to
8169b02
Compare
8169b02 to
3964f6d
Compare
|
Reworked this PR based on the review discussion:
Progress UI, cancellation, remote pagination, delegate redesign, and snapshot semantics remain outside this PR. @huacnlee @madcodelife, could you please take another look? |
|
Thank you. |
Closes #2754
Description
TableState::dump_rangelets callers materialize a selected row range whilepreserving the existing full-table
dumpAPI. The requested range is clampedto the delegate's current row count so applications can iterate over large
tables in fixed-size batches.
The DataTable story now prompts for the destination before reading table data,
then materializes 2,000 rows at a time on the foreground GPUI context. A
capacity-one channel sends those batches to one background
csv::Writer<File>,which streams directly to disk without accumulating the complete table, CSV
buffer, or
Stringin memory.TableState::dumpremains an eager whole-table API for compatibility. Largedata exporters should use
dump_rangewith an application-level batch size.Progress UI, cancellation, remote pagination, and delegate changes remain
outside this PR.
Screenshot
Not applicable. This changes export scheduling and file I/O rather than the
rendered table UI.
How to Test
Observed results:
table_dump_rangetests passed;gpui-componenttest suite passed;the current
wasm_threaddependency (RUSTC_BOOTSTRAP=1); a plain stablecheck stops in that dependency with
E0554before compiling the story;produced no output, while exporting the default table produced one header
and 5,000 records with 45 columns each;
0, the last had ID4999, and the CSVparsed successfully with consistent column counts.
The ranged dump regression tests verify that
dump_range(98..103)is clampedto a 100-row table and only calls
cell_textfor rows 98 and 99, while theexisting
dump(cx)still returns the complete table.AI Assistance
AI assisted with root-cause analysis, implementation, test design, validation
planning, and PR wording. I reviewed the final diff and ran the checks listed
above.
Checklist
cargo runfor story tests related to the changes.