Skip to content

feat(parquet): omit unknown fields from writes - #5

Draft
manuzhang wants to merge 2 commits into
add-unknown-datatypefrom
add-unknown-datatype-parquet-write
Draft

feat(parquet): omit unknown fields from writes#5
manuzhang wants to merge 2 commits into
add-unknown-datatypefrom
add-unknown-datatype-parquet-write

Conversation

@manuzhang

@manuzhang manuzhang commented Aug 26, 2026

Copy link
Copy Markdown
Owner

Which issue does this PR close?

No issue is currently linked.

What changes are included in this PR?

This is layer 2 of GitHub stack apache#8 and depends on #1.

  • Builds the physical Parquet Arrow schema by omitting Unknown fields.
  • Rejects containers that would have no valid physical Parquet representation.
  • Projects incoming record batches to the physical schema before writing while preserving nested container structure and row counts.
  • Updates NaN statistics traversal to work against the projected physical batch.

Are these changes tested?

  • Added schema and batch-projection regressions for top-level and nested Unknown fields, invalid empty containers, and name-based matching.
  • cargo test -p iceberg --lib (1,594 passed at this stack layer)
  • Full stack: make check-fmt
  • Full stack: make check-clippy
  • git diff --check

AI Disclosure

This PR was prepared with assistance from Codex.

manuzhang and others added 2 commits August 26, 2026 12:38
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Codex <codex@openai.com>
@manuzhang manuzhang changed the title add unknown datatype parquet write feat(parquet): omit unknown fields from writes Aug 26, 2026
@manuzhang
manuzhang force-pushed the add-unknown-datatype branch 2 times, most recently from 02756bf to a7efd92 Compare August 26, 2026 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant