Skip to content

_last_updated_sequence_number: physical per-row values discarded when first_row_id is null #2991

Description

@anoopj

Apache Iceberg Rust version

None

Describe the bug

Context

When reading _last_updated_sequence_number, the Arrow reader gates on the data
file's first_row_id: if it is null, the whole column is nulled and any per-row
values the file physically carries are discarded (the physical leaf is never
projected).

Introduced in #2985 alongside the coalesce path.

Behavior

This matches Java (ValueReaders.lastUpdated returns nulls when the base row id
is null), so it is not an interop regression. But the spec does not forbid such a
file, so it is a divergence between what the bytes can express and what we return.
Flagged in review as worth tracking rather than fixing in the coalesce PR.

Options

  • Keep the Java-parity behavior (current) ie a file with no row lineage has no
    meaningful per-row sequence numbers.
  • Pass non-null per-row values through, leaving null rows null when there is no
    fallback. Note that this diverges from Java.

To Reproduce

No response

Expected behavior

No response

Willingness to contribute

I can contribute a fix for this bug independently

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions