Support Shredded Lists/Array in `variant_get` by sdf-jkl · Pull Request #8354 · apache/arrow-rs

sdf-jkl · 2025-09-16T03:27:40Z

Which issue does this PR close?

closes [Variant] Support VariantPathElement::Index for Variant Arrays for variant_get #9443.

Rationale for this change

We should be able to variant_get using Indices to path through VariantArrays

What changes are included in this PR?

Are these changes tested?

Yes, unit tested.

Are there any user-facing changes?

scovich

Left a couple comments that are hopefully helpful.

Also, we should (eventually) support nesting -- arrays and structs inside arrays.
Let's get simple lists of primitives working first, tho!

…ed_list_support

scovich

I'm not sure I understand how these unit tests will translate to variant_get?

sdf-jkl · 2025-09-19T15:45:45Z

I'm not sure I understand how these unit tests will translate to variant_get?

Could you elaborate please?

I am currently trying to build just the Shredded List VariantArray test case, and while doing so learning how we could build them in shred_variant later. Once have a good way of building simple Shredded List VariantArray it will be easy to work on the rest of the unit tests for variant_get

scovich · 2025-09-19T21:18:02Z

I'm not sure I understand how these unit tests will translate to variant_get?

Could you elaborate please?

I am currently trying to build just the Shredded List VariantArray test case, and while doing so learning how we could build them in shred_variant later. Once have a good way of building simple Shredded List VariantArray it will be easy to work on the rest of the unit tests for variant_get

No worries -- the current iteration does look it produces a correct shredded variant containing a list, so I should probably just be patient and let you finish!

sdf-jkl · 2025-09-23T21:53:42Z

Hey @scovich I see that your current implementation of follow_shredded_path_element for VariantPathElement::Field when following the shredded path is successful, it returns a ShreddedPathStep::Success(field.shredding_state()) that holds a ShreddingState::Typed that holds a reference to the typed_value array. (That we later use for the next steps)

My question is: does ShreddedPathStep::Success() necessarily have to require the input ShreddingState to be a reference?

The reason I am asking is that since we use the output of follow_shredded_path_element to get the values from the shredded VariantArray, shouldn't we be free to drop the outer array once we extract the relevant typed_value?

The only way to work with list arrays I came up with so far, is to build new arrays with arrow_select::take, combining the path index and GenericListArray offsets.
But by using this method we create new arrays within the scope of the function and can't use a reference to the array in the ShreddedPathStep::Success.
(I just pushed a commit with a non-working implementation of the idea)

~~Should we instead look for another way to represent a resulting array consisting of slices instead?~~

I just saw the #8392

…ed_list_support

sdf-jkl · 2025-09-25T21:38:58Z

Hey @scovich I made it work for a one of the simple tests and it doesn't go through with the second one because Variant to Arrow does not support utf8 yet.

Do we have an issue tracking variant_to_arrow types support? If not, I can make one.

scovich · 2025-09-26T13:27:02Z

I made it work for a one of the simple tests and it doesn't go through with the second one because Variant to Arrow does not support utf8 yet.

Do we have an issue tracking variant_to_arrow types support? If not, I can make one.

I'm not sure we have a tracking issue for utf8 support in variant_to_arrow, but I've also noticed that it's an annoying gap for unit testing (we all seem to reach for string values...)

…ed_list_support

scovich · 2026-03-02T20:47:23Z

Everything looks good, code-wise -- nice and clean.

But there's still an open question of whether we intend to follow the jsonpath spec in our path step logic, as e.g. spark does?
#8354 (comment)

The jsonpath spec requires foo[100] to return NULL if foo is not an array, and also requires returning NULL if foo has fewer than 101 elements. Similarly, foo.bar should return NULL if foo is not a struct and should also return NULL if foo has no field named bar. Safe casting would only influence actual casting decisions, e.g. a variant_get call that specifically requests a string and the requested path points to a struct.

In contrast, our current struct handling code currently returns an error if safe casting is disabled and:

a Field path step encounters a "wrong" type (L169)
an Index path step encounters a "wrong" type (L224)
an Index path step is out of bounds (L99)

scovich · 2026-03-02T20:48:28Z

@alamb -- any opinions about supporting jsonpath semantics or not? Or ideas on who we should seek input from?

liamzwbao

Nice changes, thanks! One small nit on the tests.

…ed_list_support

klion26

LGMT, thanks for the contribution

…ed_list_support

sdf-jkl · 2026-03-19T15:09:28Z

@alamb following up on this.

Please check #8354 (comment)

alamb · 2026-03-21T21:18:46Z

@alamb -- any opinions about supporting jsonpath semantics or not? Or ideas on who we should seek input from?

In my opinion supporing jsonpath sounds like a good idea in general

The usecase I know of (and have) for variant_get is to mirror spark's variant_get function: https://docs.databricks.com/aws/en/sql/language-manual/functions/variant_get

So having the arrow-rs implementation match sounds good too

scovich · 2026-03-23T15:45:36Z

Filed #9606 to track the jsonpath semantics -- it goes beyond just this PR.

scovich · 2026-03-25T18:17:24Z

+                DataType::List(_) => take_list_like_index_as_shredding_state::<
+                    GenericListArray<i32>,
+                >(typed_value.as_ref(), *index)?,


Aside: It just makes my eyes hurt when fmt does stuff like this. But I don't know a way to make it better, unless we want to drastically shorten the function name ☹️

take_list_like_index_state?

Last one still won't fit 😢

VariantPathElement::Index { index } => { let state = match typed_value.data_type() { DataType::List(_) => take_list_like_index_state::<GenericListArray<i32>>( typed_value.as_ref(), *index, )?, DataType::LargeList(_) => take_list_like_index_state::<GenericListArray<i64>>( typed_value.as_ref(), *index, )?, DataType::ListView(_) => take_list_like_index_state::<GenericListViewArray<i32>>( typed_value.as_ref(), *index, )?, DataType::LargeListView(_) => take_list_like_index_state::< GenericListViewArray<i64>, >(typed_value.as_ref(), *index)?, _ => {

Making it take_list_index_state works:

VariantPathElement::Index { index } => { let state = match typed_value.data_type() { DataType::List(_) => { take_list_index_state::<GenericListArray<i32>>(typed_value.as_ref(), *index)? } DataType::LargeList(_) => { take_list_index_state::<GenericListArray<i64>>(typed_value.as_ref(), *index)? } DataType::ListView(_) => take_list_index_state::<GenericListViewArray<i32>>( typed_value.as_ref(), *index, )?, DataType::LargeListView(_) => take_list_index_state::<GenericListViewArray<i64>>( typed_value.as_ref(), *index, )?,

scovich · 2026-03-25T18:53:49Z

    // Peel away the prefix of path elements that traverses the shredded parts of this variant
    // column. Shredding will traverse the rest of the path on a per-row basis.
-    let mut shredding_state = input.shredding_state().borrow();
+    let mut shredding_state = input.shredding_state().clone();


This is ultimately cloning an ArrayRef (cheap) BinaryViewArray (not expensive, but not cheap either -- has to go through ArrayData). Should we change ShreddingState::value to Option<Cow<'a, BinaryViewArray>> to avoid unnecessary cloning for struct-only paths? Then we could get rid of BorrowedShreddingState entirely -- which BTW I think this PR currently leaves as dead code (but no clippy warnings because it's pub)

BorrowedShreddingState is still used in unshred_variant.rs, but that's it:

arrow-rs/parquet-variant-compute/src/unshred_variant.rs

Line 172 in 398962e

fn try_new_opt(shredding_state: BorrowedShreddingState<'a>) -> Result<Option<Self>> {

I tried using Cow before, but I was Cowing the whole ShreddingState, which was unnecessary.

I switched to .clone because value: Option<BinaryViewArray> is almost zero copy:

pub type BinaryViewArray = GenericByteViewArray<BinaryViewType>; // ... pub struct GenericByteViewArray<T> where T: ByteViewType + ?Sized, { data_type: DataType, // 24 bytes views: ScalarBuffer<u128>, // zero copy cloning buffers: Arc<[Buffer]>, // zero copy cloning phantom: PhantomData<T>, // zero sized nulls: Option<NullBuffer>, // mostly zero copy cloning + 3usize }

Cowing just the BinaryViewArray should work.

I'd defer to @alamb on how important this is -- I don't have a good sense of how expensive these shallow-deep clones are in practice, but I do know he was chasing a whole workstream of PR to avoid unnecessary ArrayData etc, which seems related.

@scovich the changes in this PR (#9610) will make value field an ArrayRef. This way there'd be no need for Cow or BorrowedShreddingState.

https://github.com/apache/arrow-rs/pull/9610/changes#diff-5f5ebb25cba94551493d3a9caa2ab4d94d19b1ce4ce3e3eac7662109a8966794R420

Well that solves that, then! How should we sequence that and the removal of BorrowedShreddingState?

We could push that PR forward, file a separate PR for BorrowedShreddingState and once they're done go back here.

Or push this one as is and fix it later

…th element (#9676) # Which issue does this PR close? Currently this is the only place in `main` that handles `Path` in `variant_get`. Other `variant_get` related PRs already follow the JSONPath sementics. (#9598 and #8354) - Closes #9606. # Rationale for this change Check issue # What changes are included in this PR? - Changed `variant_get` field path handling when can't cast to Struct - Updated the related unit test to check the new logic - Cleaned up some nearby tests # Are these changes tested? Yes, unit tests # Are there any user-facing changes? Yes, behavior change for `variant_get` kernel

sdf-jkl and others added 3 commits September 15, 2025 13:49

Add test shredded variant list array

9c25cc4

Add basic tests

ed961a4

Merge branch 'apache:main' into shredded_list_support

03ecb95

github-actions bot added the parquet-variant parquet-variant* crates label Sep 16, 2025

sdf-jkl mentioned this pull request Sep 16, 2025

[Variant] Support Shredded Lists/Array in variant_get #8082

Closed

scovich reviewed Sep 16, 2025

View reviewed changes

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

sdf-jkl and others added 2 commits September 16, 2025 16:21

Merge branch 'apache:main' into shredded_list_support

158d6d7

Redo test shredded array

d53c831

sdf-jkl commented Sep 17, 2025

View reviewed changes

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

sdf-jkl commented Sep 17, 2025

View reviewed changes

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

sdf-jkl commented Sep 17, 2025

View reviewed changes

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

sdf-jkl commented Sep 17, 2025

View reviewed changes

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

sdf-jkl commented Sep 17, 2025

View reviewed changes

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

scovich mentioned this pull request Sep 18, 2025

[Variant] Define new shred_variant function #8366

Merged

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

174e429

…ed_list_support

scovich reviewed Sep 19, 2025

View reviewed changes

Rebuild the shredded list array

69de7d7

sdf-jkl added 2 commits September 23, 2025 17:54

Use select::take to build the output array

cc6d787

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

8f6ad1b

…ed_list_support

sdf-jkl commented Sep 23, 2025

View reviewed changes

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

sdf-jkl added 3 commits September 25, 2025 11:46

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

bc8abd9

…ed_list_support

Pass one test

c0d2065

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

85aaa3f

…ed_list_support

Get typed values directly

40b6311

Added support for utf8, largeUtf8, utf8view

f6e88ef

sdf-jkl added 5 commits February 25, 2026 11:21

nit fix

cbfa058

use else if chain

cf94d43

add cast_options.safe docs

91589ad

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

e8e7fb1

…ed_list_support

support list-like arrays

28ec53c

sdf-jkl marked this pull request as ready for review February 26, 2026 04:10

liamzwbao reviewed Mar 2, 2026

View reviewed changes

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

match typed value instead of donwcast attempts

279b634

liamzwbao reviewed Mar 6, 2026

View reviewed changes

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

sdf-jkl added 3 commits March 10, 2026 11:25

clean up tests

18d04d7

ListLikeArray is now in arrow::array

8d838cc

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

0fe090a

…ed_list_support

liamzwbao approved these changes Mar 10, 2026

View reviewed changes

klion26 approved these changes Mar 11, 2026

View reviewed changes

sdf-jkl added 2 commits March 19, 2026 10:58

Merge branch 'main' of https://github.com/apache/arrow-rs into shredd…

51d7e42

…ed_list_support

fmt

b178554

scovich mentioned this pull request Mar 23, 2026

variant_get should follow JSONpath semantics #9606

Closed

JSONpath semantics

66873a2

scovich reviewed Mar 24, 2026

View reviewed changes

Comment thread parquet-variant-compute/src/variant_get.rs Outdated

fix index overflow by @scovich

2135838

scovich reviewed Mar 25, 2026

View reviewed changes

sdf-jkl mentioned this pull request Mar 27, 2026

[Variant] Support Binary/LargeBinary children #9610

Merged

Merge branch 'main' into shredded_list_support

7a5dcb4

sdf-jkl mentioned this pull request Apr 8, 2026

[Variant] variant_get should follow JSONPath semantics for Field path element #9676

Merged

Conversation

sdf-jkl commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

scovich left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

sdf-jkl commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scovich commented Sep 19, 2025

Uh oh!

sdf-jkl commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sdf-jkl commented Sep 25, 2025

Uh oh!

scovich commented Sep 26, 2025

Uh oh!

Uh oh!

scovich commented Mar 2, 2026

Uh oh!

scovich commented Mar 2, 2026

Uh oh!

liamzwbao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

klion26 left a comment

Choose a reason for hiding this comment

Uh oh!

sdf-jkl commented Mar 19, 2026

Uh oh!

alamb commented Mar 21, 2026

Uh oh!

scovich commented Mar 23, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

sdf-jkl commented Sep 16, 2025 •

edited

Loading

scovich left a comment •

edited

Loading

sdf-jkl commented Sep 19, 2025 •

edited

Loading

sdf-jkl commented Sep 23, 2025 •

edited

Loading