Skip to content

[GLUTEN-12094][VL] Strip default comparator from array_sort for Velox offloading#12095

Open
surendralilhore wants to merge 2 commits into
apache:mainfrom
surendralilhore:arraysort_
Open

[GLUTEN-12094][VL] Strip default comparator from array_sort for Velox offloading#12095
surendralilhore wants to merge 2 commits into
apache:mainfrom
surendralilhore:arraysort_

Conversation

@surendralilhore
Copy link
Copy Markdown
Contributor

@surendralilhore surendralilhore commented May 14, 2026

Detect Spark's default null-handling comparator in ArraySort and strip the lambda so Velox uses its 1-arg array_sort (ascending, nulls-last) which has identical semantics.

What changes were proposed in this pull request?

When Spark's array_sort is called without an explicit comparator (the default ascending sort), Spark internally
generates a LambdaFunction that wraps the simple ascending comparison in null-handling If/IsNull logic. Velox's
rewriteArraySortCall uses a SimpleComparisonMatcher that cannot parse this null-handling wrapper, causing the
expression to fall back to Vanilla Spark.

This PR detects whether the ArraySort expression uses Spark's default comparator by reconstructing it via
ArraySort.comparator() and comparing semantically. When a match is found, the lambda is stripped and only the array
argument is passed, so Velox uses its native 1-arg array_sort which already provides ascending sort with nulls-last —
identical to Spark's default semantics.

Custom comparators (e.g., descending sort) continue to be passed through as the 2-arg form.

How was this patch tested?

Added "array_sort - default comparator offloading" test in MiscOperatorSuite covering:

  • array_sort(a) — default ascending sort
  • sort_array(a) — which uses the default comparator internally
  • sort_array(a, false) — descending sort
  • array_sort(a, (l, r) -> ...) — custom comparator (2-arg form preserved)

All cases verify that ProjectExecTransformer is present in the Gluten plan.

Was this patch authored or co-authored using generative AI tooling?

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude claude-opus-4.6

Related issue: #12094

…offloading

Detect Spark's default null-handling comparator in ArraySort and strip
the lambda so Velox uses its 1-arg array_sort (ascending, nulls-last)
which has identical semantics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added the VELOX label May 14, 2026
Replace em dash characters with regular hyphens to pass scalastyle
nonascii check.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@philo-he philo-he changed the title [GLUTEN-12094][VL]Strip default comparator from array_sort for Velox offloading [GLUTEN-12094][VL] Strip default comparator from array_sort for Velox offloading May 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant