[VL] Add min/max partition stats to columnar InMemoryRelation cache for partition pruning#12092

Open

yaooqinn wants to merge 72 commits into

apache:mainfrom

yaooqinn:gluten-cache-partition-stats

Commits on May 12, 2026

[GLUTEN-XXXXX][VL] PA-1 RED: ColumnarCachedBatchKryoSuite for stats field + magic-prefix sniff
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-1 GREEN: CachedColumnarBatch stats field + Kryo magic-prefix sniff (BE)
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.1 RED: VeloxColumnarBatchSerializer::computeStats(Long FlatVector)
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.1 GREEN: ColumnStats + computeStats(BIGINT FlatVector)
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.2 RED: NaN Float partition must not participate in pruning
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.2 GREEN: REAL FlatVector scan with NaN poison guard
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.4 RED: HUGEINT (long Decimal P>=19) FlatVector min/max
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.4 GREEN: HUGEINT (long Decimal P>=19) FlatVector scan
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.5a RED: framedSerializeWithStats top-level layout
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.5a GREEN: framedSerializeWithStats top-level layout
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.5b RED: statsBlob layout (BIGINT 1-col)
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.5b GREEN: statsBlob BIGINT marshal + PA-2.5a sentinel relax
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.5c: JNI bridge serializeWithStats (non-TDD plumbing)
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-2.6: capability check supportsStatsExt (non-TDD plumbing)
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-3.0 doc-only: fix scaladoc 'graceful pass-through' claim
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-3.1 RED: buildFilter stats=null NPE under EqualTo predicate
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-3.1 GREEN: lazy-split iterator wrapper for buildFilter stats=null guard
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-3.2 RED: statsBlob binary framing (BIGINT 1-col)
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-3.2 GREEN: replace Java Ser placeholder with cpp-aligned statsBlob marshal
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-3.3 RED: framed bytes parser test (BIGINT 1-col)
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-3.3 GREEN: parseFramedBytes + write path JNI wiring
yaooqinn
committed
[GLUTEN-XXXXX][VL] PA-3.4: lock down lazy-split wrapper prune semantics + fix exhaustion bug
yaooqinn
committed
PA-3.5: add ColumnarCachedBatchE2ESuite (e2e smoke)
yaooqinn
committed
PA-4: SQLConf gate spark.gluten.sql.columnar.tableCache.partitionStats.enabled (default false)
yaooqinn
committed
PA-3.5: fix numOutputRows lower bound (0 is legal full-prune)
yaooqinn
committed

Commits on May 13, 2026

Commits on May 14, 2026

Commits on May 15, 2026