[Parquet]: GH-563: Make `path_in_schema` optional by etseidl · Pull Request #9678 · apache/arrow-rs

etseidl · 2026-04-08T20:56:49Z

Which issue does this PR close?

none

Rationale for this change

This is a proof of concept implementation for apache/parquet-format#563

What changes are included in this PR?

Since version 57.0.0, this crate has been tolerant of a missing path_in_schema. This PR adds options to cease writing the field as well. The option defaults to continuing to write the field.

See related discussion on parquet mailing list: https://lists.apache.org/thread/czm2bk45wwtkhhpqxqvmx9dk5wkwk1kt

Are these changes tested?

Yes

Are there any user-facing changes?

No, this only adds an optional behavior change that defaults to no change

Related PRs

alamb · 2026-04-08T21:37:15Z

less bloat for the win!

jhorstmann · 2026-04-10T08:00:38Z

A lot of other parquet implementations require this field, due to their generated thrift parser, even if they do not actually use the field for anything. I would be totally in favor of deprecating and skipping the field, but maybe a more compatible alternative would be to write the field as an empty list instead.

etseidl · 2026-04-10T13:48:57Z

A lot of other parquet implementations require this field, due to their generated thrift parser, even if they do not actually use the field for anything. I would be totally in favor of deprecating and skipping the field, but maybe a more compatible alternative would be to write the field as an empty list instead.

Well, parquet-java actually uses the field to populate its version of ColumnDescriptor, so an empty list will be just as damaging to an old version. The whole idea is to change the field to optional in the thrift definition, and give the ecosystem a few years for that change to percolate. After some reasonable time has passed we can default to not writing the field. But in the meantime, users who have data sensitive to metadata bloat and know they have up-to-date tooling can help themselves earlier.

alamb · 2026-04-10T17:58:09Z

A lot of other parquet implementations require this field, due to their generated thrift parser, even if they do not actually use the field for anything. I would be totally in favor of deprecating and skipping the field, but maybe a more compatible alternative would be to write the field as an empty list instead.

Well, parquet-java actually uses the field to populate its version of ColumnDescriptor, so an empty list will be just as damaging to an old version. The whole idea is to change the field to optional in the thrift definition, and give the ecosystem a few years for that change to percolate. After some reasonable time has passed we can default to not writing the field. But in the meantime, users who have data sensitive to metadata bloat and know they have up-to-date tooling can help themselves earlier.

See also related mailing list discussion:

https://lists.apache.org/thread/czm2bk45wwtkhhpqxqvmx9dk5wkwk1kt

alamb

TLDR is I think this is a great idea. I also think it woudl be ok to merge this into arrow-rs even if there is not a broader consensus on the mailing list that we should do it in the format itself

My thinking is that some usecases are basically using parquet with the same read/writer and compatibility with older java based implementations is not important. This is the same thing for some of the newer encodings too

Letting users choose between "better/faster/stronger" and "more compatible" I think is very much a good idea

alamb · 2026-04-10T17:59:27Z

+    ///
+    /// The `path_in_schema` field in the Thrift metadata is redundant and wastes a great
+    /// deal of space. Parquet file footers can be made much smaller by omitting this field.
+    /// Because the field was originally a mandatory field, this property defaults to `true`


If we choose to go this way I think it would help to give some more context here on what types of readers would be affected (basically all the parquet-java based readers prior to late 2026)

We could also perhaps provide a link to the discussion: https://lists.apache.org/thread/czm2bk45wwtkhhpqxqvmx9dk5wkwk1kt

alamb · 2026-04-10T18:00:35Z

+    /// Should the writer should emit the `path_in_schema` element of the
+    /// `ColumnMetaData` Thrift struct.


I think it is worth making it more apparently that this will cause incompatibilities with some older readers:

Suggested change

/// Should the writer should emit the `path_in_schema` element of the

/// `ColumnMetaData` Thrift struct.

/// Should the writer should emit the `path_in_schema` element of the

/// `ColumnMetaData` Thrift struct. WARNING: this will make the parquet

/// file unreadable by some older parquet readers. See LINK HERE for details

alamb · 2026-04-10T18:05:09Z

I added a list of related PRs to this issue's description:

alamb · 2026-05-01T12:38:33Z

I wonder if we should just merge this into the Rust implementation of Parquet (with a caveat that this will make metadata smaller / more effiicent at the cost of incompatibility with older / java based readers)?

etseidl · 2026-05-01T13:46:18Z

I'm game. I don't see a reason not to since it's opt in

alamb

Thanks @etseidl -- I will try and review this more carefully shortly. I need to work on FSB stuff for a while first

alamb · 2026-05-01T17:23:25Z

    {
        let buf = TrackedWrite::new(&mut buffer);
-        let writer = ParquetMetaDataWriter::new_with_tracked(buf, &metadata);
+        let writer = ParquetMetaDataWriter::new_with_tracked(buf, &metadata)


Should we perhaps make a new benchmark so we can evaluate the performance both in default mode and wihtout this part of the metadata?

Yes, that's probably worthwhile. I had made this modification for the purposes of the parquet-format submission.

etseidl · 2026-05-01T22:09:10Z

run benchmark metadata

adriangbot · 2026-05-01T22:16:52Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highcpu-48 (12 vCPU / 65 GiB) | Linux bench-c4361872261-1967-zrqkp 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  48
On-line CPU(s) list:                     0-47
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     48
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               3 MiB (48 instances)
L1i cache:                               3 MiB (48 instances)
L2 cache:                                96 MiB (48 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-47
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing deprecate_path_in_schema (feaae15) to f725bc9 (merge-base) diff
BENCH_NAME=metadata
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench metadata
BENCH_FILTER=
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-05-01T22:23:29Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highcpu-48 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  48
On-line CPU(s) list:                     0-47
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     48
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               3 MiB (48 instances)
L1i cache:                               3 MiB (48 instances)
L2 cache:                                96 MiB (48 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-47
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                               deprecate_path_in_schema               main
-----                                               ------------------------               ----
decode metadata (wide) with schema                  1.01     26.8±0.40ms        ? ?/sec    1.00     26.5±0.42ms        ? ?/sec
decode metadata (wide) with skip PES                1.00     26.3±0.37ms        ? ?/sec    1.00     26.2±0.43ms        ? ?/sec
decode metadata (wide) with skip all stats          1.00     28.2±0.31ms        ? ?/sec    1.00     28.3±0.29ms        ? ?/sec
decode metadata (wide) with skip column stats       1.01     27.6±0.38ms        ? ?/sec    1.00     27.3±0.36ms        ? ?/sec
decode metadata (wide) with skip size stats         1.00     30.4±0.37ms        ? ?/sec    1.00     30.4±0.37ms        ? ?/sec
decode metadata (wide) with stats mask              1.01     26.4±0.38ms        ? ?/sec    1.00     26.3±0.32ms        ? ?/sec
decode metadata with schema                         1.00      4.1±0.01µs        ? ?/sec    1.01      4.1±0.01µs        ? ?/sec
decode metadata with skip PES                       1.00      6.9±0.05µs        ? ?/sec    1.01      7.0±0.04µs        ? ?/sec
decode metadata with skip column stats              1.00      6.7±0.02µs        ? ?/sec    1.00      6.8±0.03µs        ? ?/sec
decode metadata with stats mask                     1.00      6.9±0.02µs        ? ?/sec    1.01      7.0±0.04µs        ? ?/sec
decode parquet metadata                             1.00      7.1±0.01µs        ? ?/sec    1.02      7.3±0.04µs        ? ?/sec
decode parquet metadata (wide)                      1.00     28.8±0.37ms        ? ?/sec    1.00     28.7±0.42ms        ? ?/sec
decode parquet metadata no path_in_schema (wide)    1.00     28.1±0.41ms        ? ?/sec  
decode parquet metadata w/ size stats (wide)        1.00     35.7±0.43ms        ? ?/sec    1.00     35.6±0.47ms        ? ?/sec
open(default)                                       1.00      7.7±0.02µs        ? ?/sec    1.02      7.8±0.02µs        ? ?/sec
open(page index)                                    1.00    132.7±0.35µs        ? ?/sec    1.00    132.7±0.37µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	145.0s
Peak memory	4.3 GiB
Avg memory	4.2 GiB
CPU user	141.8s
CPU sys	0.8s
Peak spill	0 B

branch

Metric	Value
Wall time	155.0s
Peak memory	4.3 GiB
Avg memory	4.2 GiB
CPU user	150.9s
CPU sys	0.2s
Peak spill	0 B

File an issue against this benchmark runner

alamb

I think this PR is ready except for:

Don't change the existing writer benchmark (otherwise we are not benchmarking with default settings)
Update the docs a bit more to explain the implications of not writing path_in_schema

github-actions Bot added the parquet Changes to the parquet crate label Apr 8, 2026

etseidl mentioned this pull request Apr 8, 2026

GH-563: Make ColumnMetaData.path_in_schema optional apache/parquet-format#564

Open

alamb reviewed Apr 10, 2026

View reviewed changes

etseidl marked this pull request as ready for review May 1, 2026 13:46

alamb reviewed May 1, 2026

View reviewed changes

etseidl added 2 commits May 1, 2026 11:29

make path_in_schema writing optional

5109900

add warning to docs

1d4eb04

etseidl force-pushed the deprecate_path_in_schema branch from ac323f0 to 1d4eb04 Compare May 1, 2026 18:29

etseidl added 2 commits May 1, 2026 11:45

separate out skipping path_in_schema

c79c2b7

add test

feaae15

alamb reviewed May 3, 2026

View reviewed changes

		/// Should the writer should emit the `path_in_schema` element of the
		/// `ColumnMetaData` Thrift struct.

Conversation

etseidl commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Related PRs

Uh oh!

alamb commented Apr 8, 2026

Uh oh!

jhorstmann commented Apr 10, 2026

Uh oh!

etseidl commented Apr 10, 2026

Uh oh!

alamb commented Apr 10, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

alamb commented Apr 10, 2026

Uh oh!

alamb commented May 1, 2026

Uh oh!

etseidl commented May 1, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb May 1, 2026

Choose a reason for hiding this comment

Uh oh!

etseidl May 1, 2026

Choose a reason for hiding this comment

Uh oh!

etseidl commented May 1, 2026

Uh oh!

adriangbot commented May 1, 2026

Uh oh!

adriangbot commented May 1, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

etseidl commented Apr 8, 2026 •

edited

Loading