[PROTOCOL][DISCUSSION] Empty string for string type partition columns

[dengsh12]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (protocol)

Description

@DrakeLin @Jameson-Crate and I found a mismatch between spark and protocol:

According the delta protocol, for string type partition columns, empty strings will be serialized as "" in json commits, and read back as null.

Spark behaves differently: serialized empty string as null. If the partition value in json commits is "", spark read back as "".

Example 1 — write INSERT INTO t VALUES ('a', '') (partition col p = ''):

	partitionValues in JSON commit
Protocol	{"p": ""}
Spark	{"p": null}

Example 2 — read commit already has partitionValues: {"p": ""}, then SELECT p:

	result
Protocol	null
Spark	'' (empty string)

I feel like we have several options:

• Change the protocol to align with spark
  • Not breaking, but lose the ability to distinguish "" vs null
• Change both protocol and spark
  • For string partition columns, just leave it as it was on both read and write
    • Breaking, but provide the ability to distinguish "" vs null

Or maybe other solutions?

How was this patch tested?

Does this PR introduce any user-facing changes?