[Spark] support identity creation via sparksql

[mwc360]

Resolves #7061

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

Adds support for the Spark 4.0 SQL DDL identity-column syntax in Delta, e.g.:

CREATE TABLE t (
   id1 BIGINT GENERATED ALWAYS AS IDENTITY,
   id2 BIGINT GENERATED ALWAYS AS IDENTITY (START WITH -1 INCREMENT BY 1),
   id3 BIGINT GENERATED BY DEFAULT AS IDENTITY,
   id4 BIGINT GENERATED BY DEFAULT AS IDENTITY (START WITH -1 INCREMENT BY 1)
)

Previously the syntax parsed but Spark refused to dispatch to the catalog (capability not advertised), and even if it did, Spark's V2 Column.identityColumnSpec() was being dropped by the default CatalogV2Util.v2ColumnsToStructType conversion before reaching Delta's create path, so the resulting tables had no identity metadata.

How it works

Spark 4.0 stores identity info on the V2  Column  object ( Column.identityColumnSpec() ), not in StructField metadata. The default fallback in StagingTableCatalog converts Column[] → StructType via CatalogV2Util.v2ColumnsToStructType, which drops IdentityColumnSpec. By overriding the Column[] overloads in AbstractDeltaCatalog and using a Delta-aware converter, the identity info is preserved into Delta's metadata keys before it reaches createDeltaTable . From there, the existing identity codepath ( ColumnWithDefaultExprUtils.isIdentityColumn, IdentityColumnsTableFeature auto-enable, write-time generation, admission checks) takes over unchanged.

How was this patch tested?

A new test suite plus regression testing of existing identity suites.

Does this PR introduce any user-facing changes?

No, new functionality.

[felipepessoto]

@newfront @timothyw553 could you trigger CI and help find a reviewer, please? This solves an important issue, part of the roadmap: Linux Foundation Delta Lake Roadmap (view)

[timothyw553]

hi @mwc360 the CI is failing, could you take a look?

[mwc360]

@timothyw553 - scalastyle issue is fixed, can you retrigger CI?

[mwc360]

@timothyw553 - sorry, missed something. Just tested that it compiles successfully. pls trigger CI again.

[timothyw553]

triggered

[mwc360]

@timothyw553 - fyi, everything is passing