Skip to content

[Bug][Spark] Report offending commit version in IcebergConverter incremental conversion logging#7030

Open
AnudeepKonaboina wants to merge 1 commit into
delta-io:masterfrom
AnudeepKonaboina:bug/issue-7029
Open

[Bug][Spark] Report offending commit version in IcebergConverter incremental conversion logging#7030
AnudeepKonaboina wants to merge 1 commit into
delta-io:masterfrom
AnudeepKonaboina:bug/issue-7029

Conversation

@AnudeepKonaboina

Copy link
Copy Markdown
Collaborator

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

UniForm performs incremental Iceberg conversion by translating Delta commits one at a time in IcebergConverter.runIcebergConversionForActions. The diagnostics emitted from this path reported targetSnapshot.version (the latest/head snapshot being synced) instead of deltaVersion (the commit actually being processed). As a result, conversion failures were attributed to the table head (often an unrelated operation such as a MERGE) rather than the offending commit, making failures misleading and hard to triage.

This is a diagnostics/logging-only change; conversion behavior is unchanged.

  • delta.iceberg.conversion.unsupportedActions event and its logError: rename version -> latestVersion (head) and add offendingVersion (the failing commit); rename commitInfo -> operation.
  • delta.iceberg.conversion.convertActions success event: add latestVersion (head), set version to the converted commit's deltaVersion, and rename commitInfo -> operation.

Resolves #7029

How was this patch tested?

Added unit tests in UniFormConverterSuite:

  • convertActions success event reports latestVersion and the per-commit version separately.
  • unsupportedActions failure event reports offendingVersion and the renamed keys, triggered via an add-only commit with mixed dataChange.

Current Error message :*

26/06/13 02:00:10 ERROR IcebergConverterEdge: Unsupported combination of actions for incremental conversion. Context:
version -> 62111,
commitInfo -> COMPUTE STATS,
hasAdd -> true,
hasRemove -> false,
dataChange -> Some, 
hasDv -> true

Error message after the fix:

26/06/13 02:00:10 ERROR IcebergConverterEdge: Unsupported combination of actions for incremental conversion. Context:
latestVersion -> 62111,
offendingVersion -> 62103,
operation -> COMPUTE STATS,
hasAdd -> true,
hasRemove -> false,
dataChange -> Some,
hasDv -> true

Does this PR introduce any user-facing changes?

No. Only diagnostic event fields and error-log text change.

…n diagnostics (delta-io#7029)

#### Which Delta project/connector is this regarding?

- [X] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description

UniForm performs incremental Iceberg conversion by translating Delta commits one
at a time in `IcebergConverter.runIcebergConversionForActions`. The diagnostics
emitted from this path reported `targetSnapshot.version` (the latest/head
snapshot being synced) instead of `deltaVersion` (the commit actually being
processed). As a result, conversion failures were attributed to the table head
(often an unrelated operation such as a MERGE) rather than the offending commit,
making failures misleading and hard to triage.

This is a diagnostics/logging-only change; conversion behavior is unchanged.

- `delta.iceberg.conversion.unsupportedActions` event and its `logError`: rename
  `version` -> `latestVersion` (head) and add `offendingVersion` (the failing
  commit); rename `commitInfo` -> `operation`.
- `delta.iceberg.conversion.convertActions` success event: add `latestVersion`
  (head), set `version` to the converted commit's `deltaVersion`, and rename
  `commitInfo` -> `operation`.

Resolves delta-io#7029

## How was this patch tested?

Added unit tests in `UniFormConverterSuite`:
- `convertActions` success event reports `latestVersion` and the per-commit
  `version` separately.
- `unsupportedActions` failure event reports `offendingVersion` and the renamed
  keys, triggered via an add-only commit with mixed `dataChange`.

## Does this PR introduce _any_ user-facing changes?

No. Only diagnostic event fields and error-log text change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG][Spark] UniForm Iceberg incremental conversion diagnostics report the latest snapshot version instead of the offending commit version

1 participant