[Bug][Spark] Report offending commit version in IcebergConverter incremental conversion logging#7030
Open
AnudeepKonaboina wants to merge 1 commit into
Open
Conversation
…n diagnostics (delta-io#7029) #### Which Delta project/connector is this regarding? - [X] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [ ] Other (fill in here) ## Description UniForm performs incremental Iceberg conversion by translating Delta commits one at a time in `IcebergConverter.runIcebergConversionForActions`. The diagnostics emitted from this path reported `targetSnapshot.version` (the latest/head snapshot being synced) instead of `deltaVersion` (the commit actually being processed). As a result, conversion failures were attributed to the table head (often an unrelated operation such as a MERGE) rather than the offending commit, making failures misleading and hard to triage. This is a diagnostics/logging-only change; conversion behavior is unchanged. - `delta.iceberg.conversion.unsupportedActions` event and its `logError`: rename `version` -> `latestVersion` (head) and add `offendingVersion` (the failing commit); rename `commitInfo` -> `operation`. - `delta.iceberg.conversion.convertActions` success event: add `latestVersion` (head), set `version` to the converted commit's `deltaVersion`, and rename `commitInfo` -> `operation`. Resolves delta-io#7029 ## How was this patch tested? Added unit tests in `UniFormConverterSuite`: - `convertActions` success event reports `latestVersion` and the per-commit `version` separately. - `unsupportedActions` failure event reports `offendingVersion` and the renamed keys, triggered via an add-only commit with mixed `dataChange`. ## Does this PR introduce _any_ user-facing changes? No. Only diagnostic event fields and error-log text change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
UniForm performs incremental Iceberg conversion by translating Delta commits one at a time in
IcebergConverter.runIcebergConversionForActions. The diagnostics emitted from this path reportedtargetSnapshot.version(the latest/head snapshot being synced) instead ofdeltaVersion(the commit actually being processed). As a result, conversion failures were attributed to the table head (often an unrelated operation such as a MERGE) rather than the offending commit, making failures misleading and hard to triage.This is a diagnostics/logging-only change; conversion behavior is unchanged.
delta.iceberg.conversion.unsupportedActionsevent and itslogError: renameversion->latestVersion(head) and addoffendingVersion(the failing commit); renamecommitInfo->operation.delta.iceberg.conversion.convertActionssuccess event: addlatestVersion(head), setversionto the converted commit'sdeltaVersion, and renamecommitInfo->operation.Resolves #7029
How was this patch tested?
Added unit tests in
UniFormConverterSuite:convertActionssuccess event reportslatestVersionand the per-commitversionseparately.unsupportedActionsfailure event reportsoffendingVersionand the renamed keys, triggered via an add-only commit with mixeddataChange.Current Error message :*
Error message after the fix:
Does this PR introduce any user-facing changes?
No. Only diagnostic event fields and error-log text change.