[Spark][kernel] Translate Kernel exceptions mentioned in Issue #5900#7071
[Spark][kernel] Translate Kernel exceptions mentioned in Issue #5900#7071sotikoug83 wants to merge 2 commits into
Conversation
sotikoug83
requested review from
huan233usc,
murali-db,
raveeram-db,
tdas and
timothyw553
as code owners
| } | ||
|
|
||
| /** Returns whether the table's {@code _delta_log} directory exists. */ | ||
| private boolean deltaLogDirExists() { |
There was a problem hiding this comment.
What's kernel's behavior when there is no log file?
There was a problem hiding this comment.
Kernel throws TableNotFoundException
huan233usc
left a comment
huan233usc
left a comment
There was a problem hiding this comment.
Left one comment on the protocol-version exception translation.
| p.getVersionType() == UnsupportedProtocolVersionException.ProtocolVersionType.READER; | ||
| throw (RuntimeException) | ||
| DeltaErrors.invalidProtocolVersionError( | ||
| p.getTablePath(), isReader ? p.getVersion() : 0, isReader ? 0 : p.getVersion()); |
There was a problem hiding this comment.
This loses part of the table protocol when translating to Delta's InvalidProtocolVersionException. DSv1 builds that exception with both tableProtocol.minReaderVersion and tableProtocol.minWriterVersion; here a reader-side Kernel error reports writerRequired=0 in the DELTA_INVALID_PROTOCOL_VERSION message, which is not a valid table requirement and is still not DSv1-equivalent. Can we preserve the full Protocol when doing the translation, or use a factory/error shape that does not invent the missing side?
There was a problem hiding this comment.
Fixed, UnsupportedProtocolVersionException now carries the full protocol.
2 participants
Which Delta project/connector is this regarding?
Description
Resolves #5900
(Scoped down from #7025)
When using DSv2 connector, two streaming reads surface Kernel-native exceptions instead of the Spark/Delta exceptions the DSv1 path raises. This PR translates them at their surfacing points to be consistent with DSv1:
_delta_logdirectory exists but contains no commits now throwsorg.apache.spark.sql.AnalysisException(DELTA_SCHEMA_NOT_SET) instead of Kernel'sTableNotFoundException.org.apache.spark.sql.delta.InvalidProtocolVersionExceptioninstead of Kernel'sUnsupportedProtocolVersionException.The translations are inline as such:
DeltaErrorsgainsthrowSchemaNotSet()and aninvalidProtocolVersionError(...)factory.DeltaV2TabletranslatesTableNotFoundExceptionon initial snapshot load.SparkMicroBatchStreamtranslatesUnsupportedProtocolVersionExceptioninlatestOffset.How was this patch tested?
Enabled the two previously expected-to-fail tests in
DeltaV2SourceSuite:no schema should throw an exceptionDelta sources should verify the protocol reader versionAll test runs still passed after.
Does this PR introduce any user-facing changes?
No