[Improve](streaming-job) support specifying offset for StreamingInsertJob create and alter#62490
Open
JNSimba wants to merge 3 commits intoapache:masterfrom
Open
[Improve](streaming-job) support specifying offset for StreamingInsertJob create and alter#62490JNSimba wants to merge 3 commits intoapache:masterfrom
JNSimba wants to merge 3 commits intoapache:masterfrom
Conversation
…tJob create and alter Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
… alter offset test
- CDC ALTER only allows JSON specific offset (reject named modes like initial/latest)
- ALTER offset uses PROPERTIES('offset'='{"file":"xxx","pos":"yyy"}') syntax
- Update regression cases to use PROPERTIES for ALTER
- Add cdc_stream TVF ALTER offset regression test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Member
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FROM MYSQL/POSTGRESpath.DataSourceConfigValidatorto validate offset formats (initial/snapshot/latest/earliest/JSON);earliestis MySQL-only, rejected for PostgreSQL. ImplementJdbcSourceOffsetProvider.deserializeOffsetProperty()for named modes and JSON offset parsing. Remove S3-only restriction inStreamingInsertJob.initInsertJob()andmodifyPropertiesInternal()so CDC jobs can also use offset property. On ALTER, sync offset tosourcePropertiesfor the FROM...TO path.{"lsn":"N"}inPostgresSourceReader.generatePostgresConfig(), and handleSPECIFIC_OFFSETSmode inJdbcIncrementalSourceReader.getStartOffsetFromConfig().JdbcOffset.isValidOffset()andtoSerializedJson()to return meaningful values instead of hardcoded false/null.Test plan
test_streaming_mysql_job_special_offset— earliest/latest/JSON binlog offset with data sync verification, ALTER JOB offset change, invalid format rejectiontest_streaming_postgres_job_special_offset— initial/latest with data sync, ALTER JOB with JSON LSN offset and data sync verification, earliest rejection for PG, invalid format rejectiontest_streaming_mysql_job_special_offset_restart_fe— create job with JSON binlog offset, verify data sync, restart FE, verify job recovery and continued sync🤖 Generated with Claude Code