Repository navigation
Fixes some bugs with GroupIntoBatches. #40754
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 8 commits
cb5d427
b2bcbd7
3f026b8
6e008e5
0fd6398
bab7761
f6660d9
dd1527b
ca981ea
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -805,4 +805,49 @@ private static <K> boolean checkBatchByteSizes( | |
| } | ||
| return true; | ||
| } | ||
|
|
||
| /** | ||
| * Tests that when elements arrive out of order, the watermark hold is updated to the minimum | ||
| * timestamp. With ofSize(3) and 2 elements, elements remain buffered until window expiration, | ||
| * exercising the updated timer watermark hold rather than flushing immediately in processElement. | ||
| */ | ||
| @Test | ||
| @Category({ | ||
| ValidatesRunner.class, | ||
| NeedsRunner.class, | ||
| UsesTimersInParDo.class, | ||
| UsesStatefulParDo.class, | ||
| UsesTestStream.class, | ||
| UsesOnWindowExpiration.class | ||
| }) | ||
| public void testOutOfOrderTimestampsInBatch() { | ||
| Instant baseTs = new Instant(0L); | ||
| TestStream<KV<String, String>> stream = | ||
| TestStream.create(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of())) | ||
| .advanceWatermarkTo(baseTs) | ||
| .addElements( | ||
| TimestampedValue.of( | ||
| KV.of("key", "val1"), baseTs.plus(Duration.standardSeconds(10)))) | ||
| .addElements( | ||
| TimestampedValue.of(KV.of("key", "val2"), baseTs.plus(Duration.standardSeconds(5)))) | ||
| .advanceWatermarkToInfinity(); | ||
|
|
||
| PCollection<Long> outputTimestamps = | ||
| pipeline | ||
| .apply(stream) | ||
| .apply(GroupIntoBatches.ofSize(3)) | ||
| .apply( | ||
| ParDo.of( | ||
| new DoFn<KV<String, Iterable<String>>, Long>() { | ||
| @ProcessElement | ||
| public void processElement( | ||
| @Timestamp Instant ts, OutputReceiver<Long> receiver) { | ||
| receiver.output(ts.getMillis()); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the fix looks good to me. Is the test failing without the fix? It seems like receiver.output will get fused with the GroupIntoBatches timer and i'm not sure if the elements will be dropped as late.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The elements were getting output with the wrong timestamps. Let me modify the test so that there is something like a GBK afterwards and we can directly verify that the elements get dropped due to an incorrect watermark hold. In the new example, the batch gets emitted with timestamp 2s, but the watermark hold was set to 10s. |
||
| } | ||
| })); | ||
|
|
||
| PAssert.that(outputTimestamps) | ||
| .containsInAnyOrder(baseTs.plus(Duration.standardSeconds(5)).getMillis()); | ||
| pipeline.run(); | ||
| } | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: i like
currentMinBufferedTs != oldOutputTs, set a timer when MinBufferedTs is modified is easier to read than whencurrentMinBufferedTs < oldOutputTs. they both are true when we need a new timer.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we had a bug where somehow
currentMinBufferedTs > oldOutputTsby accident, maybe it is better to have the bound instead of depending on<being equivalent to!=due to context?