Repository navigation
Conversation
… store or scheduler failure Starting a child from inside a workflow method creates the child's instance and schedules its task on the parent's execution thread. A store or scheduler failure there (for example a task insert timing out) escaped the parent's workflow method as an unclassified exception, which the executor treats as a bug in workflow code, so the parent went to ERROR and could not be re-executed. SkipperEngine.startWorkflow now reports infrastructure failures of its store and scheduler calls as InternalError, which the task handler retries. Errors the store or scheduler already classified, and requests rejected as invalid (IllegalArgumentException, which also covers input that cannot be serialized), are rethrown unchanged and still fail the caller.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a workflow method starts a child (
workflowFactory.builder(...).build(), then calling the child's@WorkflowMethod), the child's instance is created and its task is scheduled on the parent's execution thread. If the store or scheduler fails transiently there, for example a task insert timing out withjava.util.concurrent.TimeoutException, the raw exception escapes the parent's workflow method.WorkflowExecutor.wrapUnexpectedErrortreats any unclassified exception as a bug in workflow code and turns it into aNonRetryableError. The parent then goes toERROR, which is terminal, andreExecuteWorkflowswon't re-run it. A parent that fans out to many children is very likely to hit one such blip.Change
SkipperEngine.startWorkflownow runs each of its store and scheduler calls through a small helper,startWorkflowIo. The calls are: create the instance, read an existing instance on a duplicate id, schedule the workflow task, and schedule the timeout task. The helper reports infrastructure failures asInternalError. That is the type the built-in JDBC stores and schedulers already throw for SQL failures, and the task handler already retries it as a transient execution failure. The parent's task is kept and re-run aftertaskUnexpectedErrorRetryDelay. On retry, children that were already created are found by id and scheduled again, which also recovers a child whose instance committed but whose task insert failed.These pass through unchanged, so real programming errors still fail the caller:
SkipperError,InternalError, andEntityAlreadyExists, which the duplicate-id path still handlesIllegalArgumentException: requests rejected as invalid, including the serde's "cannot serialize" errorsErrors such as linkage or class-loading failures, which the helper doesn't catchstartWorkflowrunsTrade-offs
parentWorkflowIdis optional, and suspend and async workflows change threads. The engine is the one place that knows which calls are store and scheduler I/O.InternalError(cause = ...)instead of the raw exception from a custom store or scheduler. With the built-in stores they already gotInternalError, so the behavior is now the same whichever store is plugged in. Java callers could not catch a sneaky-thrown checkedTimeoutExceptionby type anyway.InternalErrorvsRetryableError.InternalErroruses the operator-configured retry delay for unexpected errors and adds no new retry policy.Not covered
createExistingWorkflowIsNoop(or that feature gate is on), the retried start returns the existing instance without scheduling it. This gap existed before this change.Testing
New
ChildWorkflowStartFailureTest. A parent starts three children (detached,runAsync), and the store or scheduler is wrapped to fail the first call for the second child:TimeoutExceptioninserting the child's taskERROR(test fails)COMPLETED, all 3 childrenCOMPLETEDTimeoutExceptioncreating the childERROR(test fails)COMPLETED, all 3 childrenCOMPLETEDIllegalArgumentExceptioncreating the childERRORERROR, not retriedBefore the fix, both timeout cases failed with
a transient child-start failure must not fail the parent ==> expected: <COMPLETED> but was: <ERROR>../gradlew spotlessCheckpasses. The root./gradlew :checkpasses locally except the MySQL suites (MySqlSchedulerTest,MySqlWorkflowStoreTest,MySqlClusterMembershipManagerTest), which fail in setup because MariaDB4j can't install its embedded database on this machine; they are left to CI.