Abort execution when platform telemetry error

[jorgee]

This pull request introduces a new mechanism to control error handling behavior in the TowerClient class by adding an abortOnError flag, which can be set via the environment variable TOWER_ABORT_ON_ERROR. When enabled, critical errors encountered while communicating with Seqera Platform will cause the workflow to abort immediately using the AbortRunException. The changes also include improved error propagation and additional tests to verify this behavior.

Error Handling Improvements:

• Added abortOnError flag to TowerClient, defaulting to true, and made it configurable via the TOWER_ABORT_ON_ERROR environment variable. This determines whether critical errors abort the workflow or are handled as warnings. [1] [2]
• Updated error handling in TowerClient methods (logHttpResponse, parseTowerResponse, and others) to throw AbortRunException when abortOnError is enabled, ensuring immediate workflow termination on critical errors. [1] [2] [3] [4] [5]

Session and Exception Propagation:

• Modified the Session class to specifically catch and log AbortRunException during observer notification, ensuring these exceptions propagate and abort the workflow as intended. [1] [2]

Tests:

• Added new tests in TowerClientTest to verify the correct detection of the abortOnError setting and to ensure that the workflow aborts as expected when errors occur and abortOnError is enabled. [1] [2]

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	1d93537
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/69e898a387e3aa0008ebb672

[jorgee]

I have updated this branch to master.
If I am not wrong, in one of our conversations, we agreed to just fail for platform cases instead of all exceptions. This PR implements this behaviour and allows to work also as before using an environment variable. I will close #6823 in favor of this one.

[bentsherman]

@pditommaso let me know if the latest changes look good to you

I like the general principle of "plugin observer errors are logged as warnings by default, observer can throw AbortRunException to fail the run"

This PR currently just adds an env var to control whether the TowerClient throws hard/soft errors. But I wonder if we should just decide for each error case whether it should be hard or soft instead of introducing an environment var

For example, if I run with -with-tower but fail to authenticate, maybe that should just always be a hard error

In any case, let's wait until #6946 is merged since it refactors the tower client (harder to resolve merge conflicts there)

[bentsherman]

From today's discussion -- @jorgee please update this PR making all failures related to sending data to Platform hard failures, removing the need for the environment variable.

[jorgee]

I have removed the environment variable and throw AbortRunException for all calls except heartbeats and onFlowComplete.

• For onFlowComplete, it is the last message and execution will be finished anyway.
• I have some doubts about the heartbeats case. Failing at the first exception could be very strict. Therefore, I have considered that a task complete message will be sent at somepoint and it will send the AbortRunException if there is a consistent connection problem. Is it fine? should I throw an AbortRunException at first error?

[bentsherman]

For the heartbeats, I think keeping them as warnings is reasonable

For the onFlowComplete, if it fails, wouldn't that leave the platform run in an indeterminate state? Or does platform still mark the run as completed?

[jorgee]

For the onFlowComplete, if it fails, wouldn't that leave the platform run in an indeterminate state? Or does platform still mark the run as completed?

I' ll try to check what it does in this case. Anyway, it will not receive the trace event. The only possibility is if it monitors the head job and use the process exit code to decide if failed or not. In that case, I think it is better to warn instead of throwing the AbortRunException

[jorgee]

Confirmed! I have run an execution skipping the trace completed event. The workflow state at platform is not undetermined. As head-job finishes, the workflow execution is considered completed, but marked as FAILED.

Last updates:

• I have updated the branch with master
• Fixed an issue introduced in TowerClient refactor where begin trace was failing but not detected by tests because the exception was not aborting the execution (commit 1d93537)