-
Notifications
You must be signed in to change notification settings - Fork 6
Support ad-hoc CWL workflows #997
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
f1d240b
3ff5288
31656b0
f763ab3
1a2d1d1
cc9020a
17f5750
255d92e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -681,6 +681,123 @@ Example multipart request structure: | |
| - :ref:`proc_ogc_api_multi_cwl` for deployment request examples | ||
| - |ogc-api-proc-part2|_ CWL Multipart Content conformance class | ||
|
|
||
| .. _app_pkg_adhoc: | ||
|
|
||
| Ad-hoc Workflow Execution | ||
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ||
|
Comment on lines
+684
to
+687
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you move the section under |
||
|
|
||
| `Weaver` supports *ad-hoc* workflow execution, allowing users to deploy and execute a :term:`CWL` workflow in a single | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please add this: # in references.rst (near corresponding definitions)
.. |ogc-api-proc-part3-cwl-ad-hoc-exec| replace:: :term:`CWL` *ad-hoc* workflow execution
.. _ogc-api-proc-part3-cwl-ad-hoc-exec: https://docs.ogc.org/DRAFTS/21-009.html#section_cwl_workflowsThen, reword as `Weaver` supports |ogc-api-proc-part3-cwl-ad-hoc-exec|_ from the |ogc-api-proc-part3|_ standard, which allows ... |
||
| request to the ``POST /jobs`` endpoint using ``multipart/mixed`` or ``multipart/related`` content types. This | ||
| eliminates the need for separate :ref:`deployment <proc_op_deploy>` and :ref:`execution <proc_op_execute>` steps | ||
| when running one-time workflows or testing workflow definitions. | ||
|
|
||
| The multipart request must contain: | ||
|
|
||
| 1. One or more :term:`CWL` parts (workflow and any dependent tools) with | ||
| ``Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/process-description`` | ||
|
Comment on lines
+696
to
+697
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The Here, the CWL parts just need to have the relevant Therefore, there could optionally be other "Process Description" parts using the indicated Consider adding a reference to |
||
| 2. One execution request part with ``Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/execute`` | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Make sure to update the URIs with HTTPS. |
||
| containing the job inputs and execution parameters | ||
|
|
||
| When an ad-hoc execution request is received: | ||
|
|
||
| - The workflow (and any dependent tools) are automatically deployed with ``ad-hoc`` tagging | ||
| - The job is immediately submitted for execution using the provided inputs | ||
| - The workflow remains deployed after execution and can be reused or cleaned up later | ||
|
|
||
| Example Ad-hoc Execution Request | ||
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| .. code-block:: http | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Consider moving this into https://github.com/crim-ca/weaver/tree/master/weaver/wps_restapi/examples and use a |
||
|
|
||
| POST /jobs HTTP/1.1 | ||
| Host: weaver.example.com | ||
| Content-Type: multipart/mixed; boundary="adhoc-boundary" | ||
|
|
||
| --adhoc-boundary | ||
| Content-Type: application/cwl+json | ||
| Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/process-description | ||
|
|
||
|
Comment on lines
+718
to
+719
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To be removed since that is not the corresponding profile for the CWL part contents. If the code already supports dealing with multiple --adhoc-boundary
Content-Type: application/json
Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/process-description
{
"id": "adhoc-workflow",
"description": "Example ad-hoc workflow deployed and executed inline."
} |
||
| { | ||
| "cwlVersion": "v1.2", | ||
| "class": "CommandLineTool", | ||
| "id": "adhoc-echo-tool", | ||
| "baseCommand": ["echo"], | ||
| "inputs": { | ||
| "message": "string" | ||
| }, | ||
| "outputs": { | ||
| "output": { | ||
| "type": "stdout" | ||
| } | ||
| }, | ||
| "requirements": { | ||
| "DockerRequirement": { | ||
| "dockerPull": "alpine:latest" | ||
| } | ||
| }, | ||
| "stdout": "output.txt" | ||
| } | ||
|
|
||
| --adhoc-boundary | ||
| Content-Type: application/cwl+json | ||
| Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/process-description | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. idem |
||
|
|
||
| { | ||
| "cwlVersion": "v1.2", | ||
| "class": "Workflow", | ||
| "id": "adhoc-workflow", | ||
| "inputs": { | ||
| "message": "string" | ||
| }, | ||
| "outputs": { | ||
| "result": { | ||
| "type": "File", | ||
| "outputSource": "echo_step/output" | ||
| } | ||
| }, | ||
| "steps": { | ||
| "echo_step": { | ||
| "run": "adhoc-echo-tool", | ||
| "in": { | ||
| "message": "message" | ||
| }, | ||
| "out": ["output"] | ||
| } | ||
| } | ||
| } | ||
|
|
||
| --adhoc-boundary | ||
| Content-Type: application/json | ||
| Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/execute | ||
|
|
||
| { | ||
| "inputs": { | ||
| "message": "Hello from ad-hoc workflow!" | ||
| }, | ||
| "outputs": { | ||
| "result": { | ||
| "transmissionMode": "reference" | ||
| } | ||
| }, | ||
| "mode": "async", | ||
| "response": "document" | ||
| } | ||
|
|
||
| --adhoc-boundary-- | ||
|
|
||
| The response will be a standard :term:`Job` status document (see :ref:`proc_op_execute`) with the ``jobID``, | ||
| ``processID`` (the deployed workflow ID), and execution status. The job can be monitored and results retrieved | ||
| using the standard :term:`Job` endpoints. | ||
|
|
||
| .. note:: | ||
| The deployed workflow and tools remain available after ad-hoc execution and are tagged with ``ad-hoc`` for | ||
| identification. They can be cleaned up using the :ref:`Undeploy <proc_op_undeploy>` operation if no longer needed. | ||
|
|
||
| .. seealso:: | ||
| - :ref:`proc_op_execute` for execution request details | ||
| - :ref:`app_pkg_multipart` for multipart :term:`CWL` packaging | ||
| - `crim-ca/weaver#834 <https://github.com/crim-ca/weaver/issues/834>`_ for implementation details | ||
|
|
||
| .. _app_pkg_workflow: | ||
|
|
||
| CWL Workflow | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs adjustment as per other comments.