Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,13 @@ Changes

Changes:
--------
- Add support for ad-hoc `CWL` workflow execution through the ``POST /jobs`` endpoint using ``multipart/mixed``
or ``multipart/related`` content types. Users can now deploy and execute a `Process` in a single request by
submitting both the `CWL` workflow definition (with ``Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/process-description``)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs adjustment as per other comments.

and execution parameters (with ``Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/execute``)
in a multipart request body. The workflow is automatically deployed (with temporary ``ad-hoc`` tagging) before
execution, eliminating the need for separate deployment and execution steps for one-time workflow runs
(resolves `#834 <https://github.com/crim-ca/weaver/issues/834>`_).
- Adjust the `CLI` to return the ``OperationResult.message`` when the response body is empty (``HTTP 204 No Content``)
to provide a clearer message to the user of what happened and indicate the state of the operation.
Previously, the ``None`` content would be reported as is making it ambiguous about what occurred or whether it failed.
Expand Down
117 changes: 117 additions & 0 deletions docs/source/package.rst
Original file line number Diff line number Diff line change
Expand Up @@ -681,6 +681,123 @@ Example multipart request structure:
- :ref:`proc_ogc_api_multi_cwl` for deployment request examples
- |ogc-api-proc-part2|_ CWL Multipart Content conformance class

.. _app_pkg_adhoc:

Ad-hoc Workflow Execution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Comment on lines +684 to +687

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you move the section under processes.rst.
I think it would make more sense there in between "Execution KVP-encoded" and "Execution Steps".
For consistency, the title could be "Execution of Ad-hoc Workflow"


`Weaver` supports *ad-hoc* workflow execution, allowing users to deploy and execute a :term:`CWL` workflow in a single

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add this:

# in references.rst (near corresponding definitions)

.. |ogc-api-proc-part3-cwl-ad-hoc-exec| replace:: :term:`CWL` *ad-hoc* workflow execution
.. _ogc-api-proc-part3-cwl-ad-hoc-exec: https://docs.ogc.org/DRAFTS/21-009.html#section_cwl_workflows

Then, reword as

`Weaver` supports |ogc-api-proc-part3-cwl-ad-hoc-exec|_ from the |ogc-api-proc-part3|_ standard, which allows ...

request to the ``POST /jobs`` endpoint using ``multipart/mixed`` or ``multipart/related`` content types. This
eliminates the need for separate :ref:`deployment <proc_op_deploy>` and :ref:`execution <proc_op_execute>` steps
when running one-time workflows or testing workflow definitions.

The multipart request must contain:

1. One or more :term:`CWL` parts (workflow and any dependent tools) with
``Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/process-description``
Comment on lines +696 to +697

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/process-description header would apply to cases where additional process metadata is provided (i.e.: #990)

Here, the CWL parts just need to have the relevant application/cwl[+json|yaml].

Therefore, there could optionally be other "Process Description" parts using the indicated Content-Profile.

Consider adding a reference to proc_ogc_api_multi_cwl section by indicating that it follows the same concepts and procedure defined there.

2. One execution request part with ``Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/execute``

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure to update the URIs with HTTPS.

containing the job inputs and execution parameters

When an ad-hoc execution request is received:

- The workflow (and any dependent tools) are automatically deployed with ``ad-hoc`` tagging
- The job is immediately submitted for execution using the provided inputs
- The workflow remains deployed after execution and can be reused or cleaned up later

Example Ad-hoc Execution Request
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: http

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider moving this into https://github.com/crim-ca/weaver/tree/master/weaver/wps_restapi/examples and use a literalinclude. That will be useful for reuse in swagger definitions.


POST /jobs HTTP/1.1
Host: weaver.example.com
Content-Type: multipart/mixed; boundary="adhoc-boundary"

--adhoc-boundary
Content-Type: application/cwl+json
Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/process-description

Comment on lines +718 to +719

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be removed since that is not the corresponding profile for the CWL part contents.

If the code already supports dealing with multiple application/json of different Content-Profile, you can move this under a separte section like:

--adhoc-boundary
Content-Type: application/json
Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/process-description

{
  "id": "adhoc-workflow",
  "description": "Example ad-hoc workflow deployed and executed inline."
}

{
"cwlVersion": "v1.2",
"class": "CommandLineTool",
"id": "adhoc-echo-tool",
"baseCommand": ["echo"],
"inputs": {
"message": "string"
},
"outputs": {
"output": {
"type": "stdout"
}
},
"requirements": {
"DockerRequirement": {
"dockerPull": "alpine:latest"
}
},
"stdout": "output.txt"
}

--adhoc-boundary
Content-Type: application/cwl+json
Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/process-description

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

idem


{
"cwlVersion": "v1.2",
"class": "Workflow",
"id": "adhoc-workflow",
"inputs": {
"message": "string"
},
"outputs": {
"result": {
"type": "File",
"outputSource": "echo_step/output"
}
},
"steps": {
"echo_step": {
"run": "adhoc-echo-tool",
"in": {
"message": "message"
},
"out": ["output"]
}
}
}

--adhoc-boundary
Content-Type: application/json
Content-Profile: http://www.opengis.net/def/ogcapi-processes/2.0/execute

{
"inputs": {
"message": "Hello from ad-hoc workflow!"
},
"outputs": {
"result": {
"transmissionMode": "reference"
}
},
"mode": "async",
"response": "document"
}

--adhoc-boundary--

The response will be a standard :term:`Job` status document (see :ref:`proc_op_execute`) with the ``jobID``,
``processID`` (the deployed workflow ID), and execution status. The job can be monitored and results retrieved
using the standard :term:`Job` endpoints.

.. note::
The deployed workflow and tools remain available after ad-hoc execution and are tagged with ``ad-hoc`` for
identification. They can be cleaned up using the :ref:`Undeploy <proc_op_undeploy>` operation if no longer needed.

.. seealso::
- :ref:`proc_op_execute` for execution request details
- :ref:`app_pkg_multipart` for multipart :term:`CWL` packaging
- `crim-ca/weaver#834 <https://github.com/crim-ca/weaver/issues/834>`_ for implementation details

.. _app_pkg_workflow:

CWL Workflow
Expand Down
Loading
Loading