Add opt-in Modal app cleanup after sandbox completion by raza-khan0108 · Pull Request #4993 · zenml-io/zenml

raza-khan0108 · 2026-06-24T18:08:42Z

Describe changes

After a step sandbox finishes, the Modal app entry remains in the Modal
dashboard as deployed, Tasks: 0. This is purely visual clutter — idle
apps incur no compute charges — but accumulates quickly for users running
many ZenML pipelines or steps on Modal (each step run creates one app
named zenml-<step-run-id>-<step-name>).

This PR adds a new stop_app_after_run: bool = False setting to
ModalSettingsMixin (shared by both the step operator and orchestrator
flavors). When enabled, ZenML overrides cleanup_step_submission() — the
hook introduced in #4990 — to call modal app stop --yes <app-name> via
subprocess after the sandbox reaches a terminal state. The app entry moves
from deployed → stopped in the Modal dashboard. Stopped apps are
still visible in the "recently stopped" section with their full log history
intact, so debugging is unaffected.

Why subprocess instead of the Python SDK?

modal.App has no public .stop() method. The Modal CLI is the only
supported programmatic path for stopping a deployed app. We call it via
subprocess.run() and inject MODAL_TOKEN_ID / MODAL_TOKEN_SECRET from
ZenML's configured credentials so the call works regardless of ambient auth
state.

Changes

modal_base_flavor.py: add stop_app_after_run: bool = False to
ModalSettingsMixin
sandbox_utils.py: add stop_modal_app() helper that runs the Modal
CLI and forwards token credentials
modal_step_operator.py: override cleanup_step_submission() (from
Add isolated step cleanup hooks #4990) — early-returns when the setting is False, otherwise calls
stop_modal_app() with warning-only error handling so cleanup can never
mask the step result

Usage

@step(
    step_operator="my-modal-operator",
    settings={
        "step_operator": ModalStepOperatorSettings(stop_app_after_run=True)
    },
)
def my_step() -> None: ...

CLAassistant · 2026-06-24T18:08:53Z

All committers have signed the CLA.

Copilot

Pull request overview

Adds an opt-in cleanup mechanism for the Modal step operator to stop per-step Modal apps after a sandbox finishes, reducing “deployed / Tasks: 0” dashboard clutter without changing default behavior.

Changes:

Introduces stop_app_after_run: bool = False in ModalStepOperatorSettings.
Adds a sandbox_utils.stop_modal_app(...) helper that invokes modal app stop via subprocess (with env-based auth forwarding).
Hooks cleanup into ModalStepOperator.get_status() (on terminal state) and cancel().

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
`src/zenml/integrations/modal/step_operators/modal_step_operator.py`	Triggers opt-in stop of the step-scoped Modal app on completion/cancelation.
`src/zenml/integrations/modal/sandbox_utils.py`	Adds CLI-based helper to stop a deployed Modal app by name.
`src/zenml/integrations/modal/flavors/modal_step_operator_flavor.py`	Adds the `stop_app_after_run` user-facing settings field and description.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        if status.is_finished:
+            settings = cast(
+                ModalStepOperatorSettings, self.get_settings(step_run)
+            )
+            if settings.stop_app_after_run:
+                self._stop_step_app(step_run, settings)
+        return status


+    stop_app_after_run: bool = Field(
+        False,
+        description="Stop the per-step Modal app after the sandbox reaches a terminal "
+        "state. When True, ZenML calls `modal app stop` on the app that was "
+        "created for this step run, removing it from the Modal dashboard's "
+        "deployed-apps list. Stopped apps are still visible in the 'recently "
+        "stopped' section and their logs remain accessible. Defaults to False "
+        "to preserve the previous behavior.",
+    )


chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5f986c5c88

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-24T18:18:59Z

+    """
+    import subprocess
+
+    cmd = ["modal", "app", "stop", "--yes", app_name]


Handle Modal 1.4.0/1.4.1 before passing --yes

This hard-codes --yes, but the Modal integration still declares support for modal>=1.4,<2 in src/zenml/integrations/modal/__init__.py; Modal's official changelog says the confirmation prompt and --yes skip flag were added in 1.4.2 (https://modal.com/docs/sdk/py/changelog#142-2026-04-16). When users are on the supported 1.4.0 or 1.4.1 releases, the cleanup command will reject the unknown option and stop_app_after_run=True will only log a warning while leaving the app deployed, so either the requirement needs to be raised or the flag needs to be conditional.

Useful? React with 👍 / 👎.

schustmi · 2026-06-25T07:12:26Z

@raza-khan0108 A few things please:

sign the CLA
rebase on develop and use it as a base for your branch
We'll merge Add isolated step cleanup hooks #4990 zero, which introduces a method specifically for such cleanup, which means we don't have to abuse the get_status method, which is both ugly as well as unreliable. Once this is merged, I'd ask you to update your PR to use the cleanup method to delete the app.

schustmi · 2026-06-25T08:24:32Z

#4990 is merged now, so as soon as you rebase on develop you can use it.

After a step sandbox finishes, the Modal app entry remains in the dashboard as 'deployed, Tasks: 0'. This is purely visual clutter— idle apps incur no compute charges—but accumulates quickly for users running many ZenML pipelines or steps on Modal. Add a new stop_app_after_run: bool = False setting to ModalSettingsMixin (shared by both the step operator and orchestrator). When True, ZenML overrides cleanup_step_submission() to call modal app stop --yes on the per-step app, moving its entry from 'deployed' to 'stopped' in the Modal dashboard. Stopped apps remain visible in the 'recently stopped' section with their full log history intact, so debugging is unaffected. Changes: - modal_base_flavor.py: add stop_app_after_run to ModalSettingsMixin - sandbox_utils.py: add stop_modal_app() helper that delegates to the Modal CLI and forwards token credentials from ZenML config - modal_step_operator.py: override cleanup_step_submission() (the hook added in zenml-io#4990) to call stop_modal_app() when the setting is enabled; failures are logged as warnings and never re-raised Fixes zenml-io#4889

raza-khan0108 · 2026-06-25T18:44:12Z

Hi @schustmi — addressed all the feedback:

✅ Signed the CLA
✅ Rebased onto develop and changed the base branch
✅ Replaced the get_status() hook with cleanup_step_submission() from Add isolated step cleanup hooks #4990
✅ Moved stop_app_after_run into ModalSettingsMixin in modal_base_flavor.py so it's shared by both the step operator and orchestrator

The PR description has been updated to reflect the new implementation. Let me know if anything else needs changing!

schustmi · 2026-07-02T13:30:21Z

+        env[MODAL_TOKEN_ID_ENV_KEY] = token_pair[0]
+        env[MODAL_TOKEN_SECRET_ENV_KEY] = token_pair[1]
+
+    result = subprocess.run(cmd, capture_output=True, text=True, env=env)


Is it not possible to do this via the modal python sdk?

I looked into using the Modal Python SDK first, but there doesn’t appear to be a public API to stop an already deployed app programmatically (e.g., no App.stop() equivalent). The supported path today is modal app stop, so this PR uses the CLI via subprocess.run().

I also made sure the call is non-blocking for pipeline outcome: cleanup is best-effort (warning-only on failure), and credentials are passed explicitly via MODAL_TOKEN_ID / MODAL_TOKEN_SECRET from the configured ZenML integration so it doesn’t depend on ambient auth state.

If Modal adds a public SDK method for this, I’m happy to switch this implementation to that.

schustmi · 2026-07-15T12:09:12Z

+        RuntimeError: If the ``modal app stop`` command exits with a
+            non-zero status code.
+    """
+    import subprocess


Import at the top

schustmi · 2026-07-15T12:09:54Z

+    Moves the app from 'deployed' to 'stopped' in the Modal dashboard,
+    eliminating the visual clutter of idle ``Tasks: 0`` app entries.
+    Stopped apps remain visible in the 'recently stopped' section with
+    their logs intact, so debugging is not affected.
+
+    This should only be called after the associated Modal sandbox has
+    reached a terminal state, because stopping the app does not wait for
+    running sandboxes to finish.


Suggested change

Moves the app from 'deployed' to 'stopped' in the Modal dashboard,

eliminating the visual clutter of idle ``Tasks: 0`` app entries.

Stopped apps remain visible in the 'recently stopped' section with

their logs intact, so debugging is not affected.

This should only be called after the associated Modal sandbox has

reached a terminal state, because stopping the app does not wait for

running sandboxes to finish.

schustmi · 2026-07-15T12:10:07Z

+        When ``stop_app_after_run`` is enabled in the step settings, this
+        calls ``modal app stop`` on the per-step app, moving its entry from
+        'deployed' to 'stopped' in the Modal dashboard. Stopped apps remain
+        visible in the 'recently stopped' section with their full log history
+        intact, so debugging is unaffected. Failures are logged as warnings
+        and never re-raised so that cleanup cannot mask the step result.


Suggested change

When ``stop_app_after_run`` is enabled in the step settings, this

calls ``modal app stop`` on the per-step app, moving its entry from

'deployed' to 'stopped' in the Modal dashboard. Stopped apps remain

visible in the 'recently stopped' section with their full log history

intact, so debugging is unaffected. Failures are logged as warnings

and never re-raised so that cleanup cannot mask the step result.

schustmi · 2026-07-15T12:10:56Z

+        if not settings.stop_app_after_run:
+            return
+
+        app_name = f"zenml-{step_run.id}-{step_run.name}"[:64]


This should be extracted in some helper function if it's used in multiple places.

schustmi · 2026-07-15T12:12:12Z

        f"Examples: 3600 (1 hour), 7200 (2 hours), {DEFAULT_TIMEOUT_SECONDS} (24 hours maximum). "
        "Execution will be terminated if it exceeds this timeout",
    )
+    stop_app_after_run: bool = Field(


This is added on the base settings class, but only respected when using the step operator. I think we should move it to the step operator settings, or implement it for all flavors inheriting from this (sandbox, orchestrator, step operator).

Also, I think stop_app or something similar makes more sense, as for example the step operator stops it after a step finishes, not a run.

schustmi · 2026-07-15T12:14:24Z

+        description="Stop the Modal app after the sandbox reaches a terminal "
+        "state. When True, ZenML calls `modal app stop` on the app created "
+        "for this run, removing it from the Modal dashboard's deployed-apps "
+        "list. Stopped apps are still visible in the 'recently stopped' "
+        "section and their logs remain accessible. Defaults to False to "
+        "preserve the previous behavior.",


Suggested change

description="Stop the Modal app after the sandbox reaches a terminal "

"state. When True, ZenML calls `modal app stop` on the app created "

"for this run, removing it from the Modal dashboard's deployed-apps "

"list. Stopped apps are still visible in the 'recently stopped' "

"section and their logs remain accessible. Defaults to False to "

"preserve the previous behavior.",

description="Stop the Modal app after the sandbox reaches a terminal "

"state.",

Copilot AI review requested due to automatic review settings June 24, 2026 18:08

Copilot started reviewing on behalf of raza-khan0108 June 24, 2026 18:09 View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jun 24, 2026

View reviewed changes

raza-khan0108 force-pushed the modal/stop-app-after-sandbox-run branch from 5f986c5 to 15319f5 Compare June 25, 2026 18:30

raza-khan0108 force-pushed the modal/stop-app-after-sandbox-run branch from 15319f5 to ec83b0d Compare June 25, 2026 18:32

raza-khan0108 changed the base branch from main to develop June 25, 2026 18:40

schustmi reviewed Jul 2, 2026

View reviewed changes

schustmi requested changes Jul 15, 2026

View reviewed changes

Uh oh!

Conversation

raza-khan0108 commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe changes

Why subprocess instead of the Python SDK?

Changes

Usage

Uh oh!

CLAassistant commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

schustmi commented Jun 25, 2026

Uh oh!

schustmi commented Jun 25, 2026

Uh oh!

raza-khan0108 commented Jun 25, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

raza-khan0108 commented Jun 24, 2026 •

edited

Loading

CLAassistant commented Jun 24, 2026 •

edited

Loading