Support for resource pools v2

[stefannica]

Overview

This PR updates resource pool support to use the new ZenML Pro Resource Manager as the source of truth for scheduling, allocation, lease management, and eviction decisions. The ZenML workspace now keeps only the runtime-facing resource request surface needed by pipeline execution, while the old resource pool control-plane entities are removed from the OSS API, client, CLI, RBAC, DB schemas, and local store implementation.

It also connects allocated resources all the way into dynamic step execution. Resource settings can now describe richer resource demands, ZenML creates Resource Manager-backed requests for eligible dynamic steps, waits for allocation before submitting remote work, and passes the resulting allocation into orchestrators or step operators that explicitly support it. Kubernetes is the first implementation to opt in: it applies Resource Manager-selected component settings and translates allocated CPU, memory, and GPU capacity into Kubernetes pod requests and limits.

flowchart TD
    A[Dynamic step run is created] --> B[ZenML builds Resource Manager demands<br/>from ResourceSettings]
    B --> C[Workspace creates resource request<br/>through Resource Manager]
    C --> D{Allocated?}

    D -- queued/pending --> E[StepLauncher polls and renews<br/>pending request lease]
    E --> D

    D -- allocated --> F[StepLauncher receives allocated<br/>ResourceRequestResponse]
    F --> G{Execution target}

    G -- step operator --> H[submit_with_allocation]
    G -- dynamic orchestrator --> I[submit_isolated_step_with_allocation]

    H --> J[Kubernetes applies target settings<br/>and pod resource overrides]
    I --> J
    J --> K[Remote step starts]
    K --> L[Heartbeat renews lease while running]
    L --> M[Resources released on completion<br/>or cancellation]

Loading

Finally, resource request leases are integrated with the existing step heartbeat mechanism. Heartbeats now keep allocated requests alive, report runtime state back to Resource Manager, and turn reclaim or eviction states into graceful step cancellation.

sequenceDiagram
    participant Step as Running step
    participant HB as Heartbeat worker
    participant Store as ZenML SQL store
    participant RM as Resource Manager

    Step->>HB: start heartbeat worker
    loop every heartbeat interval
        HB->>Store: update_step_heartbeat(timeout)
        Store->>RM: renew resource request lease
        RM-->>Store: current request status
        alt request still valid
            Store-->>HB: step remains running
        else request preempting / cancelled / expired / released
            Store-->>HB: return cancelling status
            HB->>Step: interrupt main thread
        end
    end
    Step->>Store: step shutdown / final status
    Store->>RM: release resource request

Loading

Breaking Changes

This change erases all data from the previous resource pools v1 implementation (pools, policies, resource requests. There is no migration path provided.

Implementation Details

• Extends ResourceSettings with a richer resource demand model:

  • adds PoolResourceDemand and flexible resources input support for resource names, name-to-quantity maps, single demand objects, and lists of demands
  • supports exact resource names, resource kinds, units, capacity classes, resource selectors, and class selectors
  • adds gpu_class, allocation_wait_timeout_seconds, initialization_lease_seconds, and reclaim_tolerance
  • keeps backwards-compatible input handling for legacy pool_resources and preemptible
  • converts configured resources into canonical ResourceRequestDemand objects for Resource Manager requests

• Updates dynamic runtime selection:

  • dynamic steps that request basic infrastructure resources (cpu, memory, or gpu) now resolve to StepRuntime.ISOLATED when the orchestrator supports isolated steps
  • opaque/custom pool resources do not force isolation by themselves

• Adds allocation-aware launch hooks:

  • introduces supports_resource_pool_allocation on base orchestrators and step operators, defaulting to False
  • adds submit_isolated_step_with_allocation(...) on BaseOrchestrator
  • adds submit_with_allocation(...) and launch_with_allocation(...) on BaseStepOperator
  • preserves backwards compatibility by delegating to the existing methods for components that do not opt in

• Wires allocations through the dynamic step launcher:

  • waits for a dynamic step’s resource request to become allocated before launching remote execution
  • renews the request lease while waiting for allocation
  • honors allocation wait and initialization lease timeouts from ResourceSettings
  • passes the allocated ResourceRequestResponse into the selected step operator or dynamic orchestrator submission path

• Adds Kubernetes allocation support:

  • marks the Kubernetes orchestrator and Kubernetes step operator as supporting resource pool allocations
  • applies Resource Manager target component settings on top of static stack/component settings
  • merges Resource Manager allocation-derived pod resources on top of existing pod requests and limits
  • converts allocated CPU, memory, and GPU quantities into Kubernetes-compatible resource keys and units

• Minimizes the workspace resource pool API:

  • removes legacy resource pool, policy, grant, queue, and demand control-plane models and endpoints from the workspace
  • removes the resource pool CLI group and old client/store methods
  • keeps resource request listing and retrieval for runtime visibility
  • adds renew and release resource request operations

• Delegates runtime request storage to Resource Manager:

  • adds a Resource Manager-backed resource pool store facade
  • adds a Resource Manager client, auth/login flow, transport models, response conversion, and error mapping
  • enriches Resource Manager requests with ZenML Pro organization, workspace, project, pipeline, step, component, connector, and user context
  • stores only the external resource_request_id on step runs instead of local resource request rows

• Reuses heartbeats for lease renewal and eviction:

  • extends heartbeat requests with heartbeat_liveness_timeout_seconds
  • renews resource request leases from SQL store heartbeat updates
  • reports runtime state transitions such as pending, submitted, and running
  • keeps the heartbeat worker alive until step shutdown completes, so leases continue to renew during graceful termination
  • maps Resource Manager eviction/reclaim statuses to step cancellation
  • validates that reclaim-tolerant isolated steps have heartbeat enabled

• Reflects ZenML Pro account privileges:

  • maps the external Pro user is_superuser property to ZenML’s internal is_admin user property during auth/user synchronization

Pre-requisites

Please ensure you have done the following:

• I have read the CONTRIBUTING.md document.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
• IMPORTANT: I made sure that my changes are reflected properly in the following resources:
  • ZenML Docs
  • Dashboard: Needs to be communicated to the frontend team.
  • Templates: Might need adjustments (that are not reflected in the template tests) in case of non-breaking changes and deprecations.
  • Projects: Depending on the version dependencies, different projects might get affected.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[github-actions]

⚠️ This PR has been inactive for 2 weeks and has been marked as stale.
Timeline:

• Week 2 (now): First reminder - PR marked as stale
• Week 4: PR will be automatically closed if no activity
  Please update this PR or leave a comment to keep it active. Any activity will reset the timer and remove the stale label.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​pydantic-settings@​2.14.2

View full report

[github-actions]

Documentation Link Check Results

✅ Absolute links check passed
✅ Relative links check passed
_{Last checked: 2026-07-24 20:12:44 UTC}

[github-actions]

ZenML CLI Performance Comparison (Threshold: 1.0s, Timeout: 60s, Slow: 5s)

Performance Comparison

Command	develop Time (s)	feature/resource-pools-v2 Time (s)	Difference	Status
zenml --help	1.817522 ± 0.048652	2.010735 ± 0.147306	+0.193s	✓ No significant change
zenml model list	3.295347 ± 0.049950	4.355385 ± 0.259187	+1.060s	❌ Degraded
zenml pipeline list	3.352906 ± 0.042953	4.189764 ± 0.222818	+0.837s	✓ No significant change
zenml stack --help	1.954272 ± 0.075525	1.984166 ± 0.093760	+0.030s	✓ No significant change
zenml stack list	4.257395 ± 1.893682	4.456132 ± 0.167789	+0.199s	✓ No significant change

Summary

• Total commands analyzed: 5
• Commands compared for timing: 5
• Commands improved: 0 (0.0% of compared)
• Commands degraded: 1 (20.0% of compared)
• Commands unchanged: 4 (80.0% of compared)
• Failed commands: 0
• Timed out commands: 0
• Slow commands: 0

Environment Info

• Target branch: Linux 6.17.0-1020-azure
• Current branch: Linux 6.17.0-1020-azure
• Test timestamp: 2026-07-24T19:36:30Z
• Timeout: 60 seconds
• Slow threshold: 5 seconds