First-class agentic RL: example, sandbox substrate, and Kubernetes services

[htahir1]

Draft — stacked on #5029. This branch includes the Harbor integration merge; the effective diff shrinks to ~+11.9k lines (of which ~7k are 327 generated Harbor task files) once #5029 lands in develop. Live validation (Docker smoke, GPU tier, cluster service leg) is still pending — see Verification.

What this does

Makes the agentic RL stack (verifiers + Harbor + vLLM + prime-rl) first-class in ZenML, with ZenML owning what none of those record: the sandbox rollouts run in, the gate in front of the GPU, and the lineage behind it. prime-rl keeps its loop; ZenML never pretends to orchestrate RL training.

Core (src/zenml)

• Docker sandbox flavor — container-isolated local sessions (exec with demuxed streams, archive file transfer, docker commit snapshots, attach). The laptop counterpart to the kubernetes/modal flavors.
• Flavor-agnostic sandbox image override — BaseSandbox.image_settings() capability (docker/kubernetes/modal), replacing the Modal-only branch in the Harbor bridge.
• LocalSandbox file transfer with workdir containment (absolute paths outside the session workdir are refused rather than writing to host /).
• Harbor integration on harbor 0.18 — pin bump from the dead >=0.8,<0.9; the 0.8-era allow_internet check (which refuses every current task) is replaced by harbor's own capability validation.
• Multi-node command steps on the Kubernetes step operator — node_count gated on command steps (the Baseten contract): indexed Job, rank from JOB_COMPLETION_INDEX, job-owned headless service, and a rank-dispatch so exactly one pod runs ZenML's step machinery (N concurrent StepRunner.run calls for one step_run_id would race on status/logs/success).
• KubernetesPodService — long-running services on the existing BaseService framework: deterministic naming (named services have URLs known before provisioning), server-side registration via Client.create_service, and deadline-based GC teardown (max_lifetime_seconds), so a GPU server leaked by a crashed run self-terminates.
• datasets cap raised to <5.0.0 (the <4 cap silently disabled HFDatasetMaterializer in datasets-4.x environments).

Example (examples/agentic_rl)

Gate → train → receipts, in tiers: hermetic smoke (oracle vs nop, no API keys), eval campaigns, prime-rl GRPO as one CommandStep, per-rollout traces.jsonl ingestion into a single table artifact (reward ↔ sandbox session id join), checkpoint discovery, HTML lineage report, and an opt-in serve/probe/stop leg using KubernetesPodService. 64 committed Harbor task dirs scored by the RL spike's verifier byte-identical (asserted by test); a verifiers taskset package carrying ZenMLSandboxRuntime written to upstream-contribution shape.

Key honesty rules encoded: the gate checks errors before rewards (an errored shard logs no mean reward — a reward-only gate passes broken campaigns); a crashed scorer is an errored trial, never a fake 0.0; the file-tier rollout table carries rewards + sandbox ids but not advantages/tokens (excluded upstream); the harbor-agents-vs-served-policy eval is a named cut.

Docs

component-guide/sandboxes/docker.md, a multi-node section in the step-operator guide, user-guide/agent-evals-guide/prime-rl-on-kubernetes.md (incl. cluster prerequisites: what ZenML deploys per-run vs. what you provide once), and docs/design/multi-node-command-steps-on-kubernetes.md.

Reviewer attention

• The rank-dispatch + GC-teardown design in kubernetes_step_operator.py (correctness of one-bookkeeper semantics).
• KubernetesPodService teardown philosophy (deprovision = happy path, GC deadline = guarantee) and the never-fail-the-step service registration in the example.
• The harbor 0.18 migration (network_mode handling now delegated to harbor's capability validation).

Verification

• 247 core unit tests (kubernetes/sandboxes/harbor) + 21 example tests green; mypy clean on all touched modules; ruff clean (examples/tests under CI's --extend-ignore D).
• Oracle solution scored exactly 1.0 through the byte-identical verifier locally; verifiers 0.2 runtime dispatch patch verified against the real package.
• Tier 0 smoke: GREEN, run live 2026-07-17 on the docker sandbox flavor (WSL, Docker 28.5.1): oracle 1.0 / nop 0.0 through real sandbox containers and the byte-identical verifier, gate passed at mean 0.500 over 6 rewards with 0 blocking errors, 1m27s wall. The first live run also proved the honest-failure design: two shim bugs (harbor's reward.json numeric-map precedence; nop's missing submission crashing the scorer) surfaced as 6/6 errored trials and a loud gate failure — never a fake 0.0 (fixed in ed795a9dfa).
• Found in live validation, fixed separately: local sqlite stores fail with database is locked under concurrent mapped steps (sqlite3's 5s default lock wait; branch fix/sqlite-busy-timeout).
• Still pending live runs: --train needs 2 GPUs + a prime-rl checkout; the service/multi-node legs need a cluster and are experimental until then. These are the remaining gates before undrafting.

Depends on #5029. Suggest full CI (run-slow-ci) once undrafted.

🤖 Generated with Claude Code

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​harbor@​0.18.1.dev202607160125
	pypi/​click@​8.4.2
	pypi/​hatchling@​1.31.0

View full report