Add first-class Harbor integration for agent-eval campaigns

[htahir1]

Describe changes

Promotes the Harbor example into a first-class zenml.integrations.harbor integration for running Harbor agent-eval campaigns as ZenML pipelines. ZenML owns the outer orchestration (matrix expansion, per-shard steps, retries, caching, artifacts, reports); Harbor keeps the trial eval kernel (task loading, agent loop, verifier, reward). Trials execute inside the active stack's Sandbox component via the ZenMLSandboxEnvironment bridge (relocated from the example, behavior-identical).

Integration (src/zenml/integrations/harbor/):

• models.py / utils.py (harbor-free): typed campaign contracts, stable sha256 trial identities (task + agent + model + kwargs + env + index — the cross-run baseline join key), deterministic single-cell shard packing with a trials_per_step knob.
• job_runner.py: one single-cell Harbor JobConfig per shard, run via await Job.create(...) / job.run(), per-trial results from the in-memory JobResult.trial_results. Dataset source is deliberately kept off the Harbor-side config (fixes an IndexError in Harbor 0.8's quiet-mode display for client-side-resolved registry datasets) and restored onto results afterward.
• materializers/: HarborShardResultMaterializer persists the flat summary plus Harbor's full job tree (agent/verifier logs, trajectories) as a tar archive; queryable metadata, markdown visualization, on-demand download_jobs_dir(). Pinned on the step via output_materializers so transitive-dependency drift can't silently degrade to the pydantic fallback.
• steps/: build_harbor_matrix → run_harbor_shard (mapped; logs harbor.* step metadata; optional fail_on_trial_error rescues the shard result + log archive as harbor_shard_result_<id>_failed before failing) → build_harbor_report.

Example (examples/harbor_agent_evals/, renamed from sandbox_harbor): default is a hermetic oracle-vs-nop head-to-head (no keys) that prints its own receipts — report, metadata scan, restored log tree; dataset:NAME@VERSION[:N] CLI slicing for registry benchmarks.

Docs: new top-level Agent evals guide (user-guide/agent-evals-guide/): overview, Run an eval campaign, Benchmark operations (delta reruns, error gating, cross-run identity joins, regression gates), From evals to training (SFT from winning trajectories, rejection sampling, config sweeps, eval-gated promotion). Plus the materializers table row and a sandboxes-page cross-reference. Cross-space references use absolute docs.zenml.io URLs.

Validation (all live, on Modal sandboxes)

• Local campaigns: oracle 1.000 vs nop 0.000, 6 trials / 4 shards; identical rerun fully cache-hits (47s → 19s).
• Resume-after-failure: a genuine Modal session hang on one shard; rerun cache-hit the finished shards and re-executed only the hung one.
• Registry benchmarks: terminal-bench-sample@2.0 — chess-best-move, polyglot-c-py, sqlite-with-gcov, all reward 1.000 under oracle, incl. per-task docker_image overrides.
• Batching: one shard ran 4 trials concurrently in 21s (single trial ≈ 15s).
• fail_on_trial_error: live rescue artifact verified (archive restorable, exception.txt + job.log present).
• Real LLM agent: claude-code + claude-sonnet-4-5 on chess-best-move — clean run, reward 0.000 (a genuine benchmark result), $0.1193 tracked through trial → metadata → report.
• 66 unit tests (incl. a Harbor 0.8 API canary and an import-safety canary), mypy strict clean, verified on a harbor-less Python 3.11 env (gated tests skip via harbor.job — namespace-package-proof).

Areas needing reviewer attention

• Python gating: Harbor requires >= 3.12; get_requirements() returns [] below (quiet on the interpreter fallback), check_installation() is version-guarded.
• CI installs: harbor is skipped on 3.10/3.11 legs and installed on 3.12+ (pulls litellm/datasets/supabase) so the gated tests run — flag if you'd rather confine to one leg.
• Bridge uses two Harbor protected members (_merge_env, _validate_definition override) — Harbor's intended subclass hooks.
• Test layout: tests/unit/integrations/harbor/ deliberately has NO __init__.py (would shadow the real harbor package); see the conftest guard.
• mypy: harbor ships no py.typed → ignore_missing_imports + disallow_subclassing_any=false scoped to the environment module.
• Post-release docs sweep: deepen /stacks links to /stacks/sandboxes and link the guide from the sandboxes page once published.

Non-goals / extension points

No Harbor Hub upload, no built-in baseline store, no multi-step tasks or separate verifier sandboxes (rejected with NotImplementedError), no Sandbox core changes. Known bridge limitations (no resource translation, no exec(user=...), allow_internet=false refused, docker_image override Modal-only) preserved and documented. Trajectory→training-dataset utilities are the natural follow-up (the archives already carry rollouts + rewards; see the From evals to training guide).

Related: #5030 (remote image builders re-apply .dockerignore and drop generated context files), found while validating the Modal-orchestrator path.

Pre-requisites

• I have read the CONTRIBUTING.md document.
• I have added tests to cover my changes.
• I have based my new branch on develop and the open PR is targeting develop.
• IMPORTANT: My changes are reflected properly in the following resources:
  • ZenML Docs (Agent evals guide, materializers table, sandboxes cross-reference)
  • Dashboard: Needs to be communicated to the frontend team.
  • Templates: Might need adjustments.
  • Projects

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Other (add details above)

[github-actions]

Documentation Link Check Results

✅ Absolute links check passed
✅ Relative links check passed
_{Last checked: 2026-07-17 10:02:21 UTC}

[strickvl]

cc @htahir1

Ran this integration on the Kubernetes sandbox flavor (staging EKS) as part of the RL-spike follow-ups — validation so far was Modal-only, so here's the second-flavor datapoint. Full writeup: B1_K8S_FINDINGS.md on spike/b1-harbor-k8s (forked off this branch; spike docs only, not meant to merge here).

The good news: everything behind the sandbox abstraction reproduces untouched. Hermetic oracle-vs-nop: 6/6 trials, oracle 1.000 / nop 0.000, ~30s wall clock; identical rerun cache-hits all shards in 8s with zero pods; metadata, report, and download_jobs_dir() all work.

The docker_image wall is already hollow: all three terminal-bench-sample@2.0 tasks hit the designed NotImplementedError — but KubernetesSandboxSettings.image exists, so the condition in _settings_override's docstring ("switch to the active flavor's settings class when another flavor ships an image field") is met. A five-line patch on the spike branch (commit) translating docker_image → KubernetesSandboxSettings(image=...) boots all three real Terminal-Bench images on EKS at oracle reward 1.000. Fine as a fast-follow rather than in this PR — open questions are private-registry credentials, image arch, and node disk pressure at 89-task fan-out.

Two small review-relevant observations from the failure path:

1. When start() raises, Harbor's cleanup calls exec() on the never-started env, so the user sees RuntimeError: ZenMLSandboxEnvironment used before start() + "Failed to download logs" noise above the real cause. Worth carrying the original failure in that error, or no-op'ing log download when no session was created.
2. A campaign where every trial errors reports each trial as Completed=1 and Errored=1, mean reward n/a — and a harbor.mean_reward < 1 metadata query returns zero shards (no metadata logged for errored shards). Anyone wiring a regression gate to mean reward would pass an all-error campaign; harbor.n_errored:gt:0 is the query that works. Maybe worth a sentinel reward or an errors-first example receipt.

[strickvl]

Follow-up from the same RL-spike track (see K8s comment above): I built the "From evals to training" story this PR's docs promise — a trajectory exporter that turns campaign shard artifacts into an accumulating RL-episode-candidates dataset artifact, run against real campaigns. Full write-up: B3_EXPORTER.md on spike/b3-exporter. The good news: the lineage edge works end-to-end, and the sha256 trial identity is exactly the right join key. Three findings are directly relevant to this PR, one fixable before merge:

1. trial_identity fragments across task-specification routes. It hashes the full TaskRef dump, and the dataset resolver stamps source: into the ref while a direct git+URL@COMMIT:SUBPATH ref doesn't — so two campaigns attempting the byte-identical pinned task get different identities (hit this live). Local-path tasks hash their absolute path, making identities machine-specific. Suggestion: hash only the canonical pin coordinates (effectively TaskRef.to_string()), which is a small change in utils.trial_identity and cheapest to do before merge, since shipped identities become baselines people join against.
2. Nothing records the resolved sandbox image. The job archive carries neither flavor nor image, and reconstructing it from the run's stack config gives provably wrong answers for docker_image-pinned tasks (my Terminal-Bench records claimed python:3.11-slim). The bridge is the only party that knows — one field in the shard/trial result, same neighborhood as the docker_image→K8s translation from the previous comment.
3. task_checksum is a one-line lift. Harbor writes it into every trial result.json but HarborTrialResult.from_harbor doesn't surface it — it's a content-addressed complement to the coordinate-based identity (and would fix the local-path half of finding 1).

[htahir1]

Both landed in 7d3a3bd: trial_identity now hashes the canonical pin (TaskRef.to_string()), task_checksum is on the trial result, and the bridge records sandbox_flavor/sandbox_docker_image per trial (session id == trial name made the join trivial).

Also from your K8s comment: start() failures now carry through to the cleanup-path errors, and the Completed/Errored double-count is gone — reports use n_succeeded, and an all-errored shard logs harbor.mean_reward = 0.0 so the documented reward gate fails safe.

The docker_image → KubernetesSandboxSettings translation stays a fast-follow per your note.

[htahir1]

@strickvl Can you take a look?