Summary — v0.19.0: EvaluationV2, benchmark paradigm

Brings the SDK to parity with the Nucleus EvaluationV2 UI. Evaluation creation is benchmark-scoped only — dataset/slice-scoped creation (an unreleased earlier iteration of this branch) was removed before release as the platform moved to the benchmark paradigm.

Added

• Benchmarks. create_benchmark() (members from item_ids, (dataset_id, ref_id) items pairs, a slice_id, or a dataset_id — exactly one required; membership frozen at creation), list_benchmarks(), get_benchmark(), update_benchmark(), delete_benchmark(), list_benchmark_items(), plus the Benchmark resource (refresh() / update() / delete() / items() / create_evaluation_v2()) in nucleus/benchmark.py.
• Benchmark evaluations. create_benchmark_evaluation_v2(benchmark_id, model_run_id, ...) — every benchmark item is scored (uncovered items count as FN, keeping leaderboard scores comparable). EvaluationV2 exposes benchmark_id, rollup_groups, exclusion_rules, exclusion_stats. No only_items_with_predictions / eval-scope slice_id (invalid in the benchmark paradigm; the server rejects the former).
• Rollup groups. RollupGroup (class_name + labels) as the primary label configuration on benchmark eval create and presets (create/update_evaluation_v2_preset(rollup_groups=...), mutually exclusive with legacy allowed_label_matches; parsed from both key casings). Preset seeding prefers rollup_groups, falling back to legacy matches.
• Exclusion rules. MetadataExclusionRule / LabelExclusionRule / BoxAreaExclusionRule via exclusion_rules on eval create and presets.
• Presets. list/create/update/delete_evaluation_v2_preset + EvaluationV2Preset resource; preset= seeds benchmark eval creation (explicit args override).
• Results. EvaluationV2.charts() (mAP, per-class AP, confusion matrix, PR/F1 curves, TIDE, AP by size) and examples() (paginated TP/FP/FN; match_type optional) with EvaluationV2FilterArgs filtering; cancel() / retry(); Dataset.evaluation_label_schema().
• Benchmark leaderboards. leaderboard_ranking(metric_type, benchmark_ids, ...) (metrics: MAP_50, MAP_50_95, AP_SMALL/MEDIUM/LARGE, PRECISION, RECALL, F1; scope/collapse) and leaderboard_f1_curve(benchmark_ids, ..., top_n=5).
• Filter schema discovery. EvaluationV2.filter_schema() / get_evaluation_v2_filter_schema() return the evaluation's filter vocabulary (gt_labels, pred_labels, metadata_fields with inferred value types).
• New DTOs: EvaluationV2FilterSchema, LeaderboardRankingEntry, LeaderboardF1CurveEntry, BenchmarkItemsPage.

Removed (relative to earlier commits on this branch — never released)

• create_evaluation_v2 (dataset/slice-scoped), create_evaluations_v2_batch, BatchEvaluationResult, only_items_with_predictions. Reading existing evals (including old slice-scoped ones) still works.

Fixed

• CI build_test pylint failure (pre-existing W0718 on the per-job catch) — suppressed with a black-stable # pylint: disable-next line.

Tests / Version

• tests/test_benchmarks.py, tests/test_leaderboard.py, plus rollup/benchmark coverage in the eval/preset test files (59 unit tests, mock-based).
• pyproject.toml → 0.19.0; single consolidated CHANGELOG entry (the unreleased 0.18.9 section was folded in).

Server dependency: the leaderboard and filter-schema methods call new REST mirrors (POST /nucleus/leaderboard/ranking, POST /nucleus/leaderboard/f1Curve, GET /nucleus/evaluationsV2/:id/filterSchema) that ship separately in scaleapi. Unit tests pass regardless; live calls 404 until that deploys. All benchmark CRUD / benchmark-eval / rollup-group / preset endpoints are already live (DE-8209).

resolves https://linear.app/scale-epd/issue/DE-8209

Greptile Summary

This PR brings the Nucleus Python SDK to parity with the EvaluationV2 UI under the benchmark paradigm. It introduces benchmarks (frozen item sets for evaluation), benchmark evaluations with rollup groups, exclusion rules, presets, leaderboard ranking, filter-schema discovery, and the cancel/retry lifecycle actions.

• New modules added: nucleus/benchmark.py, nucleus/evaluation_v2_exclusions.py, nucleus/evaluation_v2_preset.py — each fully tested with 59 mock-based unit tests.
• Breaking removal: create_evaluation_v2 (dataset/slice-scoped) is gone; create_benchmark_evaluation_v2 via benchmarks/{id}/evaluationsV2 is the sole creation path.
• Connection layer: A new patch() method mirrors the existing post() and get() helpers, used for update_evaluation_v2_preset and update_benchmark.

sequenceDiagram
    participant U as User
    participant NC as NucleusClient
    participant API as Nucleus API

    U->>NC: "create_benchmark(name, slice_id=...)"
    NC->>API: POST /benchmarks
    API-->>NC: "{benchmark_id, ...}"
    NC-->>U: Benchmark

    U->>NC: "create_benchmark_evaluation_v2(benchmark_id, model_run_id, preset=preset)"
    Note over NC: Seed from preset if no explicit label config
    NC->>API: "POST /benchmarks/{id}/evaluationsV2"
    API-->>NC: "{evaluation_id}"
    NC->>API: "GET /evaluationsV2/{id}"
    API-->>NC: EvaluationV2 payload
    NC-->>U: EvaluationV2

    U->>NC: evaluation.wait_for_completion()
    loop poll
        NC->>API: "GET /evaluationsV2/{id}"
        API-->>NC: "{status: ...}"
    end
    NC-->>U: EvaluationV2 (succeeded)

    U->>NC: "evaluation.charts(iou_threshold=0.5)"
    NC->>API: "POST /evaluationsV2/{id}/charts"
    API-->>NC: EvaluationV2Charts
    NC-->>U: EvaluationV2Charts

    U->>NC: leaderboard_ranking("MAP_50", [benchmark_id])
    NC->>API: POST /leaderboard/ranking
    API-->>NC: "[{rank, score, ...}]"
    NC-->>U: List[LeaderboardRankingEntry]

_{Reviews (8): Last reviewed commit: "Make the eval/benchmark docstrings user-..." | Re-trigger Greptile}