Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -174,3 +174,27 @@ exclude = [
"src/benchflow/rewards/rubric_config.py",
"src/benchflow/experimental/mcp/reviewer_server.py",
]

[tool.uv]
# Dependency cooldown (hard rule): never resolve to a release younger than ~7
# days. `exclude-newer` caps every `uv lock` at this timestamp; bump it to a date
# >= 7 days in the past when intentionally taking updates. The CI check
# `tests/test_dep_cooldown.py` fails if this date is ever within the last 7 days.
#
# Security exception: a freshly-published CVE fix (younger than the cap) is
# allowed via a per-package override in `exclude-newer-package` below, with a
# comment recording why. Remove the override once the package is >= 7 days old.
exclude-newer = "2026-06-08T00:00:00Z"

[tool.uv.exclude-newer-package]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Pin uv before using exclude-newer-package

In environments that still have uv 0.7.x, this new table is an unknown [tool.uv] key: uv sync --locked reports unknown field exclude-newer-package, ignores the lockfile cutoff as removed, and tries to re-resolve from PyPI instead of installing the committed lock. Since the repo only requires users to have uv and does not set tool.uv.required-version, local setup and any older CI image can fail or bypass the intended cooldown; pin/install a uv version that supports per-package cutoffs before relying on this syntax.

Useful? React with 👍 / 👎.

# Grandfathered exceptions for packages the repo already depends on that are
# still younger than the cooldown. Each entry should age out (be removed) once
# the package crosses 7 days old.
#
# litellm[proxy]==1.89.0 (2026-06-13) is an exact pin newer than the cap, so
# resolution can't proceed without allowing it through.
litellm = "2026-06-14T00:00:00Z"
# starlette 1.3.1 (2026-06-12) fixes CVE-2026-54282/54283 but is < 7 days old;
# allow it through so the cooldown doesn't roll back to the vulnerable 1.2.1.
# (aiohttp 3.14.1 / python-multipart 0.0.32 are already >= 7 days old.)
starlette = "2026-06-13T00:00:00Z"
37 changes: 37 additions & 0 deletions tests/test_dep_cooldown.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
"""Hard rule: the dependency cooldown — never resolve to a release younger than
a 7-day window.

`[tool.uv] exclude-newer` in pyproject caps every `uv lock` at a fixed timestamp.
This test fails if that timestamp is ever within the last 7 days, so the lock can
never include a brand-new (and thus less-vetted) release. When you intentionally
take dependency updates, advance `exclude-newer` to a date still >= 7 days in the
past. A genuinely urgent (e.g. security) fix younger than the cap is allowed only
via an explicit, commented `[tool.uv.exclude-newer-package]` override.
"""

from __future__ import annotations

import datetime
import tomllib
from pathlib import Path

COOLDOWN_DAYS = 7
_PYPROJECT = Path(__file__).resolve().parents[1] / "pyproject.toml"


def test_uv_exclude_newer_enforces_dependency_cooldown() -> None:
cfg = tomllib.loads(_PYPROJECT.read_text(encoding="utf-8"))
raw = cfg.get("tool", {}).get("uv", {}).get("exclude-newer")
assert raw, (
"[tool.uv] exclude-newer is missing from pyproject.toml — it enforces the "
f"{COOLDOWN_DAYS}-day dependency cooldown and must be set."
)
cutoff = datetime.datetime.fromisoformat(str(raw).replace("Z", "+00:00"))
if cutoff.tzinfo is None:
cutoff = cutoff.replace(tzinfo=datetime.UTC)
age = datetime.datetime.now(datetime.UTC) - cutoff
assert age >= datetime.timedelta(days=COOLDOWN_DAYS), (
f"[tool.uv] exclude-newer ({raw}) is only {age.days}d old; the dependency "
f"cooldown requires it to be >= {COOLDOWN_DAYS} days in the past. Bump it "
f"to an older date (>= {COOLDOWN_DAYS}d ago) when taking dependency updates."
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Grandfather overrides have no CI expiry enforcement

The test guards [tool.uv] exclude-newer but never inspects [tool.uv.exclude-newer-package]. The per-package overrides are the one channel through which a package newer than the global cooldown can enter the lockfile, yet nothing checks that those entries carry a required comment, or that they're removed once the package crosses the 7-day mark. In practice the litellm and starlette entries will silently linger forever after they age out, and a future engineer can add another entry without any guardrail. Adding a second assertion that iterates cfg["tool"]["uv"].get("exclude-newer-package", {}) and verifies each override timestamp is still within some reasonable tolerance (e.g., entry date ≥ global exclude-newer date) would close this gap without much code.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 151c63d7 on the pushed head 44133f5c. The original expiry side was already covered; this now also enforces the rationale side without relying on TOML comments. Each [tool.uv.exclude-newer-package] entry must have a matching non-empty [tool.benchflow.dependency-cooldown.override-rationale] entry, and stale rationale entries without an override fail too. Validation passed locally: tests/test_dep_cooldown.py tests/test_lock_tool.py (24 passed), uv lock --check, tools/lock.py --check -> 2026-06-26T00:00:00Z, ruff, ty, format check. GitHub CI is still running on the new head.

Comment on lines +50 to +87

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The error message uses age.days, which is the integer days component of the timedelta, not the total elapsed days rounded. For a gap like 6 days 23 h 50 min, it prints "6d old" while the assertion correctly sees it as still less than 7 days — the diagnostic makes the boundary look farther away than it is. Prefer age.total_seconds() / 86400 for accurate sub-day precision.

Suggested change
assert age >= datetime.timedelta(days=COOLDOWN_DAYS), (
f"[tool.uv] exclude-newer ({raw}) is only {age.days}d old; the dependency "
f"cooldown requires it to be >= {COOLDOWN_DAYS} days in the past. Bump it "
f"to an older date (>= {COOLDOWN_DAYS}d ago) when taking dependency updates."
)
assert age >= datetime.timedelta(days=COOLDOWN_DAYS), (
f"[tool.uv] exclude-newer ({raw}) is only {age.total_seconds() / 86400:.1f}d old; "
f"the dependency cooldown requires it to be >= {COOLDOWN_DAYS} days in the past. "
f"Bump it to an older date (>= {COOLDOWN_DAYS}d ago) when taking dependency updates."
)

43 changes: 25 additions & 18 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading