Skip to content

Threshold ERA5 land fractions in public benchmark regions - #326

Draft
sylvesterkaczmarek wants to merge 2 commits into
google-research:mainfrom
sylvesterkaczmarek:fix/benchmark-land-mask
Draft

sylvesterkaczmarek wants to merge 2 commits into
google-research:mainfrom
sylvesterkaczmarek:fix/benchmark-land-mask

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek sylvesterkaczmarek commented Nov 6, 2026 •

Copy link
Copy Markdown

Fixes #316.

Convert the ERA5 land fraction to a boolean >= 0.5 mask before passing it to binning.Regions in the public benchmark. This matches the existing binning.LandSea convention and prevents mostly-ocean cells from being counted as land. The Regions API and the non-land regional masks are unchanged.

Add regression tests that execute the actual benchmark entrypoint with dataset loading and pipeline execution mocked. They inspect the real aggregator and cover fractional values, the 0.5 boundary, NaNs, reversed latitude, binary-mask compatibility and unchanged non-land regions. A synthetic RMSE case changes from the incorrect land-only value of 6 to 2, while global RMSE remains 6. The regression test is now under weatherbenchX/, so the existing pytest weatherbenchX command discovers it without changing the workflow.

Validation on Python 3.11:

  • New and adjacent binning/aggregation tests: 42 passed. Three of the four new tests fail against the original benchmark and the binary-mask control already passes.
  • Before moving the test file, the full local suite passed 269 tests plus 10 subtests, with Abseil flags initialized before pytest. The plain pytest command hits existing unparsed test_tmpdir errors with this environment's Abseil version; the same error was reproduced in a separate unmodified upstream checkout. No unrelated test-runner change is included.
  • After moving the test unchanged to weatherbenchX/public_benchmark_test.py, all four focused regression cases passed on Python 3.12. Existing test discovery picked up these four cases among 269 collected tests. The Google workflow-security scan passed on the new commit; hosted project CI still requires maintainer approval.
  • Ruff's correctness checks, Pyink formatting of the new test and git diff --check passed.

Existing published land-region results were not recomputed or replaced. Regenerating those results will change their land-region scores and requires maintainer coordination; non-land regions retain their masks. No ERA5 or forecast dataset was downloaded, no cloud pipeline was submitted, and no published benchmark data was modified.

To reproduce the full local test launch when Abseil flags are not initialized by the runner:

from absl import flags
from absl.testing import absltest
import pytest
flags.FLAGS(["pytest"])
raise SystemExit(pytest.main([
    "weatherbenchX"
]))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Land regions on the WeatherBench website count any cell with some land as land

1 participant