Skip to content

Latest commit

 

History

History
285 lines (235 loc) · 18.4 KB

README.md

File metadata and controls

285 lines (235 loc) · 18.4 KB

Extracting Static Attributes for Watersheds (multimet/static_extractor)

The multimet directory houses data workflows for preparing watershed inputs for OpenHydroNet. Its multimet.static_extractor submodule builds the static watershed attribute tables required by the models.

To predict river flow in a watershed, OpenHydroNet needs a table of unchanging ("static") facts about that watershed—such as its area, elevation, slope, soil type, land cover, and long-term average weather.

If you are working with watersheds from the published Caravan dataset, those tables are already provided. If you want to run models on your own watersheds, this tool takes a map file of your watershed boundaries and creates a Caravan-compatible CSV table for you.

It calculates these attributes using the same community datasets and methods used by Caravan:

  • HydroATLAS (BasinATLAS v10, Level 12): Geography, elevation, slope, soil, land cover, lakes, and human footprint.
  • ERA5-Land (1981–2020): Long-term climate averages (precipitation, temperature, evaporation, aridity, snow fraction, and wet/dry spell frequency).

Before You Start

1. What Your Input File Needs

  • File format: A map file containing polygon boundaries for one or more watersheds in GeoJSON (.geojson), Shapefile (.shp), GeoPackage (.gpkg), or GeoParquet (.parquet) format.
  • Coordinates: A defined coordinate reference system (standard latitude and longitude EPSG:4326 / WGS84, or any projected coordinate system, which will be converted to EPSG:4326).
  • Watershed ID column: Each polygon must have a unique ID in the gauge_id column (or in the column you specify with --id-column <column_name>).

2. Reference Datasets

You provide the paths where the HydroATLAS and ERA5-Land reference datasets are stored:

  • HydroATLAS (--gdb-path or --gcs-gdb-uri): Local path to BasinATLAS_v10.gdb (or BasinATLAS_v10_lev12.shp). If you do not already have it on disk, pass --gcs-gdb-uri gs://open-multimet/ancillary-data/hydroatlas/BasinATLAS_v10.gdb and the tool will download it into --gdb-path (~4.9 GB).
  • ERA5-Land Climate Data:
    • When using --era5-source hybas, pass --era5-cache-dir /path/to/era5_climate. If you do not already have the continental climate tables on disk, also pass --gcs-era5-climate-uri gs://open-multimet/ancillary-data/hydroatlas/era5_climate (~550 MB) to download them into --era5-cache-dir.
    • When using --era5-source gridded, pass --gridded-era5-uri gs://open-multimet/gridded-data-archives/ERA5_LAND/daily_surface.zarr (or a local Zarr path). This streams the daily grid slices directly from the Zarr store without downloading the archive to your machine.
  • Running Without Downloading Cloud Files (--no-download):
    • If you have limited local disk space and do not want to download the HydroATLAS or ERA5 files to your computer, pass --no-download along with --gcs-gdb-uri gs://open-multimet/ancillary-data/hydroatlas and --gcs-era5-climate-uri gs://open-multimet/ancillary-data/hydroatlas/era5_climate (or --gridded-era5-uri). The tool will stream the required HydroATLAS and ERA5 data directly from Google Cloud Storage in memory without saving reference files to disk.
    • Alternatively, if you download the files to disk for a run and want them deleted automatically when the run finishes, add --clean-cache.

3. Choosing --era5-source (hybas vs. gridded)

Every run requires the --era5-source flag to tell the tool how to calculate climate statistics:

Option How It Works Speed When to Use It
--era5-source hybas Combines pre-calculated climate summaries from the standard HydroATLAS sub-basins that overlap your watershed. (If you also pass --gridded-era5-uri, the four *_ERA5_LAND evaporation columns are computed from the daily weather grid). Fast Recommended for most users. Works well whenever your watersheds are roughly the size of standard river sub-basins (~100 $\text{km}^2$) or larger.
--era5-source gridded Reads 40 years (1981–2020) of daily ERA5-Land weather grids over your exact polygon boundary and calculates every climate number from scratch. Slower (several seconds per basin) Use this if your watersheds are very small, have custom boundaries that do not follow natural river sub-basins, or if you want to stream climate data directly from a Zarr store without downloading continental climate tables.

Command-Line Usage

1. Extract Attributes for One File (extract-caravan-static)

Use extract-caravan-static (or its alias extract-static-attributes) when you have a single file of watershed polygons and want a single CSV table of attributes.

# Using local HydroATLAS and pre-calculated sub-basin climate tables
extract-caravan-static \
    --input /path/to/watershed_polygons.geojson \
    --output /path/to/extracted_caravan_attributes.csv \
    --gdb-path /path/to/BasinATLAS_v10.gdb \
    --era5-source hybas \
    --era5-cache-dir /path/to/era5_climate

# Downloading HydroATLAS and climate tables from Google Cloud Storage on first run
extract-caravan-static \
    --input /path/to/watershed_polygons.geojson \
    --output /path/to/extracted_caravan_attributes.csv \
    --gdb-path /path/to/BasinATLAS_v10.gdb \
    --gcs-gdb-uri gs://open-multimet/ancillary-data/hydroatlas/BasinATLAS_v10.gdb \
    --era5-source hybas \
    --era5-cache-dir /path/to/era5_climate \
    --gcs-era5-climate-uri gs://open-multimet/ancillary-data/hydroatlas/era5_climate

# Streaming directly in memory from Google Cloud Storage without downloading to disk
extract-caravan-static \
    --input /path/to/watershed_polygons.geojson \
    --output /path/to/extracted_caravan_attributes.csv \
    --gcs-gdb-uri gs://open-multimet/ancillary-data/hydroatlas \
    --era5-source hybas \
    --gcs-era5-climate-uri gs://open-multimet/ancillary-data/hydroatlas/era5_climate \
    --no-download

# Streaming climate numbers directly from daily ERA5-Land grids on Google Cloud Storage
extract-caravan-static \
    --input /path/to/watershed_polygons.geojson \
    --output /path/to/extracted_caravan_attributes.csv \
    --gdb-path /path/to/BasinATLAS_v10.gdb \
    --era5-source gridded \
    --gridded-era5-uri gs://open-multimet/gridded-data-archives/ERA5_LAND/daily_surface.zarr

# Specifying a custom ID column and running across 8 CPU cores
extract-caravan-static \
    --input /path/to/basins.shp \
    --output /path/to/attributes.csv \
    --gdb-path /path/to/BasinATLAS_v10.gdb \
    --era5-source hybas \
    --era5-cache-dir /path/to/era5_climate \
    --id-column station_id \
    --workers 8

All Flags for extract-caravan-static

Flag Required? Default What It Does
--input, -i Yes — Path to your input watershed boundary file (.geojson, .shp, .gpkg, or .parquet).
--output, -o Yes — Path where the output CSV file will be saved.
--gdb-path, -g Required unless --no-download None Local path to BasinATLAS_v10.gdb, shapefile, or GeoParquet file.
--era5-source Yes — How to calculate ERA5 climate numbers: hybas (from pre-calculated sub-basin tables) or gridded (from daily ERA5-Land grids).
--era5-cache-dir Required if hybas (unless --no-download) None Local folder where continental ERA5 climate tables (<continent>_climate_indices.txt) are stored.
--gridded-era5-uri Required if gridded None Google Cloud Storage (gs://...) URI or local folder path for the daily ERA5-Land Zarr dataset. Can also be passed with --era5-source hybas to compute the four *_ERA5_LAND columns.
--gcs-gdb-uri Required if --no-download without --gdb-path None Google Cloud Storage (gs://...) URI for HydroATLAS data. Downloaded into --gdb-path by default, or streamed directly in memory when --no-download is set.
--gcs-era5-climate-uri Required if hybas with --no-download None Google Cloud Storage (gs://...) URI for continental ERA5 climate tables. Downloaded into --era5-cache-dir by default, or streamed directly in memory when --no-download is set.
--no-download No Disabled Stream HydroATLAS and ERA5 data directly from Google Cloud Storage in memory without downloading files to local disk.
--id-column No gauge_id Name of the column in your input file that holds the unique watershed ID.
--workers, -w No 1 Number of CPU processes to run in parallel. Increase this (for example, -w 8) when extracting many watersheds.
--min-overlap-threshold No 0.0 Minimum overlap area in $\text{km}^2$ required for a sub-basin fragment to be included (unless the watershed covers more than 50% of that sub-basin). Useful for ignoring tiny border slivers along the edge of a polygon.
--clean-cache No Disabled Delete the local --gdb-path and --era5-cache-dir folders after the command finishes.
--verbose, -v No Disabled Print detailed progress and debugging messages while running.

2. Extract Attributes for Many Folders at Once (extract-caravan-static-batch)

Use extract-caravan-static-batch (or its alias extract-static-attributes-batch) when you have multiple dataset folders containing watershed boundary files and want to process all of them in a single run.

# Process all dataset folders inside a parent folder and also save one combined CSV
extract-caravan-static-batch \
    --parent-dir /path/to/watershed_folders/ \
    --output-dir /path/to/output_attributes/ \
    --gdb-path /path/to/BasinATLAS_v10.gdb \
    --era5-source hybas \
    --era5-cache-dir /path/to/era5_climate \
    --workers 16 \
    --combine

# Process a specific list of folders
extract-caravan-static-batch \
    --input-dirs /data/shapes/camels /data/shapes/camelsaus /data/shapes/lamah \
    --output-dir /path/to/output_attributes/ \
    --gdb-path /path/to/BasinATLAS_v10.gdb \
    --era5-source hybas \
    --era5-cache-dir /path/to/era5_climate \
    --workers 16

All Flags for extract-caravan-static-batch

You must provide at least one of --parent-dir, --input-dirs, or --input-files.

Flag Required? Default What It Does
--parent-dir, -p One input flag required — One or more parent folders (local path or gs://...) that contain dataset subfolders of watershed files.
--input-dirs, -d One input flag required — Space-separated list of specific dataset folders to process.
--input-files, -f One input flag required — Space-separated list of specific polygon files (.shp, .geojson, .gpkg, .parquet) to process.
--output-dir, -o Yes — Local folder or Google Cloud Storage (gs://...) path where output files will be written.
--gdb-path, -g Required unless --no-download None Local path to BasinATLAS_v10.gdb, shapefile, or GeoParquet file.
--era5-source Yes — How to calculate ERA5 climate numbers: hybas or gridded.
--era5-cache-dir Required if hybas (unless --no-download) None Local folder where continental ERA5 climate tables are stored.
--gridded-era5-uri Required if gridded None Google Cloud Storage (gs://...) URI or local path for the daily ERA5-Land Zarr dataset.
--staging-dir Required for gs:// inputs/outputs None Local folder used to stage downloaded input shapefiles or output CSVs when reading from or writing to gs:// paths.
--gcs-gdb-uri Required if --no-download without --gdb-path None Google Cloud Storage (gs://...) URI for HydroATLAS data. Downloaded into --gdb-path by default, or streamed directly in memory when --no-download is set.
--gcs-era5-climate-uri Required if hybas with --no-download None Google Cloud Storage (gs://...) URI for continental ERA5 climate tables. Downloaded into --era5-cache-dir by default, or streamed directly in memory when --no-download is set.
--no-download No Disabled Stream HydroATLAS and ERA5 data directly from Google Cloud Storage in memory without downloading files to local disk.
--id-column No gauge_id Name of the column in your input files that holds the unique watershed ID.
--workers, -w No 1 Number of CPU processes to run in parallel.
--combine No Disabled In addition to per-dataset files, save a single merged table (attributes_caravan_combined.csv) containing all watersheds across all processed folders.
--partition-outputs, -P No Disabled When enabled, writes separate HydroATLAS and Caravan climate tables (attributes_hydroatlas_<dataset>.csv, attributes_caravan_<dataset>.csv, and attributes_<dataset>.parquet) inside each dataset's output subfolder instead of a single flat CSV.
--gcs-output-uri No None Optional Google Cloud Storage (gs://...) destination where finished files should be uploaded after saving them locally in --output-dir.
--no-resume No Disabled Re-run and overwrite datasets even if their output files already exist in --output-dir (by default, already finished datasets are skipped).
--min-overlap-threshold No 0.0 Minimum overlap area in $\text{km}^2$ required for a sub-basin fragment to be included.
--clean-staging No Disabled Delete --staging-dir after the batch run finishes.
--clean-cache No Disabled Delete --gdb-path, --era5-cache-dir, and --staging-dir when the batch run finishes.
--no-progress No Disabled Turn off interactive progress bars.
--verbose, -v No Disabled Print detailed progress and debugging messages while running.

Using It in Python

1. Extract Attributes from a File to a DataFrame and CSV

from multimet.static_extractor import StaticAttributesExtractor

# Using local files on disk (or pass no_download=True with gcs_gdb_uri / gcs_era5_climate_uri to stream in memory)
extractor = StaticAttributesExtractor(
    gdb_path="/path/to/BasinATLAS_v10.gdb",
    era5_source="hybas",
    era5_cache_dir="/path/to/era5_climate",
)
df = extractor.extract_attributes_from_file(
    input_path="basins.geojson",
    output_csv_path="caravan_attributes.csv",
    id_column="gauge_id",
)
print(df.head())

2. Extract Attributes for a Single Polygon in Python

import shapely.geometry
from multimet.static_extractor import StaticAttributesExtractor

extractor = StaticAttributesExtractor(
    gdb_path="/path/to/BasinATLAS_v10.gdb",
    era5_source="hybas",
    era5_cache_dir="/path/to/era5_climate",
)

# Polygon coordinates in (longitude, latitude)
polygon = shapely.geometry.Polygon([
    [-86.9, 40.4],
    [-86.8, 40.4],
    [-86.8, 40.5],
    [-86.9, 40.5],
    [-86.9, 40.4],
])

result = extractor.extract_attributes_for_polygon(
    polygon,
    catchment_id="my_basin_01",
    era5_source="hybas",
)

attrs = result["caravan_attributes"]
print("Drainage Area (km²):", result["total_area_km2"])
print("Mean Elevation (m):", attrs["ele_mt_sav"])
print("Mean Precipitation (mm/yr):", attrs["pre_mm_syr"])

3. Add Extracted Attributes to a Zarr Store

extractor.append_attributes_to_zarr(
    master_zarr_path="/data/multimet/caravan.zarr",
    basin_id="my_basin_01",
    attributes=attrs,
)

Checking Results Against Published Caravan Data (benchmark-static-extractor)

If you want to compare the numbers produced by this package against published Caravan values across reference basins, run benchmark-static-extractor:

# Compare all basins in a reference dataset
benchmark-static-extractor \
    --dataset /path/to/benchmark_basins_500.parquet \
    --gdb-path /path/to/BasinATLAS_v10.gdb \
    --era5-source hybas \
    --era5-cache-dir /path/to/era5_climate \
    --workers 14 \
    -o /path/to/benchmark_results/

# Quick check on a sample of 50 basins
benchmark-static-extractor \
    --dataset /path/to/benchmark_basins_500.parquet \
    --gdb-path /path/to/BasinATLAS_v10.gdb \
    --era5-source hybas \
    --era5-cache-dir /path/to/era5_climate \
    --samples 50 \
    --workers 8 \
    -o /path/to/benchmark_results/

This writes three files into --output-dir:

  • benchmark_report.md — a summary report comparing extracted values to published Caravan values.
  • benchmark_attribute_metrics.csv — one row per attribute showing correlation and error metrics.
  • benchmark_basin_metrics.csv — one row per basin showing area differences and the attribute with the largest discrepancy.

All Flags for benchmark-static-extractor

Flag Required? Default What It Does
--dataset Yes — Path to the .parquet reference dataset file.
--gdb-path Required unless --no-download None Local path to BasinATLAS_v10.gdb, shapefile, or GeoParquet file.
--era5-source Yes — How to calculate ERA5 climate numbers during the benchmark: hybas or gridded.
--output-dir, -o Yes — Folder where the benchmark report and CSV tables are saved.
--era5-cache-dir Required if hybas (unless --no-download) None Folder containing continental ERA5 climate tables.
--gridded-era5-uri Required if gridded None Google Cloud Storage (gs://...) URI or local path for the daily ERA5-Land Zarr dataset.
--gcs-gdb-uri Required if --no-download without --gdb-path None Google Cloud Storage (gs://...) URI for HydroATLAS data.
--gcs-era5-climate-uri Required if hybas with --no-download None Google Cloud Storage (gs://...) URI for continental ERA5 climate tables.
--no-download No Disabled Stream HydroATLAS and ERA5 data directly from Google Cloud Storage in memory without downloading files to local disk.
--samples No All basins Number of basins to sample if you want a quick check instead of running all basins in --dataset.
--regions, --datasets No All datasets Space-separated list of Caravan datasets to include (camels, camelsaus, camelsbr, camelscl, camelsgb, hysets, lamah).
--size-tiers No All sizes Space-separated list of basin size groups to include (1_micro, 2_small, 3_medium, 4_large, 5_macro).
--workers No 8 Number of parallel CPU processes to use.