beta(0.4.0): 45 new functions + PMTiles DataSource#33

Draft

mjohns-databricks wants to merge 443 commits into

mainfrom

This pull request is big! We're only showing the most recent 250 commits

Commits on May 30, 2026

ci: heavyweight Python build must not test lightweight (pyrx)
Michael Johns
committed
ci(pyrx-0.4.0): skip heavyweight build on pushes to this branch
Michael Johns
committed
feat(pyrx): core reproject_to_srid (tile-returning warp)
Michael Johns
committed
feat(pyrx): rst_transform + rst_to_webmercator (tile-returning)
Michael Johns
committed
feat(pyrx): core resample family + shared resampling_enum helper
Michael Johns
committed
feat(pyrx): rst_resample / rst_resample_to_size / rst_resample_to_res
Michael Johns
committed
style(pyrx): black/isort formatting on core resample + util files
Michael Johns
committed
feat(pyrx): core edit ops (clip/update_type/init_nodata)
Michael Johns
committed
feat(pyrx): rst_clip / rst_updatetype / rst_initnodata
Michael Johns
committed
feat(pyrx): core rasterize + fill_nodata (vector bridge)
Michael Johns
committed
feat(pyrx): rst_rasterize + rst_fillnodata
Michael Johns
committed
feat(pyrx): core polygonize (raster regions -> vector polygons)
Michael Johns
committed
feat(pyrx): rst_polygonize (ARRAY<struct> return)
Michael Johns
committed
feat(pyrx): explicit SQL registration of gbx_rst_* (pyspark-backed)
Michael Johns
committed
docs(pyrx): add lightweight raster API reference page
Michael Johns
committed
feat(pyrx): core spectral indices (ndvi/evi/savi/ndwi/nbr)
Michael Johns
committed
feat(pyrx): rst_ndvi/evi/savi/ndwi/nbr + SQL registration
Michael Johns
committed
feat(pyrx): core terrain slope/aspect/hillshade (NumPy Horn)
Michael Johns
committed
feat(pyrx): rst_slope/aspect/hillshade + SQL registration
Michael Johns
committed
feat(pyrx): core terrain ruggedness (tri/tpi/roughness)
Michael Johns
committed
feat(pyrx): rst_tri/tpi/roughness + SQL registration
Michael Johns
committed
feat(pyrx): core color_relief (DEM -> RGB(A) via color table)
Michael Johns
committed
feat(pyrx): rst_color_relief + SQL registration
Michael Johns
committed
deps(pyrx): add scipy (DBR 17.3) + numexpr for band math
Michael Johns
committed
feat(pyrx): core threshold + focal filter/convolve
Michael Johns
committed
feat(pyrx): rst_threshold/filter/convolve + SQL registration
Michael Johns
committed
feat(pyrx): core mapalgebra (numexpr over A,B,C... inputs)
Michael Johns
committed
feat(pyrx): rst_mapalgebra + SQL registration
Michael Johns
committed
docs(pyrx): list indices, terrain, and band-math functions in the reference
Michael Johns
committed
feat(pyrx): core derivedband (user pyfunc, GDAL pixel-func ABI)
Michael Johns
committed
feat(pyrx): rst_derivedband + SQL registration + docs
Michael Johns
committed
docs(pyrx): wire pyrx-functions page into the sidebar
Michael Johns
committed
docs(tiers): add Execution Tier availability badge component
Michael Johns
committed
docs(tiers): add 'Choosing an Execution Tier' comparison page
Michael Johns
committed
docs(tiers): elevate the lightweight tier via navbar + homepage
Michael Johns
committed
docs(tiers): add unified Raster Functions page (shared subset)
Michael Johns
committed
docs(tiers): add Heavyweight-only Raster Functions page
Michael Johns
committed
docs(tiers): cut over raster reference to the two-page tier split
Michael Johns
committed
docs(tiers): mark GridX/VectorX/PMTiles as Heavyweight tier
Michael Johns
committed
docs(tiers): tier-aware Readers/Writers overviews
Michael Johns
committed
docs(tiers): alias the lightweight tier as rx in examples
Michael Johns
committed
docs(tiers): release-notes entry for the 0.4.0 lightweight tier
Michael Johns
committed
docs(tiers): add lightweight install path + tier mention in intro
Michael Johns
committed
docs(tiers): add tier links across heavyweight usages + heavyweight-only link
Michael Johns
committed
docs(tiers): fully separate the two installation paths
Michael Johns
committed
docs(tiers): consolidate raster functions onto one page with per-function tier badges
Michael Johns
committed
docs(tiers): add Verification section to the lightweight install tab
Michael Johns
committed
docs(tiers): accurate lightweight install (Volume-staged wheel, %pip, cluster library)
Michael Johns
committed
docs(tiers): restore long-form RasterX Function Reference with per-function tier badges
Michael Johns
committed
docs(tiers): relocate VRT section under heavyweight-only; tile-structure link; dual-import usage example
Michael Johns
committed
feat(pyrx): core tiling (separate_bands/retile/to_overlapping_tiles)
Michael Johns
committed
feat(pyrx): rst_separatebands/retile/tooverlappingtiles + SQL registration
Michael Johns
committed
feat(pyrx): core make_tiles (size-based tiling)
Michael Johns
committed
feat(pyrx): rst_maketiles + SQL registration
Michael Johns
committed
docs(tiers): streamline RasterX setup into one tier-aware section
Michael Johns
committed
docs(tiers): fix SQL-setup rendering + graduate tiling functions to both tiers
Michael Johns
committed
docs(tiers): make the Usage Examples Python snippet usage-only (assume Setup)
Michael Johns
committed
docs(tiers): two-tier README blurb + lightweight-vs-heavyweight quick start
Michael Johns
committed
feat(pyrx): rio-tiler-backed rst_tilexyz + rst_xyzpyramid (web-mercator XYZ)
Michael Johns
committed
chore(pyrx): add rio-tiler==9.0.6 to dev-container deps
Michael Johns
committed
docs: refresh README known-limitations + quickstart image; tidy pyrx doc-test imports
Michael Johns
committed
docs(tiers): per-function lightweight notes + split heavyweight-only to its own page
Michael Johns
committed
feat(pyrx): h3/quadbin rastertogrid aggregations (10 functions)
Michael Johns
committed
feat(pyrx): port 24 heavyweight-only accessor/coordinate/operation functions
Michael Johns
committed
feat(pyrx): 5 tile aggregators via transparent grouped-agg + as_tile compose
Michael Johns
committed
docs(vectorx): correct stale VectorX summaries (MVT + TIN + legacy migration)
Michael Johns
committed
feat(pyrx): TIN/IDW block -- dtmfromgeoms + gridfrompoints (+ _agg)
Michael Johns
committed
docs: refresh GridX blurb (BNG+quadbin+custom), homepage cards, quick-start two tiers
Michael Johns
committed
feat(pyrx): port 6 remaining no-new-dep heavyweight-only functions
Michael Johns
committed
feat(pyrx): rst_proximity (scipy) + rst_cog_convert (rio-cogeo)
Michael Johns
committed
feat(pyrx): rst_contour + rst_viewshed -- RasterX parity complete
Michael Johns
committed

Commits on Jun 5, 2026

feat(bench): isolated uv venv for lightweight pyrx (gbx:venv:sync)
Michael Johns
committed
fix(bench): check uv exit codes in gbx:venv:sync + minor cleanup
Michael Johns
committed
feat(bench): corpus manifest model
Michael Johns
committed
feat(bench): result row schema + JSONL IO + summary
Michael Johns
committed
feat(bench): function registry with core/column call bindings
Michael Johns
committed
feat(bench): seeded single-tile generator with validity knobs
Michael Johns
committed
fix(bench): non-negative int16 range for valid spectral indices + truthful nodata comments
Michael Johns
committed
feat(bench): corpus generator, validity gate, datagen CLI
Michael Johns
committed
refactor(bench): hoist datagen imports + globally-unique row_pool cellids
Michael Johns
committed
feat(bench): gbx:bench:gen-data command
Michael Johns
committed
feat(bench): output fingerprint module + result field for consistency
Michael Johns
committed
fix(bench): coerce numpy scalars in output fingerprint for JSON safety
Michael Johns
committed
feat(bench): runner timing primitives + env capture
Michael Johns
committed
feat(bench): pure-core benchmark loop with consistency fingerprint
Michael Johns
committed
feat(bench): spark-path benchmark loop (lightweight)
Michael Johns
committed
feat(bench): runner CLI writing JSONL shard
Michael Johns
committed
feat(bench): gbx:bench:lightweight command
Michael Johns
committed
feat(bench): route gbx:test:pyrx through the isolated venv (--docker to opt out)
Michael Johns
committed
feat(bench): expand registry to ds-in accessor/terrain/band-math/warp parity fns
Michael Johns
committed

Commits on Jun 6, 2026

feat(bench): pretty-printed JSON sibling with expanded fingerprints
Michael Johns
committed
feat(bench): human-friendly summary.md with auto-derived insights
Michael Johns
committed
style(bench): apply isort/black formatting + flake8 cleanup
Michael Johns
committed
feat(bench): drop pretty-JSON output (summary.md supersedes it)
Michael Johns
committed
feat(bench): forward gbx.bench.* system properties to the test JVM
Michael Johns
committed
feat(bench): Scala BenchRow + JSONL writer matching Python schema
Michael Johns
committed
feat(bench): Scala corpus.json manifest reader
Michael Johns
committed
feat(bench): Scala GDAL per-band fingerprint matching python schema
Michael Johns
committed
feat(bench): Scala 19-function dispatch (pure-core + column + min_bands)
Michael Johns
committed
feat(bench): HeavyRunner pure-core + spark-path (warm-up, na_by_design)
Michael Johns
committed
fix(bench): harden spark-path warm-up (band-aware + non-aborting)
Michael Johns
committed
feat(bench): HeavyBenchSuite on-demand entry (system-property driven)
Michael Johns
committed
feat(bench): gbx:bench:heavyweight command
Michael Johns
committed
feat(bench): min_bands -> na_by_design parity in lightweight runner
Michael Johns
committed
feat(bench): band-aware guarded spark-path warm-up (lightweight, Scala parity)
Michael Johns
committed
feat(bench): fingerprint consistency comparison (tolerance, dtype/nodata-count rules)
Michael Johns
committed
fix(bench): wire abs_tol into fingerprint closeness (avoid false-divergent near-zero stats)
Michael Johns
committed
feat(bench): join shards into per-cell speedup + consistency
Michael Johns
committed
feat(bench): comparison.csv + cross-API summary with insights
Michael Johns
committed
feat(bench): compare CLI writing comparison.csv + summary.md
Michael Johns
committed
feat(bench): gbx:bench:compare command
Michael Johns
committed
feat(bench): note likely nodata/border cause on divergent neighborhood-op cells
Michael Johns
committed
feat(bench): gbx:bench:all full-pipeline orchestration
Michael Johns
committed
build(bench): test-jar goal to ship bench Scala classes (geobrix-0.4.0-tests.jar)
Michael Johns
committed
feat(bench): HeavyBenchMain py4j-callable cluster entry
Michael Johns
committed
fix(bench): idempotent cluster-safe GDAL init in HeavyBenchMain (pure-core driver path)
Michael Johns
committed
feat(bench): bench_results Delta helper (rows_to_dataframe + to_delta)
Michael Johns
committed
feat(bench): cluster bench-notebook builder (lightweight in-proc + heavyweight via py4j)
Michael Johns
committed
feat(bench): gbx:bench:cluster submitter + command (cluster-gated)
Michael Johns
committed
fix(bench): guard mutually-exclusive --heavyweight-only/--lightweight-only before submit
Michael Johns
committed
feat(pyrx): shared NoData/edge helpers (read_masked, propagate_invalid, emit)
Michael Johns
committed
fix(pyrx): terrain ops mask input NoData + mark kernel border NoData (GDAL parity)
Michael Johns
committed
fix(pyrx): band-math ops mask input NoData (ndvi/ndwi/nbr/savi/evi/index)
Michael Johns
committed
fix(pyrx): mapalgebra masks input NoData + sets output NoData
Michael Johns
committed
fix(pyrx): tri uses GDAL Riley formula; hillshade floors valid to 1 (GDAL parity)
Michael Johns
committed
fix(pyrx): hillshade uses GDAL gdaldem rational form (heavy/light parity)
Michael Johns
committed
feat(pyrx): _gdaldem_scale helper replicating GDAL 3.11 auto-scale
Michael Johns
committed
feat(pyrx): slope/aspect/hillshade auto-scale from CRS (GDAL-normal)
Michael Johns
committed
fix(pyrx): rst_slope binding drops scale, adds xscale/yscale (auto-scale default)
Michael Johns
committed
fix(rasterx): RST_Slope omits -s unless explicit (GDAL auto-scale default)
Michael Johns
committed
fix(rasterx): Python rst_slope default omits scale (GDAL auto-scale parity)
Michael Johns
committed

Commits on Jun 7, 2026

test(bench): slope auto-scales (drop masking scale=1.0); cover projected CRS
Michael Johns
committed
fix(pyrx): aspect is scale-invariant to match gdaldem (drop xscale/yscale)
Michael Johns
committed
test(bench): comparator near-zero abs floor (fix false-divergent aspect min)
Michael Johns
committed
docs: slope/hillshade auto-scale from CRS (release notes)
Michael Johns
committed
test(bench): symmetric per-engine summaries; mpix/s + tolerance legend in summary.md
Michael Johns
committed
docs: add Benchmarking page (cluster + local) + execution-tiers summary
Michael Johns
committed
chore: gitignore generated bench-corpus sample data
Michael Johns
committed
Merge pull request #35 from databrickslabs/pyrx-0.4.0
mjohns-databricks
authored
feat(bench): core/full selection sets (--set) + registered_rst coverage helper
Michael Johns
committed
fix(bench): core set = all current representative functions (not a 5-fn subset)
Michael Johns
committed
feat(bench): scalar accessor coverage (15 fns)
Michael Johns
committed
feat(bench): coordinate accessor coverage (7 fns)
Michael Johns
committed
feat(bench): timing-only fingerprint flag + map/struct coverage (6 fns); retrofit prior downgrades
Michael Johns
committed
feat(bench): tile-out transforms with scalar args (13 fns)
Michael Johns
committed
feat(bench): tile-out transforms with geometry/expression args (10 fns) — bucket E complete
Michael Johns
committed
feat(bench): coverage & parity scorecard in summary.md
Michael Johns
committed
docs(bench): document --set core|full; fix full-set harness bugs (clip crash, mapalgebra spec)
Michael Johns
committed
fix(bench): clean pure-core runs for asformat/initnodata/filter (heavy) + clip/viewshed (light geom)
Michael Johns
committed
fix(rasterx/bench): align rst_threshold/rst_derivedband band count (heavy/light parity)
Michael Johns
committed
fix(bench): heavy runner flushes JSONL per function (crash-resilient shards)
Michael Johns
committed
fix(pyrx): focal NoData-skip + edge window-shrink to match GDALBlock
Michael Johns
committed
fix(pyrx): convolve edge back to nearest (un-regress); filter NoData fix stands
Michael Johns
committed
test(pyrx): cover focal filt min/max (NoData-skip + edge shrink)
Michael Johns
committed
feat(bench): FnSpec.sources dependency map for all registered functions
Michael Johns
committed
feat(bench): authoritative per-function store + change->function resolution
Michael Johns
committed
style(pyrx): black-format focal min/max test lines
Michael Johns
committed
feat(bench): gbx:bench:changed — benchmark only functions affected by current changes
Michael Johns
committed
feat(bench): gbx:bench:seed (bootstrap store) + gbx:bench:clean (prune runs/orphans/all)
Michael Johns
committed
feat(bench): gbx:bench:status (scorecard from store) + pre-push staleness warning
Michael Johns
committed
feat(bench): cover C1 readers + C2 subdataset fns
Michael Johns
committed
feat(bench): cover C3 multi-tile fns (tile_array adapter)
Michael Johns
committed
feat(bench): cover C4 tiling fns (raster_collection fingerprint)
Michael Johns
committed

Commits on Jun 8, 2026

fix(bench): materialize C3 synth tiles at gen-data (heavy runs first)
Michael Johns
committed
feat(bench): add heavy-light Δms and Δ% columns to summary
Michael Johns
committed
fix(rasterx): combineavg preserves band count (per-band mean)
Michael Johns
committed
fix(pyrx): maketiles uses power-of-4 split to match heavy
Michael Johns
committed
fix(pyrx): tooverlappingtiles overlap is percentage to match heavy
Michael Johns
committed
fix(pyrx): merge overlap is last-wins to match heavy GDAL VRT
Michael Johns
committed
test(bench): xyzpyramid count-only (render-engine-specific pixels)
Michael Johns
committed
docs: note v0.3.0 behavior changes (initnodata, derivedband)
Michael Johns
committed
fix(rasterx): maketiles memSize robust for in-memory datasets
Michael Johns
committed
fix(bench): tooverlappingtiles default overlap matches pyrx spec (25)
Michael Johns
committed
feat(bench): dggs_grid + vector fingerprint kinds
Michael Johns
committed
chore(bench): readable run-dir names (seed/changed-<ts>-<set>)
Michael Johns
committed
feat(bench): cover B-grid DGGS fns (h3/quadbin grids + tessellate)
Michael Johns
committed
feat(bench): cover B-vec fns (contour, polygonize) -- bucket B done
Michael Johns
committed
fix(rasterx): combineavg must exclude NoData from per-band divisor
Michael Johns
committed
test(bench): exclude on-demand HeavyBenchSuite from default/CI run
Michael Johns
committed
fix(pyrx): polygonize must group integer regions like GDAL
Michael Johns
committed
fix(bench): vector comparator gates on measure, count informational
Michael Johns
committed
test(bench): skip bench collection when pyrx deps absent (CI)
Michael Johns
committed
fix(rasterx): tessellate must include fringe cells clipping raster edge
Michael Johns
committed
feat(bench): gbx:bench:changed --functions for explicit re-validation
Michael Johns
committed
fix(bench): heavyweight on-demand override needs non-empty exclude tag
Michael Johns
committed
fix(pyrx): polygonize rounds to int32 like GDAL, not truncate
Michael Johns
committed
fix(pyrx): tessellate must drop zero-coverage fringe cells
Michael Johns
committed
feat(bench): per-FnSpec rel_tol override; contour=0.02
Michael Johns
committed
revert: heavy tessellate kRing widening (redundant)
Michael Johns
committed
perf(pyrx): vectorize focal min/max/median (drop generic_filter)
Michael Johns
committed
perf(pyrx): vectorize raster_to_grid cell-id + group-reduce
Michael Johns
committed
feat(perf): gbx:perf:vectorscan standing check for pyrx core
Michael Johns
committed
feat(bench): deterministic geometry corpus + geometry input_kind
Michael Johns
committed
test(bench): update H3/quadbin parity pins to vectorized cells API
Michael Johns
committed
feat(bench): cover geometry-in fns (bucket D)
Michael Johns
committed
feat(bench): Spark groupBy aggregator harness (bucket A)
Michael Johns
committed
style: apply black formatting to two test files
Michael Johns
committed
style: match CI black for test_vector_raster_bridge
Michael Johns
committed
docs: favor lightweight pyrx tier across homepage + quickstart + tiers
Michael Johns
committed
fix(bench): gridfrompoints IDW parity via all-points max_pts
Michael Johns
committed
fix(bench): run aggregators in spark-path mode in bench:changed
Michael Johns
committed
docs: hold tier-consistency claim pending validation
Michael Johns
committed
fix(pyrx): proximity matches GDAL integer-domain targets
Michael Johns
committed
docs: accurate consistency — call out convolve edge difference
Michael Johns
committed
fix(bench): route heavy spark-path GDAL init through locked GDALManager
Michael Johns
committed
fix(bench): single-partition heavy aggregate to dodge local GDAL race
Michael Johns
committed
fix(bench): unset JAVA_TOOL_OPTIONS so heavy bench JVM avoids JDWP clash
Michael Johns
committed
revert: redundant JAVA_TOOL_OPTIONS unset in bench:heavyweight
Michael Johns
committed
fix(rasterx): deterministic fold order for merge/rasterize aggregators
Michael Johns
committed
fix(agg): order merge fold by raw content bytes, drop vsimem tiebreak
Michael Johns
committed
docs: complete benchmarking results (107 fns) + precise consistency
Michael Johns
committed
perf(pyrx): idw_grid all-points path via cdist, skip cKDTree
Michael Johns
committed
docs: refresh benchmarking table from current store (gridfrompoints fix)
Michael Johns
committed
docs: benchmarking table to 512² per fn + speedup ordering note
Michael Johns
committed
fix(pyrx): namespace packages + typing_extensions pin for cluster install
Michael Johns
committed
fix(pyrx): pin httpcore>=1.0.9 to avoid on-cluster h11 conflict
Michael Johns
committed
fix(bench): lit-wrap string args so 3 fns run in spark-path
Michael Johns
committed
feat(bench): cluster run truncate + status-led output + row-tile-px
Michael Johns
committed
feat(bench): stream interim results to Delta + live host polling
Michael Johns
committed
feat(bench): tighten cluster poll to 5s + show "(of N)" progress
Michael Johns
committed

Commits on Jun 9, 2026

fix(bench): run heavy tier on a UC Volume (Spark reads) + real srid
Michael Johns
committed
fix(bench): heavy-vs-light comparison works from installed wheel
Michael Johns
committed
feat(bench): cluster streaming + per-run table + inline summaries
Michael Johns
committed
fix(rasterx): route RST_FillNodata scratch files to MEM driver
Michael Johns
committed
feat(bench): explain-only mode, agg parallelism fix, iter_ metrics
Michael Johns
committed

Commits on Jun 10, 2026

perf(bench): parallelize heavy spark-path + aggregator (drop coalesce)
Michael Johns
committed
perf(pyrx): stream combineavg_agg reducer + fix _open_all leak
Michael Johns
committed
perf(bench): parallelize light agg spark-path + bound fan-out
Michael Johns
committed
feat(bench): --redo-functions + collapse setup cell + resume scoping
Michael Johns
committed
test(pyrx): cover combineavg streaming + _open_all corrupt tile
Michael Johns
committed
feat(bench): second-scale metrics + summary.md path render
Michael Johns
committed
fix(bench): serialize heavy geometry aggregators (GDAL race)
Michael Johns
committed
refactor(bench): extract heavy ms->s remap to a tested module fn
Michael Johns
committed
chore(qc): exclude internal bench package from docs-match-code
Michael Johns
committed
test(bench): cover agg fan-out (tile parallel, geometry serial)
Michael Johns
committed
feat(bench): move headline timing cols right after mode
Michael Johns
committed
perf(bench): minimal warm-up for lightweight aggregator spark-path
Michael Johns
committed
perf(bench): minimal warm-up for heavy spark-path + aggregators
Michael Johns
committed
refactor(rasterx): thread-safe OGR registration via GDALManager.initOgr
Michael Johns
committed
perf(bench): un-serialize geometry aggregators + inline summary render
Michael Johns
committed
fix(bench): render compare summary inline -- exit in its own cell
Michael Johns
committed
fix(bench): --redo-functions re-runs its fns, not just purges them
Michael Johns
committed
chore(bench): default pure-core to 1 warmup / 3 measured
Michael Johns
committed
docs(pyrx): note viewshed first-call numba compile cost
Michael Johns
committed
perf(rasterx): windowed-read retile replaces per-tile gdal.Translate
Michael Johns
committed

Commits on Jun 11, 2026

fix(bench): normalize heavy jsonl ms->s in read_jsonl
Michael Johns
committed
test(pyrx): guard against Spark config mutation (Serverless target)
Michael Johns
committed
fix(bench): refuse spark-path run when row pool < --row-counts
Michael Johns
committed
docs(bench): refresh benchmarking to 1024² + cluster spark-path results
Michael Johns
committed
test(bench): cover _normalize_row heavyweight ms->s deserialization
Michael Johns
committed