to_neural_lam() to wmg.save for neural-lam tensor-on-disk format#123
to_neural_lam() to wmg.save for neural-lam tensor-on-disk format#123prajwal-tech07 wants to merge 5 commits intomllam:mainfrom
Conversation
…ormat Add o_neural_lam() function that saves graph components to the tensor-on-disk format expected by eural_lam.utils.load_graph(). Supports all three archetypes: - Keisler (flat single-scale) - GraphCast (flat multiscale) - Oskarsson (hierarchical) Output files: - g2m/m2g: single tensors (edge_index, features) - m2m: lists of tensors per level - mesh_features: position-normalized node features per level - mesh_up/mesh_down: hierarchical inter-level edges (when hierarchical=True) Edge features are written raw (unnormalized) - neural-lam normalizes at load time. Mesh node positions are pre-normalized by max(abs(pos)). Includes comprehensive tests covering all 3 archetypes, shape validation, dtype checks, normalization verification, and edge cases. Refs: mllam/neural-lam#384
88fdd98 to
c2e7b5e
Compare
- Apply black formatting to save.py and test_save.py - Remove unused 'import numpy as np' from test_save.py (flake8 F401)
|
Fixed the pre-commit linting failure in a1ec5b5:
All pre-commit hooks and tests (24/24) pass locally now. The CI should be green on the next run. |
Reverts the strict ==2.3.1 pin back to the flexible >=2.5.3 version requirement matching upstream main, so we don't restrict users to a specific version.
|
Reverted the strict All pre-commit hooks and tests (24/24) pass. |
Rename the save function to better describe what it does (saves torch tensors to disk) rather than namespacing it to neural-lam specifically. Updates all references in tests accordingly.
Follow rename in weather-model-graphs (mllam/weather-model-graphs#123).
|
Renamed to_neural_lam → The corresponding neural-lam PR (mllam/neural-lam#596) has been updated to use the new function name as well. |
Describe your changes
Add a
to_neural_lam()function towmg.savethat serializes graph components (fromwmg.create.archetype.*(..., return_components=True)) to the tensor-on-disk.ptformat expected byneural_lam.utils.load_graph().This is the first step in bridging
weather-model-graphsandneural-lam— putting the serialization logic in wmg so that neural-lam's CLI can remain a thin wrapper.What it does:
{"g2m", "m2m", "m2g"}dict returned by any archetype withreturn_components=True.ptfiles (g2m/m2g as single tensors, m2m/mesh_features as lists)max(abs(pos)), matching the existing neural-lam conventionSupports all three archetypes:
Also adds two internal helpers (
_graph_to_edge_tensors,_graph_to_node_features) that cleanly strip heterogeneous node/edge attributes before PyG conversion.Tests: 22 new tests across 4 test classes covering all archetypes, shape/dtype validation, normalization checks, and edge cases.
No new dependencies — uses the existing
[pytorch]optional extras (torch, torch-geometric).Issue Link
Refs mllam/neural-lam#384
Type of change
Checklist before requesting a review
pullwith--rebaseoption if possible).Checklist for reviewers
Each PR comes with its own improvements and flaws. The reviewer should check the following:
Author checklist after completed review
reflecting type of change (add section where missing):
Checklist for assignee