Skip to content

feat: support for embedded files - #316

Open
b8raoult wants to merge 5 commits into
mainfrom
feat/embedded-files
Open

feat: support for embedded files#316
b8raoult wants to merge 5 commits into
mainfrom
feat/embedded-files

Conversation

@b8raoult

@b8raoult b8raoult commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

Description

Embedded files in Anemoi checkpoints

Anemoi checkpoints are zip archives that bundle model weights together with
metadata and supporting arrays. This branch adds the ability to embed
arbitrary files
inside a checkpoint, list them, extract them, remove them,
and read them back transparently at runtime through a new checkpoint://
URI scheme.

The goal is to create standalone checkpoint file for inference. Imagine that we add the bathymetry as NetCDF in a checkpoint, we should be able to have the following run config:

checkpoint: my-checkpoint.ckpt

input:
    grib:
       path: /path/to/data.grib
       
constant-forcing:
   netcdf:
       path: checkpoint://bathymetry.nc
      

Embedded files are stored inside the checkpoint zip under a dedicated folder:

<top-level-dir>/anemoi-embedded-files/<name>

A new embedded-files CLI

The command is exposed through the anemoi-utils entry point. Exactly one
action (--list, --add, --remove, --extract, or --cat) must be given.

anemoi-utils embedded-files PATH (--list | --add | --remove | --extract | --cat NAME)
                            [--file FILE [FILE ...]]
                            [--overwrite]
                            [--directory DIR]

List embedded files

anemoi-utils embedded-files my-checkpoint.ckpt --list

Prints each embedded file name followed by its size and modification time, e.g.:

config.yaml {
  "size": 1234,
  "modified_time": "2026-06-17 10:42:00"
}

Add files

The file's base name is used as the embedded name. You cannot embed the
checkpoint into itself.

# Add one or more files
anemoi-utils embedded-files my-checkpoint.ckpt --add --file config.yaml notes.txt

# Overwrite an embedded file that already exists
anemoi-utils embedded-files my-checkpoint.ckpt --add --file config.yaml --overwrite

Without --overwrite, adding a file whose name already exists raises an error.

Remove files

Remove embedded files by name:

anemoi-utils embedded-files my-checkpoint.ckpt --remove --file notes.txt config.yaml

Extract files

Extract by name to the current directory, or to a chosen directory:

# Extract to the current working directory
anemoi-utils embedded-files my-checkpoint.ckpt --extract --file config.yaml

# Extract to a specific directory, overwriting if present
anemoi-utils embedded-files my-checkpoint.ckpt --extract --file config.yaml \
    --directory ./extracted --overwrite

Extraction fails if a target file already exists unless --overwrite is given,
and fails if a requested name is not present in the checkpoint.

Read via the checkpoint:// scheme (--cat)

--cat is a test/demo of the checkpoint:// resolution: it resolves
checkpoint://<name> to a local path through checkpoint_file(...) and prints
the decoded contents to stdout.

anemoi-utils embedded-files my-checkpoint.ckpt --cat checkpoint://config.yaml

What problem does this change solve?

What issue or task does this change relate to?

Additional notes

As a contributor to the Anemoi framework, please ensure that your changes include unit tests, updates to any affected dependencies and documentation, and have been tested in a parallel setting (i.e., with multiple GPUs). As a reviewer, you are also responsible for verifying these aspects and requesting changes if they are not adequately addressed. For guidelines about those please refer to https://anemoi.readthedocs.io/en/latest/

By opening this pull request, I affirm that all authors agree to the Contributor License Agreement.

@b8raoult
b8raoult marked this pull request as ready for review June 17, 2026 09:33
@b8raoult
b8raoult requested review from HCookie and gmertes June 17, 2026 09:33
@HCookie HCookie moved this from To be triaged to Reviewers needed in Anemoi-dev Jun 22, 2026
@github-actions github-actions Bot added the enhancement New feature or request label Jun 25, 2026
@github-actions github-actions Bot added the tests label Jun 25, 2026
@MeraX

MeraX commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

What problem does this change solve?

Is this idea just about storing the data of constant forcing with the checkpoint or are other files considered as well? For the latter, did you consider storing the respective tensors directly in the model?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Reviewers needed

Development

Successfully merging this pull request may close these issues.

3 participants