Skip to content
Open
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
d00838c
Start with data_module_template
jnnr Apr 17, 2025
382e994
Set up rules, env and scripts for capacity factors
jnnr Apr 17, 2025
8995967
Delete dummy rule
jnnr Jun 18, 2025
393d8f8
Set first column of spatial units as index
jnnr Jun 18, 2025
7f6689d
Adapt rule's wildcards
jnnr Jun 18, 2025
62f7c95
Use the same script for wind and pv capacityfactors
jnnr Jun 18, 2025
622861c
Allow use of point/raster layout
jnnr Jun 18, 2025
177c28b
Enable configuration to download cutout
jnnr Jun 18, 2025
1c2d22c
Move download rule to automatic.smk
jnnr Jun 20, 2025
8f490a2
Add simple unpolished plot and map
jnnr Jun 20, 2025
b22620a
Attempt to solve missing file when integrating data module via GitHub
jnnr Jun 20, 2025
7451085
Update template to v0.1.13
jnnr Jun 20, 2025
c203a62
Add integration test
jnnr Jun 20, 2025
cd3c2b4
Harmonize paths between rules
jnnr Jun 20, 2025
16e0d36
Force latest atlite 0.4.1 and cfgrib to handle new cdsapi
jnnr Aug 8, 2025
dccd8c0
Adhere to new atlite api
jnnr Aug 8, 2025
88d9dfa
Adapt and make schema stricter
jnnr Aug 8, 2025
5251b4c
Adapt example and test config
jnnr Aug 8, 2025
f9a6b00
Allow to pass monthly_requests
jnnr Aug 8, 2025
ecf5c99
Set masked to True when reading raster layouts
jnnr Aug 8, 2025
1f2e8f8
Set nan values in layout to zero
jnnr Oct 20, 2025
83b4780
Move atlite call to backend
jnnr Nov 3, 2025
3274159
Wrap processing in functions
jnnr Nov 3, 2025
51b9e08
Drop pre-defined keyword argument in function call
jnnr Nov 3, 2025
0278af0
Add plots
jnnr Nov 5, 2025
e88b910
Use parquet files for shapes
jnnr Nov 5, 2025
b5d37b3
Call function with correct argument
jnnr Nov 5, 2025
bcace4d
Describe interface
jnnr Nov 5, 2025
0579889
Drop separate plot rule as plots are already created alongside data
jnnr Nov 6, 2025
eec3901
Validate schemas of point layout and shapes
jnnr Nov 6, 2025
9fe6e8b
Configure examples
jnnr Nov 6, 2025
7548372
Extend README
jnnr Nov 12, 2025
9c88280
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 12, 2025
7edc44d
Commit pixi.lock and update gitignore
sjpfenninger Nov 12, 2025
ad7e446
Update to newest template; fix INTERFACE and integration Snakefile
sjpfenninger Nov 12, 2025
187e6e0
Define default target after module re-import
jnnr Nov 13, 2025
c358ceb
Merge branch 'main' into feature-module-setup
sjpfenninger Nov 13, 2025
71268ac
Delete obsolete files
jnnr Nov 13, 2025
16f85bc
Prepend _ to follow naming convention for python modules
jnnr Nov 13, 2025
212491c
Delete dummy content
jnnr Nov 13, 2025
9d83d7a
Fix name of python module when adding to source path
jnnr Nov 13, 2025
1ba5f13
Make sure all python modules are in source_path
jnnr Nov 14, 2025
1bfb4c3
Fix reference
jnnr Nov 14, 2025
f16dfe3
Relative import
jnnr Nov 14, 2025
23f559c
Update tests/integration/test_config.yaml
jnnr Nov 14, 2025
bde2d46
Copier update and change license to Apache
jnnr Jun 2, 2026
b01f549
Drop user/automatic from INTERFACE.yaml
jnnr Jun 2, 2026
07caabb
Rename spatial_units as shapes
jnnr Jun 2, 2026
2c9f739
Wrap download into function
jnnr Jun 2, 2026
5a24951
Rename functions for clarity
jnnr Jun 2, 2026
82eb839
Not define new variables for techs_specs
jnnr Jun 2, 2026
1402f4e
Generic naming
jnnr Jun 2, 2026
a1b65c1
Write specific error message
jnnr Jun 2, 2026
8c96e3e
Rename test to conform to new template
jnnr Jun 8, 2026
c4ec7c6
Run copier recopy
jnnr Jun 8, 2026
dd43ce3
Delete dummy script
jnnr Jun 8, 2026
257b561
Use pathvars
jnnr Jun 9, 2026
2e2d502
Bump pandera to handle errors during validation
jnnr Jun 9, 2026
0a6bae8
Avoid using ancient on output
jnnr Jun 10, 2026
b8c9dc2
Refine plots
jnnr Jun 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .copier-answers.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Changes here will be overwritten by Copier
# DO NOT MANUALLY MODIFY THIS FILE
_commit: latest
_src_path: https://github.com/calliope-project/data-module-template.git
author_email: j.a.c.launer@tudelft.nl
author_family_name: Launer
author_given_name: Jann
github_org: calliope-project
license: MIT
module_description: This data module produces capacity factors for PV and wind at
arbitrary spatial resolution.
module_long_name: PV and wind capacity factors
module_short_name: module_pv_wind
22 changes: 22 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Documentation: https://EditorConfig.org
# Inspired by Django .editorconfig file

root = true

[*]
indent_style = space
indent_size = 4
trim_trailing_whitespace = true
end_of_line = lf
insert_final_newline = true
charset = utf-8

[*.{yaml,yml,.cff}]
indent_size = 2

[*.md]
trim_trailing_whitespace = true
indent_size = 4

[*.{diff,patch}]
trim_trailing_whitespace = false
29 changes: 29 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
### OS-specific
.DS_Store

### Editors
*~
\#*\#
**/.vscode
.$*

### Byte-compiled files
__pycache__
*.pyc

### Environments
.pixi/

### Snakemake
.snakemake/
gurobi.log

### Notebooks
.ipynb_checkpoints

# Ignore mutable folders
**/logs/*
**/resources/*
**/results/*
**/temp/*
**/tmp/*
47 changes: 47 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
default_language_version:
python: python3

repos:
# Generic 'file quality' testing
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
# Machine-friendliness
- id: trailing-whitespace
- id: end-of-file-fixer
# Repo quality
- id: check-added-large-files
args: [--enforce-all]
- id: forbid-submodules
- id: check-case-conflict
- id: check-illegal-windows-names
- id: check-merge-conflict
- id: debug-statements
# Detect config. file issues
- id: check-json
- id: check-yaml
args: [--unsafe]

# Python file formatting
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.9.6
hooks:
- id: ruff
args: [--fix, --exit-non-zero-on-fix]
- id: ruff-format

# Snakemake file formatting
- repo: https://github.com/snakemake/snakefmt
rev: v0.10.2
hooks:
- id: snakefmt

# Spelling
- repo: https://github.com/codespell-project/codespell
rev: v2.4.1
hooks:
- id: codespell
files: .*\.(py|smk|md)$|^Snakefile$

ci: # https://pre-commit.ci/
autoupdate_schedule: monthly
19 changes: 19 additions & 0 deletions .readthedocs.yaml
Comment thread
jnnr marked this conversation as resolved.
Outdated
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
version: 2

build:
os: ubuntu-22.04
tools:
python: "3.12"
jobs:
install:
- asdf plugin add pixi
- asdf install pixi latest
- asdf global pixi latest
build:
html:
- pixi run build-docs -d $READTHEDOCS_OUTPUT/html

mkdocs:
configuration: mkdocs.yaml
5 changes: 5 additions & 0 deletions .snakemake-workflow-catalog.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Integration with https://snakemake.github.io/snakemake-workflow-catalog/#
usage:
software-stack-deployment:
conda: true
report: false
7 changes: 7 additions & 0 deletions AUTHORS
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
This is the list of contributors for copyright purposes.

This does not necessarily list everyone who has contributed to this software's
code or documentation. For a full contributor list, see:
<https://github.com/calliope-project/module_pv_wind/graphs/contributors>

Jann Launer, <j.a.c.launer@tudelft.nl>
11 changes: 11 additions & 0 deletions CITATION.cff
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Citation file
# For more information, see:
# https://citation-file-format.github.io/
cff-version: 1.2.0
message: If you use this software or data produced by it, please cite it using the metadata from this file.
title: "clio - module_pv_wind: PV and wind area potentials and capacity factors "
repository: "https://github.com/calliope-project/module_pv_wind"
license: MIT
authors:
- given-names: Jann
family-names: Launer
17 changes: 17 additions & 0 deletions INTERFACE.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Module Input-Output structure for automated doc. generation
resources:
user:
"cutout_{name_cutout}.nc": Cutout of meteorological data provided by user.
"tech_specs/{name_tech}.yaml": Technology specification of PV/wind.
"layout/{name_layout}.tif": Raster layout file describing the spatial distribution of capacity.
"layout/{name_layout}.parquet": Point layout file describing the spatial distribution of capacity.
"spatial_units/{name_spatial_units}.parquet": Geometries of spatial units (e.g., administrative regions).
Comment thread
jnnr marked this conversation as resolved.
Outdated
automatic:
Comment thread
jnnr marked this conversation as resolved.
Outdated
"cutout_era5.nc": Cutout of meteorological data downloaded from ERA5.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the auto-downloaded ERA5 data have at least one wildcard in its path?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the filename could reflect both spatial and temporal scope.

However, as I argued above, better not tie this to the shapefiles. You will re-download large files if you want to analyse the same spatial scope at different resolutions (NUTS0, NUTS3, etc.). Better: Use a wildcard, but take that wildcard from the config, as we just discussed in person.

results:
"{name_cutout}/{name_spatial_units}/{name_layout}/capacityfactors_{name_tech}.nc": Capacity factors for the specified technology.
wildcards:
name_cutout: Cutout name
name_tech: Technology name
name_layout: Layout name
name_spatial_units: Spatial units name
21 changes: 21 additions & 0 deletions LICENSE
Comment thread
jnnr marked this conversation as resolved.
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2025 AUTHORS

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
41 changes: 41 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# PV and wind capacity factors

This data module produces capacity factors for PV and wind at arbitrary spatial resolution.

A modular `snakemake` workflow built for [`clio`](https://clio.readthedocs.io/) data modules.

## Using this module

This module can be imported directly into any `snakemake` workflow.
Please consult the integration example in `tests/integration/Snakefile` for more information.

## Development

We use [`pixi`](https://pixi.sh/) as our package manager for development.
Once installed, run the following to clone this repo and install all dependencies.

```shell
git clone git@github.com:calliope-project/module_pv_wind.git
cd module_pv_wind
pixi install --all
```

For testing, simply run:

```shell
pixi run test-integration
```

To view the documentation locally, use:

```shell
pixi run serve-docs
```

To test a minimal example of a workflow using this module:

```shell
pixi shell # activate this project's environment
cd tests/integration/ # navigate to the integration example
snakemake --use-conda --cores 2 # run the workflow!
```
9 changes: 9 additions & 0 deletions config/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Configuration

This workflow is part of the [clio project](https://clio.readthedocs.io/en/latest/).
Please consult our documentation for more details.
Other useful resources are:

- `INTERFACE.yaml`: user input files (placed in `resources/user`) and module output files (placed in `results`).
- `workflow/internal/config.schema.yaml`: general configuration options.
- `tests/integration/`: a simple example of how to use this module.
11 changes: 11 additions & 0 deletions config/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# A minimal example of how to configure this module
download_cutout: True
backend: atlite
layout: raster # raster or point
aggregate: true
cutout_params:
module: era5
x: [3.4, 7.2]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the x and y bounds come from the input geometries (I guess spatial_units/{name_spatial_units}.parquet)?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That could be done. However, if you need Europe in NUTS0, NUTS2, NUTS3 and for several years, you will download a big dataset several times.

y: [50.7, 53.5]
time: ["2017-01", "2017-02"]
features: ['height', 'wind', 'influx', 'temperature']

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be auto-populated based on what technologies need to be run?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally yes. But as the variables are all united in one atlite.Cutout object, you would probably rerun the download if techs have partial overlaps. You can also not specify the features in atlite, but I decided to limit to these, as they are all that are needed for pv and wind, as far as I know.

4 changes: 4 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Home

Welcome to the documentation of the `module_pv_wind` data module!
Please consult the [specification guidelines](./specification.md) and the [`clio` documentation](https://clio.readthedocs.io/) for more information.
67 changes: 67 additions & 0 deletions mkdocs.yaml
Comment thread
jnnr marked this conversation as resolved.
Outdated
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# yaml-language-server: $schema=https://squidfunk.github.io/mkdocs-material/schema.json

site_name: PV and wind area potentials and capacity factors
site_url: https://module_pv_wind.readthedocs.io/
repo_name: module_pv_wind
repo_url: https://github.com/calliope-project/module_pv_wind/
edit_uri: edit/main/docs/
theme:
name: material
features:
- navigation.footer
- navigation.instant
- navigation.tabs
- navigation.tabs.sticky
- navigation.indexes
- search.suggest
- search.highlight
- toc.follow
palette:
# Palette toggle for automatic mode
- media: "(prefers-color-scheme)"
toggle:
icon: material/brightness-auto
name: Switch to light mode
primary: red
accent: pink
# Palette toggle for light mode
- media: "(prefers-color-scheme: light)"
scheme: default
toggle:
icon: material/weather-sunny
name: Switch to dark mode
primary: red
accent: pink
# Palette toggle for dark mode
- media: "(prefers-color-scheme: dark)"
scheme: slate
toggle:
icon: material/weather-night
name: Switch to system preference
primary: red
accent: pink
markdown_extensions:
- admonition
- pymdownx.arithmatex:
generic: true
- pymdownx.betterem:
smart_enable: all
- pymdownx.details
- pymdownx.emoji:
emoji_index: !!python/name:material.extensions.emoji.twemoji
emoji_generator: !!python/name:material.extensions.emoji.to_svg
- pymdownx.highlight:
anchor_linenums: true
- pymdownx.snippets:
check_paths: true
restrict_base_path: true
- pymdownx.superfences:
custom_fences:
- name: mermaid
class: mermaid
format: !!python/name:mermaid2.fence_mermaid_custom
- toc:
anchorlink: true
plugins:
- search
copyright: Copyright &copy; since 2025 <a href="https://github.com/calliope-project/module_pv_wind/blob/main/AUTHORS">module_pv_wind contributors</a> (MIT licensed)
3 changes: 3 additions & 0 deletions mypy.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[mypy]
disable_error_code = import-untyped
exclude = (^|/)\.(snakemake|pixi)(/|$)
Loading