Replace ccpp-prebuild and ccpp-capgen v0 with ccpp-capgen v1 by climbfuji · Pull Request #762 · NCAR/ccpp-framework

climbfuji · 2026-06-30T21:59:31Z

Description

This PR replaces the two code generators ccpp-prebuild and ccpp-capgen (version 0) with the new, unified ccpp-capgen (version 1). It supports all host models (CAM-SIMA, CCPP-SCM, NEPTUNE, UFS) and combines the features from ccpp-capgen v0 with the performance and operational robustness of ccpp-prebuild.

This pull request will be updated continuously with links to updates of the CCPP technical documentation and additional information. See also #759 for background information and further materials.

User interface changes?: Yes (see documentation)

This feature branch feature/capgen-v1 will support the three ccpp-prebuild-based models CCPP-SCM, NEPTUNE, and UFS, which will transition to ccpp-capgen v1 before CAM-SIMA. Once CAM-SIMA transitions, this PR will be merged and feature/capgen-v1 will be locked.

Todo

The LLM-generated documentation and materials in doc/ need to be reviewed and used to update the existing CCPP technical documentation (https://github.com/ncar/ccpp-doc). Once that process is complete, we can remove or trim down the additional documentation in doc/.
The LLM-generated code should be reviewed file-by-file. LLMs are known for their verbosity, and we can likely remove or trim down sizeable part sof the code. Open question: Should we do the same for the unit tests?

Issues

To check/may not be implemented yet:

Testing

See #759

…ation

…v17p8 suite

… feature/capgen-ng

… table

…g sites)

…gen-ng: entire constituent state now per instance

@dustinswales

…ts (#764) This PR targets the feature/capgen-v1 branch with follow-up and cleanup changes from manual code reviews (capgen/metadata/parse-tools/*) plus a few selected, minor improvements. This branch is being used by the CAM-SIMA feature branches that replace the original capgen v0 with the new capgen v1. - Temporary: run CI tests also for pull requests against branch feature/capgen-v1. - Simplify capgen/metadata/parse_tools/parse_source.py. Some of the removed features were (ab)used by CAM-SIMA and are now in the updated capgen compatibility layer in the CAM-SIMA feature branch. - Simplify capgen/metadata/parse_tools/parse_log.py. Requires a tiny change in the CAM-SIMA compatibility layer (now in the (CAM-SIMA feature branch). - Add documentation, simplify, and add a comment about a latent bug that doesn't hurt is today w.r.t. escaping periods in decimals in capgen/metadata/parse_tools/fortran_conditional.py. - capgen/metadata/parse_tools/io_helpers.py: better documentation, preserve default umask. - Simplify capgen/metadata/parse_tools/xml_tools.py: remove unneeded arguments, trim down docstring tests (covered in unit tests and end-to-end tests). Requires small changes in the CAM-SIMA compatibility layer (now in the (CAM-SIMA feature branch). - Strip down capgen/metadata/parse_tools/parse_checkers.py, update capgen/metadata/metadata_table.py accordingly. - Not cleanup: Fix the trailing comments after the variable conversions in the Group cap ([feature/capgen-v1-cleanup-01] Add correct comment to supported variable conversions (#767) + bug fixes + unit test updates #768) by @dustinswales. - Not cleanup: Add additional report option --suite-variables to retrieve suite-owned variables from datatable. - Not cleanup: Log variable transformations during capgen call at default logging level (0d48ab0). --------- Co-authored-by: Courtney Peverley <courtneyp@ucar.edu> Co-authored-by: Dustin Swales <dustin.swales@noaa.gov>

Format update of `feature/capgen-v1` from `develop` to resolve conflicts after merging #765 into develop / #767 into `feature/capgen-v1`. There's a small leftover change in `end-to-end-tests/advection_auto_clone/cld_liq.F90` that was made in #765 but not in #767. The other test data change in `test/advection_test/test_host_data.F90` (see https://github.com/NCAR/ccpp-framework/pull/765/changes#diff-04bf1167afd493d2ea37bd03d093e200ba8b563383bfd1cca3fc9126aca9c4a9) is already in `feature/capgen-v1`. Co-authored-by: Courtney Peverley <courtneyp@ucar.edu>

… feature/capgen-v1

gold2718

I am having trouble giving this (moving target) a thorough review but have a question and a couple of concerns. The question is about the protected attribute (see below).
Concern 1: By removing and changing the old unit tests, it is harder to verify that the parser / generator works correctly. How do you verify test coverage relative to the earlier (admittedly not great) version? Are all the tests functionally replicated?
Concern 2: Validation of metadata against the Fortran source has been relegated to capgen/ccpp_validator.py. While running this tool can be built into the build process, the old version ran on the already parsed metadata. At best, this will slow down the validation process as the metadata has to be re-parsed. However, is the metadata in fact parsed the same? Is there a reason this could not be an optional part of capgen that works on the parsed metadata?

gold2718 · 2026-07-09T20:18:34Z

+        source = 'suite'
+    elif host_entry is None and suite_var is None:
+        # Case 2 or 3.
+        if intent == 'out':


I may have missed it, but is the protected attribute checked anywhere? If not, can you introduce a check for host_entry.protected and intent in ('out', 'inout')?

I'll get back to you on that one.

Just checking in to see if this got resolved offline since we are nearly ready to process this PR.

Yes, see PR #773. But this change has no significance for the UFS, since the protected attribute isn't used yet.

climbfuji · 2026-07-28T17:43:19Z

I am having trouble giving this (moving target) a thorough review but have a question and a couple of concerns. The question is about the protected attribute (see below).

I hope that no more updates are needed for now. The most recent update to CAM-SIMA necessitated a number of changes and my limited bandwidth (technically on PTO) didn't allow me to create yet another set of branches and PRs. Future changes required due to CAM-SIMA updates will need to go in yet another staging branch to avoid updating the feature/capgen-v1 branch.

Concern 1: By removing and changing the old unit tests, it is harder to verify that the parser / generator works correctly. How do you verify test coverage relative to the earlier (admittedly not great) version? Are all the tests functionally replicated?

This is indeed the case. The existing tests from capgen v0 and prebuild were copied over 1:1, then the validation trimmed down to just the Fortran part where applicable. The blocked-data test for prebuild was removed, because we never intended to support blocked data structures. New tests were added for new functionality (e.g. multi-instance). An additional advection test with the now-deprecated "auto-clone constituents" (= defined in metadata only) enabled via a legacy shim. Several passes were done with the LLM to ensure that the current tests cover everything the old tests did, plus more. But also to remove duplicates between unit tests and docstring tests.

[Edit] In addition to an already better test coverage, we benefit greatly from the integration with the four host models. These exercise many different aspects of capgen, and the fact that the regression tests all pass and are bit-for-bit identical for CAM-SIMA, CCPP-SCM, NEPTUNE, and UFS, provides us with real evidence that the features that matter (i.e. that are actually being used) are correct.

Concern 2: Validation of metadata against the Fortran source has been relegated to capgen/ccpp_validator.py. While running this tool can be built into the build process, the old version ran on the already parsed metadata. At best, this will slow down the validation process as the metadata has to be re-parsed. However, is the metadata in fact parsed the same? Is there a reason this could not be an optional part of capgen that works on the parsed metadata?

This was a deliberate decision. A separation of concerns means that ccpp_capgen doesn't (need to) know anything about parsing Fortran code and can rely on metadata only. Separating these two steps also enables offline validation of metadata vs fortran without the need for a host model, and it's easier to see immediately if the validation step fails as part of the cmake build integration, or if it is the actual cap generation. Yes, the metadata is read twice in this approach, but the measured runtimes speak for themselves. For all 58 suites of CCPP-SCM, these are the runtimes:

ccpp_prebuild: no validation, only cap generation: 12s
ccpp_capgen v0: validation Fortran vs Metadata for schemes only, plus cap generation: 230s
ccpp_capgen v1: validation Fortran vs Metadata for schemes AND host data, plus cap generation: 10-11s

It's cleaner, easier to understand, faster, and more aligned with modern software development practices (separation of concerns).

gspetro-NOAA · 2026-07-30T16:03:45Z

Could we get approvals on this PR so that we can process ufs-community/ufs-weather-model#3288 ASAP?

dustinswales · 2026-07-30T16:52:08Z

@gspetro-NOAA As part of 3288, we are switching to a new codebase and branch, feature/capgen-v1. So there are no reviews needed here for the overlaying UWM changes.

gspetro-NOAA · 2026-08-05T16:21:21Z

Testing for ufs-community/ufs-weather-model#3288 completed successfully. This PR can be merged.

gspetro-NOAA · 2026-08-05T20:04:35Z

@dustinswales Does this PR need to be merged if we are sticking w/the feature/capgen-v1 branch for the WM? Just trying to make sure I'm not waiting for a merge that won't happen...

dustinswales · 2026-08-05T20:17:00Z

@dustinswales Does this PR need to be merged if we are sticking w/the feature/capgen-v1 branch for the WM? Just trying to make sure I'm not waiting for a merge that won't happen...

@gspetro-NOAA No merging here.

BrianCurtis-NOAA · 2026-08-06T13:44:36Z

Hey @dustinswales could you help confirm and/or clarify the direction for ccpp-framework as it relates to UFS?

What I think I understand is that all development will be in feature/capgen-v1 until CAM-SIMA work is finished, which it will then be merged into develop and UFS will only be tracking feature/capgen-v1 for a minimal timeframe?

climbfuji · 2026-08-06T13:47:22Z

Hey @dustinswales could you help confirm and/or clarify the direction for ccpp-framework as it relates to UFS?

What I think I understand is that all development will be in feature/capgen-v1 until CAM-SIMA work is finished, which it will then be merged into develop and UFS will only be tracking feature/capgen-v1 for a minimal timeframe?

I can answer this.

Yes, the plan is to point to feature/capgen-v1 until CAM-SIMA transitions to this codebase. We anticipate this to happen around the new fiscal year. At that point, we will merge feature/capgen-v1 into develop.

BrianCurtis-NOAA · 2026-08-06T13:49:39Z

Hey @dustinswales could you help confirm and/or clarify the direction for ccpp-framework as it relates to UFS?
What I think I understand is that all development will be in feature/capgen-v1 until CAM-SIMA work is finished, which it will then be merged into develop and UFS will only be tracking feature/capgen-v1 for a minimal timeframe?

I can answer this.

Yes, the plan is to point to feature/capgen-v1 until CAM-SIMA transitions to this codebase. We anticipate this to happen around the new fiscal year. At that point, we will merge feature/capgen-v1 into develop.

Thanks for the quick reply @climbfuji ! This seems fair to me.

climbfuji · 2026-08-10T12:29:44Z

To all reviewers. Not being able to update this feature branch while waiting for reviews (how long?) is causing real problems.

But since several models now point to this branch and reviewing a large set of changes is easier when the code doesn't change under the hood, should we branch off again and create feature/capgen-v1-continued-development from a6716e9 (the head of feature/capgen-v1 branch as of August 10, 2026) and then merge all update PRs (as of 08/10/2026, there are two) into the new branch feature/capgen-v1-continued-development?

If we were to do this, we could refer to that new branch when addressing change requests from the review of feature/capgen-v1, and we could do one final review of feature/capgen-v1-continued-development and update of feature/capgen-v1 before we merge the consolidated feature/capgen-v1 into develop.

gspetro-NOAA · 2026-08-10T14:01:10Z

To all reviewers. Not being able to update this feature branch while waiting for reviews (how long?) is causing real problems.

But since several models now point to this branch and reviewing a large set of changes is easier when the code doesn't change under the hood, should we branch off again and create feature/capgen-v1-continued-development from a6716e9 (the head of feature/capgen-v1 branch as of August 10, 2026) and then merge all update PRs (as of 08/10/2026, there are two) into the new branch feature/capgen-v1-continued-development?

If we were to do this, we could refer to that new branch when addressing change requests from the review of feature/capgen-v1, and we could do one final review of feature/capgen-v1-continued-development and update of feature/capgen-v1 before we merge the consolidated feature/capgen-v1 into develop.

I think development should be able to continue as usual on this branch because @dustinswales had specifically been using the c4b7d9a commit in PR 3288, and this PR was not going to be merged.

climbfuji · 2026-08-10T14:02:25Z

To all reviewers. Not being able to update this feature branch while waiting for reviews (how long?) is causing real problems.
But since several models now point to this branch and reviewing a large set of changes is easier when the code doesn't change under the hood, should we branch off again and create feature/capgen-v1-continued-development from a6716e9 (the head of feature/capgen-v1 branch as of August 10, 2026) and then merge all update PRs (as of 08/10/2026, there are two) into the new branch feature/capgen-v1-continued-development?
If we were to do this, we could refer to that new branch when addressing change requests from the review of feature/capgen-v1, and we could do one final review of feature/capgen-v1-continued-development and update of feature/capgen-v1 before we merge the consolidated feature/capgen-v1 into develop.

I think development should be able to continue as usual on this branch because @dustinswales had specifically been using the c4b7d9a commit in PR 3288, and this PR was not going to be merged.

Correct - I am mainly thinking about the reviewers.

climbfuji · 2026-08-10T18:52:37Z

I found a different way.

For all reviewers. If you want to review a stable version without the latest updates, you can use this URL:

https://github.com/NCAR/ccpp-framework/pull/762/changes/BASE..c4b7d9aed935e8985309e2d974ed43cb46324e50

This compares commit c4b7d9a from the pull request branch (feature/capgen-v1) to the base branch (develop). Likewise,

https://github.com/NCAR/ccpp-framework/pull/762/changes/c4b7d9aed935e8985309e2d974ed43cb46324e50..HEAD

compares the missing commits from c4b7d9a to the head of the feature/capgen-v1 branch.

This way, you can do a first pass of your review based on the static commit c4b7d9a, and then look at the additional commits and differences added to feature/capgen-v1 in the meantime.

…les, reorganize how Claude keeps track of future work (#773) For feature/capgen-v1: - Add missing check for protected variables (per metadata), mirroring the functionality in capgen v0 (capgen/generator/suite_resolver.py and capgen/metadata/metadata_table.py). - Reorganize bookkeeping of future/deferred work in Claude so that it travels between different computers/users. Ultimately, at the time we merge feature/capgen-v1 into develop, any open work items will be converted into GitHub issues if still required (all other files).

…74 (name collision in capgen v1) (#776) 1. Fix for issue #774 - host module import name clash with control variables passed via argument list, plus tests. Instead of implementing some black magic, we simply bail out and instruct the user to fix the name collision. This is easy - simply change the "local name" in the `type = control` metadata table, which is only used as as the l.h.s. in the explicit argument lists to the CCPP calls (`call ccpp_physics_run` etc.) and then passed down to the suite and group caps. 2. Also: tests for issue #772 in capgen v0 (not an issue in v1, but adding the same test to demonstrate). --------- Co-authored-by: Courtney Peverley <courtneyp@ucar.edu>

jimmielin

Thanks @climbfuji. Sharing a couple of findings from Claude for correctness bugs. I think all of them should be pretty easy to resolve.

jimmielin · 2026-08-11T22:04:28Z

+    unit_backward = ''
+    if needs_unit and bwd_fn is not None and intent in ('out', 'inout'):
+        unit_backward_expr = '{}_l'.format(local)
+        unit_backward = _apply_transform_formula(bwd_fn, unit_backward_expr, host_kind)
+    elif needs_kind and intent in ('out', 'inout'):
+        unit_backward = _kind_cast_expr(
+            scheme_var.type, '{}_l'.format(local), host_kind,
+            local=local, std_name=std_name, scheme_name=scheme_name,
+        )
+    elif needs_vert_flip and not needs_unit and intent in ('out', 'inout'):
+        unit_backward = '{}_l'.format(local)
+
+    needs_transform = needs_unit or needs_kind or needs_vert_flip
+
+    # ---- local variable names (transformation temp + pointer) ------------
+    # ``used_local_names`` stores the LOWERCASED names so collision
+    # detection is Fortran-case-insensitive (see _local_name_conflict).
+    temp_name = ''
+    ptr_name  = ''
+    if needs_transform:
+        candidate = '{}_l'.format(local)
+        temp_name = _local_name_conflict(candidate, used_local_names)


I think because unit_backward is built before _local_name_conflict checks here, there could be a bug if I declare a (rather evil) scheme local as p_hpa_l then at the same time I request a Pa -> hPa unit conversion for [p] standard_name = air_pressure this will end up reading the wrong name for unit_backward.

I would suggest moving this block to above unit_forward (which does not have this bug since it is after the local name conflict resolver).

jimmielin · 2026-08-11T22:07:37Z

+                    )
+                pending_props[key] = val
+            # Try to build the MetadataTable as soon as we have name + type.
+            if 'name' in pending_props and 'type' in pending_props and current_table is None:


This check will cause a .meta table with a missing type (for example, a typo into typ = scheme) to be silently ignored.

I would suggest at the next header after the parse if pending_props is non-empty but no table was built to raise an error saying a key is required.

jimmielin · 2026-08-11T22:10:33Z

+    if host_allocatable:
+        # Allocatable actual arguments must omit explicit dimension ranges:
+        # the callee declares the dummy as allocatable too and assumes the
+        # array shape from the actual.
+        subscript: str = ''
+        used_dim_std: Set[str] = set()


If host_allocatable is True and subscript = '' then _build_merged_subscript is never called, so flip_vertical=needs_vert_flip, if True, is ignored, but needs_transform is also True at this point so the temp pipeline is used.

I think if needs_vert_flip and host_allocatable is true it should error out (and maybe a xfail test be added to this case in TestVerticalFlipTransform?)

climbfuji · 2026-08-11T22:50:24Z

Thanks for the reports @jimmielin. I'll take a closer look next week; if you want to propose bug fixes for those, please go ahead.

climbfuji added 30 commits May 13, 2026 06:40

Remove prebuild and capgen

ab42cdb

Update GitHub workflows for capgen-ng

9a9fbc1

Add directories capgen-ng, unit-tests, end-to-end-tests

fc007c6

Add doc/

43fb5b2

Update GitHub workflows

f72cb29

.github/workflows/unit-tests.yaml: drop Python 3.8, EOL Oct 2024

eaaea49

Minor bug fixes in capgen-ng code and unit tests from ccpp-scm integr…

9912ba4

…ation

Update documentation

837bcd1

Update doc

cfd4bf7

Update capgen-ng: ccpp_datafile --scheme-files

b9a28d2

Update capgen-ng/** and unit-test/* with bug fixes from CCPP-SCM GFS …

0f83989

…v17p8 suite

Update doc/* and add doc/briefing.md

f2c5918

Temporarily add end-to-end-tests.sh

2a9892b

Merge branch 'develop' of https://github.com/ncar/ccpp-framework into…

ac6d4d8

… feature/capgen-ng

Update capgen-ng and unit-tests: all GFS v17 p8 suites in SCM

8fd7b3e

Minor updates for GFS v16 suites in SCM

687f585

Minor updates from last CCPP-SCM suites

e25e253

Add --trace flag to capgen call

f29e7e7

Shorten Fortran names of auto-generated subroutines/functions

a35632a

Update capgen-ng: move number_of_instances from host table to control…

205ec79

… table

Bug fixes for NEPTUNE, rename ccpp_static_api

0909524

Bug fix: local_subscript lost in active expressions (and three siblin…

1043666

…g sites)

ccpp_physics_final and ccpp_final idempotent

96ffd0c

Fix final phase idempotency

2704aea

Save documentation updates

abe5f11

Add end-to-end-tests/instances_advection and bug fixes/updates to cap…

523ac77

…gen-ng: entire constituent state now per instance

Add doc/briefing_pm.md

eae0d48

Clean up dead code in capgen-ng/**

68eff47

Manual update of doc/briefing_pm.md

4d39dad

Update spelling in docs/*

f4c0ef1

This was referenced Jul 9, 2026

Updates for ccpp-capgen v1 NOAA-GFDL/GFDL_atmos_cubed_sphere#419

Merged

Updates for ccpp-capgen v1 NOAA-EMC/ufsatm#1125

Merged

Updates for ccpp-capgen v1 NOAA-EMC/CMEPS#173

Merged

Updates for ccpp-capgen v1 ufs-community/ufs-weather-model#3288

Merged

climbfuji mentioned this pull request Jul 15, 2026

ccpp-capgen v1: a unified successor to ccpp-prebuild and ccpp-capgen v0 #759

Open

17 tasks

climbfuji and others added 6 commits July 16, 2026 15:16

Merge branch 'develop' of https://github.com/ncar/ccpp-framework into…

ed284a8

… feature/capgen-v1

cime_config/host_framework_deps.py documentation updates

af65334

Always create and eport constituents, even if table has size zero

e1f90d1

Update doc/constituents_overhaul.md

a6716e9

gold2718 reviewed Jul 27, 2026

View reviewed changes

climbfuji mentioned this pull request Aug 10, 2026

[feature/capgen-v1] Add missing check for (metadata-)protected variables, reorganize how Claude keeps track of future work #773

Merged

climbfuji and others added 2 commits August 10, 2026 12:54

jimmielin reviewed Aug 11, 2026

View reviewed changes

Uh oh!

Conversation

climbfuji commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Todo

Issues

Testing

Uh oh!

gold2718 left a comment

Choose a reason for hiding this comment

Uh oh!

gold2718 Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

climbfuji Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

gspetro-NOAA Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

climbfuji Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

climbfuji commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gspetro-NOAA commented Jul 30, 2026

Uh oh!

dustinswales commented Jul 30, 2026

Uh oh!

gspetro-NOAA commented Aug 5, 2026

Uh oh!

gspetro-NOAA commented Aug 5, 2026

Uh oh!

dustinswales commented Aug 5, 2026

Uh oh!

BrianCurtis-NOAA commented Aug 6, 2026

Uh oh!

climbfuji commented Aug 6, 2026

Uh oh!

BrianCurtis-NOAA commented Aug 6, 2026

Uh oh!

climbfuji commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gspetro-NOAA commented Aug 10, 2026

Uh oh!

climbfuji commented Aug 10, 2026

Uh oh!

climbfuji commented Aug 10, 2026

Uh oh!

jimmielin left a comment

Choose a reason for hiding this comment

Uh oh!

jimmielin Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

jimmielin Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

jimmielin Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

climbfuji commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

climbfuji commented Jun 30, 2026 •

edited

Loading

climbfuji commented Jul 28, 2026 •

edited

Loading

climbfuji commented Aug 10, 2026 •

edited

Loading