Skip to content
Open
Changes from 5 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
234a26a
#868 add text describing nlevels and ndata
arporter Oct 21, 2025
060a1ea
#868 update the rules for kernel arguments
arporter Oct 21, 2025
716c908
#868 change nlevels to nlayers
arporter Oct 22, 2025
5a04ba3
Merge branch 'master' into 868_nlayers_ndata_mdata
arporter Nov 5, 2025
e878688
#868 extend docs to allow for naming of nlayers values
arporter Nov 5, 2025
5cdb284
Merge branch 'master' into 868_nlayers_ndata_mdata
arporter Apr 2, 2026
3d4e9b9
#868 update nlayers docs
arporter Apr 2, 2026
a0f3f15
Merge branch 'master' into 868_nlayers_ndata_mdata
arporter Apr 8, 2026
5db865a
Merge branch 'master' into 868_nlayers_ndata_mdata
arporter Apr 8, 2026
a0132df
Merge branch '868_nlayers_ndata_mdata' of github.com:stfc/PSyclone in…
arporter Apr 8, 2026
1bfa1af
#868 update the nlayers text to make it clear that named values are j…
arporter Apr 8, 2026
a6833a1
#868 add initial code to get nlevels metadata for a field arg
arporter Apr 8, 2026
0f54ac6
Merge branch '868_nlayers_ndata_mdata' of github.com:stfc/PSyclone in…
arporter Apr 8, 2026
b5751bb
Merge branch 'master' into 868_nlayers_ndata_mdata
arporter Apr 8, 2026
6e5b79a
Merge branch 'master' into 868_nlayers_ndata_mdata
arporter Apr 10, 2026
892245f
#868 update nlayers documentation to remove special gh_runtime tag
arporter Apr 10, 2026
1c18889
Merge branch 'master' into 868_nlayers_ndata_mdata
arporter Jun 5, 2026
07e8a29
Merge branch 'master' into 868_nlayers_ndata_mdata
arporter Jun 5, 2026
748d38d
#868 update docs for nlayers and ndata to use string values
arporter Jun 5, 2026
529158b
Merge branch 'master' into 868_nlayers_ndata_mdata
arporter Jun 8, 2026
66617c4
#868 begin adding support for ndata metadata
arporter Jun 8, 2026
64c2d26
Merge branch 'master' into 868_nlayers_ndata_mdata
arporter Jun 8, 2026
7a8c9e9
#868 WIP on initial implementation [skip ci]
arporter Jun 9, 2026
707db35
#868 tidy implementation
arporter Jun 9, 2026
268985b
#868 WIP updating psyir-metadata handling classes [skip ci]
arporter Jun 9, 2026
4b8ea08
#868 WIP extending metadata handling [skip ci]
arporter Jun 16, 2026
81d39f0
#868 more work extending metadata support
arporter Jun 16, 2026
9082088
Merge branch 'master' into 868_nlayers_ndata_mdata
arporter Jun 17, 2026
22c474d
#868 allow for tuple of fparser classes
arporter Jun 17, 2026
2f4820c
#868 rm unused utility and improve tests
arporter Jun 17, 2026
3f420a3
#868 simplify intergrid
arporter Jun 18, 2026
887a9d5
#868 fix cov of common_arg_metadata
arporter Jun 18, 2026
58d9787
#868 more coverage
arporter Jun 18, 2026
d56eebb
#868 add type hints
arporter Jun 18, 2026
36c031a
#868 extend support for inter-grid and vector inter-grid
arporter Jun 18, 2026
cfdf291
#868 improve doc string and typing
arporter Jun 22, 2026
c09444d
#868 update setup-python action to v5
arporter Jun 22, 2026
512ecdc
#868 update lfric workflow to use updated lfric_core with nlevels/ndata
arporter Jun 24, 2026
03203e7
#868 add use of ndata to testkern_mod and fix compilation tests
arporter Jun 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 74 additions & 19 deletions doc/user_guide/lfric.rst
Original file line number Diff line number Diff line change
Expand Up @@ -162,24 +162,30 @@ least rank (number of dimensions) one. Scalar arrays are identified with
Field
+++++

LFRic API fields, identified with ``GH_FIELD`` metadata, represent
FEM discretisations of various dynamical core prognostic and diagnostic
LFRic API fields, identified with ``GH_FIELD`` metadata, represent FEM
discretisations of various dynamical core prognostic and diagnostic
variables. In FEM, variables are discretised by placing them into a
function space (see :ref:`lfric-function-space`) from which they
inherit a polynomial expansion via the basis functions of that space.
Field values at points within a cell are evaluated as the sum of a set
of basis functions multiplied by coefficients which are the data points.
Points of evaluation are determined by a quadrature object
of basis functions multiplied by coefficients which are the data
points. Points of evaluation are determined by a quadrature object
(:ref:`lfric-quadrature`) and are independent of the function space
the field is on. Placement of field data points, also called degrees of
freedom (hereafter "DoFs"), is determined by the function space the field
is on.
the field is on. Placement of field data points, also called degrees
of freedom (hereafter "DoFs"), is determined by the function space the
field is on. An LFRic multi-data field can have more than one value
associated with each data point.

LFRic fields passed as arguments to any :ref:`LFRic kernel
<lfric-kernel-valid-data-type>` can be of ``real`` or ``integer``
primitive type. In the LFRic infrastructure, these fields are
represented by instances of the ``field_type`` and ``integer_field_type``
classes, respectively.

Typically, a field will have the same number of vertical layers as the
model mesh. However, this is not a requirement and the number of layers
can be as few as one (a 2D field).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest avoiding the idea that there is a particular model mesh. For example, we will have apps that regrid data from one number of levels to another.

Suggested change
Typically, a field will have the same number of vertical layers as the
model mesh. However, this is not a requirement and the number of layers
can be as few as one (a 2D field).
While an application may model a set of fields that all have the same number of
vertical levels, this is not a requirement. An application may have fields on a range
of different levels including fields on a single level (a 2D field).


.. _lfric-field-vector:

Field Vector
Expand Down Expand Up @@ -919,8 +925,8 @@ All three CMA-related kernel types must obey the following rules:
1) Since a CMA operator only acts within a single column of data,
stencil operations are not permitted.

2) No vector quantities (e.g. ``GH_FIELD*3`` - see below) are
permitted as arguments.
2) No vector quantities (e.g. ``GH_FIELD*3`` - see below) or
multi-data fields are permitted as arguments.

3) The kernel must operate on cell-columns.

Expand Down Expand Up @@ -1636,7 +1642,7 @@ to have stencil accesses, these two options are mutually exclusive.
The metadata for each case is described in the following sections.

Stencil Metadata
________________
""""""""""""""""


Stencil metadata specifies that the corresponding field argument is accessed
Expand Down Expand Up @@ -1716,8 +1722,7 @@ be found in ``examples/lfric/eg5``.
.. _lfric-intergrid-mdata:

Inter-Grid Metadata
___________________

"""""""""""""""""""

The alternative form of the optional fifth metadata argument for a
field specifies which mesh the associated field is on. This is
Expand Down Expand Up @@ -1748,6 +1753,45 @@ meshes cannot be on the same function space while those on the same
mesh must also be on the same function space.


Number of Layers Metadata
"""""""""""""""""""""""""

If a particular field argument to a kernel has a number of vertical levels
that is not the same as the extruded mesh then this must be specified using

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be misunderstanding this. I am assuming it relates to a kernel which, for example, takes an argument which may be, say, an 70-level field but could be a 50-level or 40-level field. But the kernel should only set 40 levels (or fewer) of the field. Most typically, perhaps, it would be useful when setting the bottom level of a field of any number of levels.

Suggested change
that is not the same as the extruded mesh then this must be specified using
that may be different from the mesh of the input field then this must be specified using

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what the use case is I'm afraid. However, when we discussed with Chris and Iva we were definitely talking about fields that did not have e.g. 70 levels because that then changed the values in the dof map. On the other hand, if all fields actually have the same number of levels but only populate a subset of them then all of their dofmaps will be identical (for a given FS). I think this needs clarification from @TeranIvy or @tommbendall?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coming back to this now, we recently agreed with @TeranIvy and @christophermaynard that the nlevels extension was the more straightforward and we would do this first. However, I've now re-discovered this thread and realised I still need an answer to the question: does a field with e.g. 40 levels actually have 70 but only makes use of the first 40? If so, it will have the same dofmap as the 70-level field.
Then there's the special case of 2D fields - perhaps this answers my first question as presumably you don't only make use of the 1st of 70 levels as that's a lot of wasted storage? Tagging @stevemullerworth and @tommbendall too.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each field needs to have correct metadata in the kernel. The metadata can be used to identify the correct dofmap and the correct nlayers for the field. That's a strict rule.

An application can create fields with a 40-level extrusion and a 70-level extrusion (and a 2d field which in practice is a 1-level "extrusion"). Each extrusion requires a different 3D mesh and gives you a different nlayers, dofmap etc.

My suggested change refers to a kernel that, for example, applies fieldinput to fieldoutput. fieldoutput has 70 levels. fieldinput has 40 levels. So the kernel API could have two nlayers arguments and two sets of dofmap arguments. The two fields would be associated with different LFRic function spaces that are each attached to different 3D meshes and so different extrusions.

It's up to the kernel to know that it is looping over the fieldinput nlayers value and not the fieldoutput nlayers value.

Technically, one could write a kernel that loops over a subset of layers. But in that case, the number of layers would need to be a separate kernel input and the nlayers input would be ignored (except to check that we're not looping over more than nlayers levels).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Steve, I understand things a bit more now. Currently, we pass nlayers obtained from the first field/operator kernel argument:
image
I think what we're saying therefore is that any field/operator that has a number of levels that differs from the first field/operator argument must have nlayers=xx in its metadata. In turn, this will mean an additional dofmap is supplied to the kernel for each distinct combination of function space and nlayers. A kernel developer is of course free to specify nlayers for every field/operator kernel argument. That would just mean we would pass one of the dofmaps to the kernel twice but that's not a massive overhead (as opposed to requiring that all kernels be re-written).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or, have I misunderstood and the proposal is to mandate that all kernel metadata specify nlayers for every field/operator argument?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just catching up on this. A change to enforce nlayers metadata for every field/operator would be a very daunting task! So I would support the former proposal: only including an nlayers metadata argument for the fields/operators that differ from the first field/operator.

For existing kernels that have fields/operators on different meshes, we are currently managing this by declaring the fields/operators to be on different function spaces.

@arporter arporter Apr 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now have:
image
If people are OK with that, the next question is whether to require nlayers to be an integer (in which case the 'labels' will have to be at least declared and initialised locally in the module containing the kernel) or whether we go for a string of some fixed length. That will avoid the need to declare anything but will require a character member of arg_type. As I've said elsewhere, I'm a bit uneasy about this because we've not done it before and there may be a good reason for that (and then again, it may just be because we've always wanted to restrict it to certain recognised values and integer parameters were an efficient way of enforcing that using Fortran alone).

the ``NLAYERS`` option to ``GH_FIELD``, e.g.::

arg_type(GH_FIELD, GH_REAL, GH_READ, W3, NLAYERS=1)

The value specified for ``NLAYERS`` may be a literal if it is known at
compile time. Alternatively, it may be given a named value (one of
``GH_NLAYERSM1`` TODO) or the special value
``GH_RUNTIME``. A named value means that the number of levels is to be
determined at runtime by querying the field object (in the generated
PSy layer). If two different field arguments are on the same function
space but both have ``NLAYERS=GH_RUNTIME`` then it is assumed that
they may have *different* values of ``NLAYERS`` and hence a separate
dofmap is passed to the kernel for each. However, if two or more field
arguments are on the same function space and have the same, named number
of layers which is not ``GH_RUNTIME`` then only one dofmap is passed to
the kernel for those arguments.
Comment thread
arporter marked this conversation as resolved.
Outdated


Multi-Data Metadata
"""""""""""""""""""

A multi-data field is the same as a standard field apart from having multiple
values associated with each DoF. This is indicated in the field metadata by
the optional ``NDATA`` argument to GH_FIELD, e.g.::

arg_type(GH_FIELD, GH_REAL, GH_READ, W2, NDATA=4)

The value specified for ``NDATA`` may be a literal if it is known at
compile time. Alternatively, it may be given the special value
``GH_RUNTIME`` which means that the number of data values at each DoF
is to be determined at runtime by querying the field object (in the
generated PSy layer).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For LFRic atmosphere, It would be useful to have the same flexibility for giving named values as is done for NLAYERS: in fact, it is more useful to have the flexibility here than for NLAYERS as there is less mixing and matching of layer numbers.

We have a lot of physics kernels with ~100 fields with a handful of multidata choices distinguished by ANY_DISCONTINUOUS_SPACE_1, ANY_DISCONTINUOUS_SPACE_2 etc. If they are converted to GH_RUNTIME we would need separate dofmaps etc. for each of the many fields rather than for each of the ANY_DISCONTINUOUS_SPACE choices. Example kernel here [MOSRS password protected]:

https://code.metoffice.gov.uk/trac/lfric_apps/browser/main/trunk/interfaces/physics_schemes_interface/source/kernel/bl_exp_kernel_mod.F90

@tommbendall tommbendall Nov 6, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just commenting that I agree with this point! In fact I think the most common situation is that NDATA will be a runtime variable, but there might be many multidata fields being passed as arguments the same kernel, but all using the same NDATA value.

I'm not sure what the best solution to this is... because the NDATA values are science-related, I don't think we want any specific values hard-coded in arguments_mod.

Could we define a local variable for NDATA to take? e.g.

type(ndata_type) :: LAND_TILES
type(ndata_types) :: NUM_AEROSOLS

arg_type(GH_FIELD, GH_REAL, GH_READ, W2, NDATA=4)
type(arg_type) :: meta_args(4) = (/  &
     arg_type(GH_FIELD, GH_REAL, GH_READWRITE, W3),  &
     arg_type(GH_FIELD, GH_REAL, GH_READ,  W3, NDATA=LAND_TILES), &
     arg_type(GH_FIELD, GH_REAL, GH_READ,  WTHETA, NDATA=NUM_AEROSOLS), &
     arg_type(GH_FIELD, GH_REAL, GH_READ,  W3, NDATA=NUM_AEROSOLS)  &
/)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the sort of thing I had envisaged (though originally I was thinking of a direct but more meaningful replacement for ANY_*SPACE entries).

One of the challenges of moving away from ANY_DISCONTINUOUS_SPACE settings was deciding how and where to name scientifically-meaningful versions given that arguments_mod is in core. One could have a physics module to store them which would enable a common set to be used by several kernels.

I think they would need to be initialised so as not to cause any compiler warnings? If so, that's another reason for hiding them in a module.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, that's the same as I was imagining for NLAYERS so it's straightforward to do from a PSyclone point of view. I can see your issue with names in core and physics but I'll let you wrangle that :-)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something else to consider that Tom M has just mentioned is that we have two types of multidata field. Those with ndata_first and those without. We may also need to capture that

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @tommbendall , I'm just starting to pick this back up. Has there been any progress on this point - i.e. is it something that we need to support? I might as welll get it into the design now if so...

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we would want to support both types (different data layouts will have optimisation advantages for different types of science kernel), but it may actually make no difference to the PSyclone interface!

I think you say this in another comment but fields are always 1D arrays. The only difference it makes is whether the science kernel should be written:

do n = 1, ndata
  do k = 1, nlayers
    idx = map(df) + k-1 + nlayers*(n-1)
    ... ! do calculation

or

do k = 1, nlayers
  do n = 1, ndata
    idx = map(df) + n-1 + ndata*(k-1)
    ... ! do calculation

which I assume PSyclone doesn't need to know about



Column-wise Operators (CMA)
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down Expand Up @@ -2080,7 +2124,12 @@ conventions, are:
4) If the field entry stencil access is of type ``XORY1D`` then
add an additional ``integer`` direction argument of kind
``i_def`` and with intent ``in``.

5) If the field is multi-data then the kernel must be passed the
value of ``NDATA``: add an additional ``integer``, scalar
argument of kind ``i_def`` and intent ``in``.
6) If the field has a custom number of vertical levels then pass this as
an additional ``integer``, scalar argument of kind ``i_def`` and
intent ``in``.
3) If the current entry is a field vector then for each dimension
of the vector, include a field array. The field array name is
specified as
Expand All @@ -2106,21 +2155,26 @@ conventions, are:
the data type and kind specifed in the metadata. The ScalarArray
must be denoted with intent ``in`` to match its read-only nature.

4) For each function space in the order they appear in the metadata arguments
(the ``to`` function space of an operator is considered to be before the
4) DoF maps for function spaces are handled in the order they appear in the
metadata arguments (the ``to`` function space of an operator is considered
to be before the
``from`` function space of the same operator as it appears first in
lexicographic order)
lexicographic order). Note that if two fields on a given function space have
differing numbers of vertical layers, then each requires that a
dofmap be supplied (because the number of vertical layers alters the
*values* within the map). For each required DoF map:

1) Include the number of local degrees of freedom (i.e. number per-cell)
for the function space. This is an ``integer`` of kind ``i_def`` and
has intent ``in``. The name of this argument is
``"ndf_"<field_function_space>``.

2) If there is a field on this space

1) Include the unique number of degrees of freedom for the function
space. This is an ``integer`` of kind ``i_def`` and has intent ``in``.
The name of this argument is ``"undf_"<field_function_space>``.
2) Include the **dofmap** for this function space. This is an ``integer``
2) Include the **dofmap** itself. This is an ``integer``
array of kind ``i_def`` with intent ``in``. It has one dimension
sized by the local degrees of freedom for the function space.

Expand Down Expand Up @@ -2443,8 +2497,9 @@ dofmap for both the to- and from-function spaces of the CMA
operator. Since it does not have any LMA operator arguments it does
not require the ``ncell_3d`` and ``nlayers`` scalar arguments. (Since a
column-wise operator is, by definition, assembled for a whole column,
there is no loop over levels when applying it.)
The full set of rules is then:
there is no loop over levels when applying it.) Note that fields with
non-standard ``nlayers`` or ``ndata > 1`` cannot be supplied as
arguments to CMA kernels. The full set of rules is:

1) Include the ``cell`` argument. ``cell`` is an ``integer`` of kind
``i_def`` and has intent ``in``.
Expand Down
Loading