Skip to content

Fix AppVeyor 64bit build #1832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ax3l merged 11 commits into from
Jan 26, 2026
Merged

Fix AppVeyor 64bit build #1832

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
c641507
Fix `download_samples.ps1` Move
ax3l Jan 13, 2026
39af352
CI: Fix AppVeyor x64 (64bit) build
ax3l Jan 14, 2026
4143acb
Fix instantiations for AttributeWithShapeAndResource
franzpoeschel Jan 23, 2026
0636ce5
`dtype_from_numpy`: Add `long` alias
ax3l Jan 26, 2026
085ef1c
`PatchRecordComponent`: Match Identical Types
ax3l Jan 26, 2026
fa2284e
`dtype_from_numpy`: "l"/"L"
ax3l Jan 26, 2026
bcf44d2
`7_extended_write_serial.py`: Reformat dtype
ax3l Jan 26, 2026
65a2ba9
Fix numpy API call?
franzpoeschel Jan 26, 2026
fc8cb3c
Consistency: `dtype_to_numpy` and `dtype_from_numpy`
ax3l Jan 26, 2026
f2b5b80
Update: `APITest.py` `testDataset`
ax3l Jan 26, 2026
39e6dc2
Cleaning
ax3l Jan 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .appveyor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ environment:
- TARGET_ARCH: x64
CONDA_PY: 3.10
CONDA_INSTALL_LOCN: C:\\Miniconda37-x64
APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2017
APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
platform: x64
SHARED: OFF

Expand Down Expand Up @@ -101,12 +101,12 @@ before_build:
# Compiler & Generator Selection
- cmd: if "%APPVEYOR_BUILD_WORKER_IMAGE%"=="Visual Studio 2017" set OPENPMD_CMAKE_GENERATOR=Visual Studio 15 2017
- cmd: if "%APPVEYOR_BUILD_WORKER_IMAGE%"=="Visual Studio 2019" set OPENPMD_CMAKE_GENERATOR=Visual Studio 16 2019
- cmd: if "%TARGET_ARCH%"=="x64" set OPENPMD_CMAKE_GENERATOR=%OPENPMD_CMAKE_GENERATOR% Win64
# - cmd: if "%TARGET_ARCH%"=="x86" "C:\Program Files (x86)\Microsoft Visual Studio 15.9\VC\vcvarsall.bat" x86
# - cmd: if "%TARGET_ARCH%"=="x64" "C:\Program Files (x86)\Microsoft Visual Studio 15.9\VC\vcvarsall.bat" amd64

# CMake configure
- cmd: cmake -G "%OPENPMD_CMAKE_GENERATOR%" -DCMAKE_BUILD_TYPE=%CONFIGURATION% -DBUILD_SHARED_LIBS=%SHARED% -DBUILD_TESTING=ON -DopenPMD_USE_PYTHON=ON -DPython_EXECUTABLE="%CONDA_INSTALL_LOCN%\python.exe" -DCMAKE_INSTALL_PREFIX="%CONDA_INSTALL_LOCN%" -DCMAKE_INSTALL_BINDIR="Library\bin" ".."
- cmd: if "%TARGET_ARCH%"=="x64" cmake -G "%OPENPMD_CMAKE_GENERATOR%" -A "%TARGET_ARCH%" -DCMAKE_BUILD_TYPE=%CONFIGURATION% -DBUILD_SHARED_LIBS=%SHARED% -DBUILD_TESTING=ON -DopenPMD_USE_PYTHON=ON -DPython_EXECUTABLE="%CONDA_INSTALL_LOCN%\python.exe" -DCMAKE_INSTALL_PREFIX="%CONDA_INSTALL_LOCN%" -DCMAKE_INSTALL_BINDIR="Library\bin" ".."
- cmd: if "%TARGET_ARCH%"=="x86" cmake -G "%OPENPMD_CMAKE_GENERATOR%" -DCMAKE_BUILD_TYPE=%CONFIGURATION% -DBUILD_SHARED_LIBS=%SHARED% -DBUILD_TESTING=ON -DopenPMD_USE_PYTHON=ON -DPython_EXECUTABLE="%CONDA_INSTALL_LOCN%\python.exe" -DCMAKE_INSTALL_PREFIX="%CONDA_INSTALL_LOCN%" -DCMAKE_INSTALL_BINDIR="Library\bin" ".."

build_script:
- cmd: cmake --build . --config %CONFIGURATION% -j 2
Expand Down
2 changes: 1 addition & 1 deletion examples/7_extended_write_serial.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@
d = Dataset(partial_particleOff.dtype, mpiDims)
electrons["positionOffset"]["x"].reset_dataset(d)

dset = Dataset(np.dtype("uint64"), extent=[2])
dset = Dataset(np.dtype(np.uint64), extent=[2])
Comment thread
ax3l marked this conversation as resolved.
Outdated
electrons.particle_patches["numParticles"].reset_dataset(dset)
electrons.particle_patches["numParticlesOffset"]. \
reset_dataset(dset)
Expand Down
10 changes: 8 additions & 2 deletions include/openPMD/backend/PatchRecordComponent.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,10 @@ template <typename T>
inline void PatchRecordComponent::store(uint64_t idx, T data)
{
Datatype dtype = determineDatatype<T>();
if (dtype != getDatatype())
if (dtype != getDatatype() && !isSameInteger<T>(getDatatype()) &&
!isSameFloatingPoint<T>(getDatatype()) &&
!isSameComplexFloatingPoint<T>(getDatatype()) &&
!isSameChar<T>(getDatatype()))
Comment on lines 162 to +166
Copy link
Copy Markdown
Member Author

@ax3l ax3l Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not urgently needed but sensible.
We should group these together into a single call like isEquivalentRepresentation<T>(getDatatype()) where we use it in the code base.

Copy link
Copy Markdown
Contributor

@franzpoeschel franzpoeschel Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our Datatype equality operator already has these semantics:

/** Comparison for two Datatypes
 *
 * Besides returning true for the same types, identical implementations on
 * some platforms, e.g. if long and long long are the same or double and
 * long double will also return true.
 */
inline bool isSame(openPMD::Datatype const d, openPMD::Datatype const e)
{
    // exact same type
    if (static_cast<int>(d) == static_cast<int>(e))
        return true;

    bool d_is_vec = isVector(d);
    bool e_is_vec = isVector(e);

    // same int
    bool d_is_int, d_is_sig;
    std::tie(d_is_int, d_is_sig) = isInteger(d);
    bool e_is_int, e_is_sig;
    std::tie(e_is_int, e_is_sig) = isInteger(e);
    if (d_is_int && e_is_int && d_is_vec == e_is_vec && d_is_sig == e_is_sig &&
        toBits(d) == toBits(e))
        return true;

    // same float
    bool d_is_fp = isFloatingPoint(d);
    bool e_is_fp = isFloatingPoint(e);

    if (d_is_fp && e_is_fp && d_is_vec == e_is_vec && toBits(d) == toBits(e))
        return true;

    // same complex floating point
    bool d_is_cfp = isComplexFloatingPoint(d);
    bool e_is_cfp = isComplexFloatingPoint(e);

    if (d_is_cfp && e_is_cfp && d_is_vec == e_is_vec && toBits(d) == toBits(e))
        return true;

    return false;
}

We should try merging your code into that instead of creating a new call

Copy link
Copy Markdown
Contributor

@franzpoeschel franzpoeschel Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--> #1854

{
std::ostringstream oss;
oss << "Datatypes of patch data (" << dtype << ") and dataset ("
Expand All @@ -187,7 +190,10 @@ template <typename T>
inline void PatchRecordComponent::store(T data)
{
Datatype dtype = determineDatatype<T>();
if (dtype != getDatatype())
if (dtype != getDatatype() && !isSameInteger<T>(getDatatype()) &&
!isSameFloatingPoint<T>(getDatatype()) &&
!isSameComplexFloatingPoint<T>(getDatatype()) &&
!isSameChar<T>(getDatatype()))
{
std::ostringstream oss;
oss << "Datatypes of patch data (" << dtype << ") and dataset ("
Expand Down
124 changes: 55 additions & 69 deletions include/openPMD/binding/python/Numpy.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,17 @@ namespace openPMD
{
inline Datatype dtype_from_numpy(pybind11::dtype const dt)
{
// Use explicit PEP 3118 / struct format character codes for portable
// cross-platform behavior. This ensures consistency with
// dtype_from_bufferformat() and avoids platform-dependent interpretation
// of numpy type name strings.
// ref: https://docs.python.org/3/library/struct.html#format-characters
// ref: https://numpy.org/doc/stable/reference/arrays.interface.html
// ref: https://docs.scipy.org/doc/numpy/user/basics.types.html
// ref: https://github.com/numpy/numpy/issues/10678#issuecomment-369363551
if (dt.char_() == pybind11::dtype("b").char_())
char const c = dt.char_();
switch (c)
{
case 'b': // signed char
if constexpr (std::is_signed_v<char>)
{
return Datatype::CHAR;
Expand All @@ -43,7 +51,7 @@ inline Datatype dtype_from_numpy(pybind11::dtype const dt)
{
return Datatype::SCHAR;
}
else if (dt.char_() == pybind11::dtype("B").char_())
case 'B': // unsigned char
if constexpr (std::is_unsigned_v<char>)
{
return Datatype::CHAR;
Expand All @@ -52,42 +60,41 @@ inline Datatype dtype_from_numpy(pybind11::dtype const dt)
{
return Datatype::UCHAR;
}
else if (dt.char_() == pybind11::dtype("short").char_())
case 'h': // short
return Datatype::SHORT;
else if (dt.char_() == pybind11::dtype("intc").char_())
return Datatype::INT;
else if (dt.char_() == pybind11::dtype("int_").char_())
return Datatype::LONG;
else if (dt.char_() == pybind11::dtype("longlong").char_())
return Datatype::LONGLONG;
else if (dt.char_() == pybind11::dtype("ushort").char_())
case 'H': // unsigned short
return Datatype::USHORT;
else if (dt.char_() == pybind11::dtype("uintc").char_())
case 'i': // int
return Datatype::INT;
case 'I': // unsigned int
return Datatype::UINT;
else if (dt.char_() == pybind11::dtype("uint").char_())
case 'l': // long
return Datatype::LONG;
case 'L': // unsigned long
return Datatype::ULONG;
else if (dt.char_() == pybind11::dtype("ulonglong").char_())
case 'q': // long long
return Datatype::LONGLONG;
case 'Q': // unsigned long long
return Datatype::ULONGLONG;
else if (dt.char_() == pybind11::dtype("clongdouble").char_())
return Datatype::CLONG_DOUBLE;
else if (dt.char_() == pybind11::dtype("cdouble").char_())
return Datatype::CDOUBLE;
else if (dt.char_() == pybind11::dtype("csingle").char_())
return Datatype::CFLOAT;
else if (dt.char_() == pybind11::dtype("longdouble").char_())
return Datatype::LONG_DOUBLE;
else if (dt.char_() == pybind11::dtype("double").char_())
return Datatype::DOUBLE;
else if (dt.char_() == pybind11::dtype("single").char_())
case 'f': // float
return Datatype::FLOAT;
else if (dt.char_() == pybind11::dtype("bool").char_())
case 'd': // double
return Datatype::DOUBLE;
case 'g': // long double
return Datatype::LONG_DOUBLE;
case 'F': // complex float
return Datatype::CFLOAT;
case 'D': // complex double
return Datatype::CDOUBLE;
case 'G': // complex long double
return Datatype::CLONG_DOUBLE;
case '?': // bool
return Datatype::BOOL;
else
{
default:
pybind11::print(dt);
throw std::runtime_error(
std::string("Datatype '") + dt.char_() +
std::string("' not known in 'dtype_from_numpy'!")); // _s.format(dt)
std::string("Datatype '") + c +
std::string("' not known in 'dtype_from_numpy'!"));
}
}

Expand Down Expand Up @@ -159,6 +166,8 @@ inline Datatype dtype_from_bufferformat(std::string const &fmt)

inline pybind11::dtype dtype_to_numpy(Datatype const dt)
{
// Use explicit PEP 3118 format character codes for portable behavior.
// This ensures round-trip consistency with dtype_from_numpy().
using DT = Datatype;
switch (dt)
{
Expand All @@ -177,80 +186,57 @@ inline pybind11::dtype dtype_to_numpy(Datatype const dt)
case DT::SCHAR:
case DT::VEC_SCHAR:
return pybind11::dtype("b");
break;
case DT::UCHAR:
case DT::VEC_UCHAR:
return pybind11::dtype("B");
break;
// case DT::SCHAR:
// case DT::VEC_SCHAR:
// pybind11::dtype("b");
// break;
case DT::SHORT:
case DT::VEC_SHORT:
return pybind11::dtype("short");
break;
return pybind11::dtype("h");
case DT::INT:
case DT::VEC_INT:
return pybind11::dtype("intc");
break;
return pybind11::dtype("i");
case DT::LONG:
case DT::VEC_LONG:
return pybind11::dtype("int_");
break;
return pybind11::dtype("l");
case DT::LONGLONG:
case DT::VEC_LONGLONG:
return pybind11::dtype("longlong");
break;
return pybind11::dtype("q");
case DT::USHORT:
case DT::VEC_USHORT:
return pybind11::dtype("ushort");
break;
return pybind11::dtype("H");
case DT::UINT:
case DT::VEC_UINT:
return pybind11::dtype("uintc");
break;
return pybind11::dtype("I");
case DT::ULONG:
case DT::VEC_ULONG:
return pybind11::dtype("uint");
break;
return pybind11::dtype("L");
case DT::ULONGLONG:
case DT::VEC_ULONGLONG:
return pybind11::dtype("ulonglong");
break;
return pybind11::dtype("Q");
case DT::FLOAT:
case DT::VEC_FLOAT:
return pybind11::dtype("single");
break;
return pybind11::dtype("f");
case DT::DOUBLE:
case DT::VEC_DOUBLE:
case DT::ARR_DBL_7:
return pybind11::dtype("double");
break;
return pybind11::dtype("d");
case DT::LONG_DOUBLE:
case DT::VEC_LONG_DOUBLE:
return pybind11::dtype("longdouble");
break;
return pybind11::dtype("g");
case DT::CFLOAT:
case DT::VEC_CFLOAT:
return pybind11::dtype("csingle");
break;
return pybind11::dtype("F");
case DT::CDOUBLE:
case DT::VEC_CDOUBLE:
return pybind11::dtype("cdouble");
break;
return pybind11::dtype("D");
case DT::CLONG_DOUBLE:
case DT::VEC_CLONG_DOUBLE:
return pybind11::dtype("clongdouble");
break;
return pybind11::dtype("G");
case DT::BOOL:
return pybind11::dtype("bool"); // also "?"
break;
return pybind11::dtype("?");
case DT::UNDEFINED:
default:
throw std::runtime_error(
"dtype_to_numpy: Invalid Datatype '{...}'!"); // _s.format(dt)
break;
throw std::runtime_error("dtype_to_numpy: Invalid Datatype!");
}
}
} // namespace openPMD
10 changes: 5 additions & 5 deletions share/openPMD/download_samples.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,10 @@ Invoke-WebRequest https://github.com/openPMD/openPMD-example-datasets/raw/566b35
7z.exe x -r example-3d-bp4.tar
7z.exe x -r legacy_datasets.tar.gz
7z.exe x -r legacy_datasets.tar
Move-Item -Path example-3d\hdf5\* samples\git-sample\
Move-Item -Path example-thetaMode\hdf5\* samples\git-sample\thetaMode\
Move-Item -Path example-3d-bp4\* samples\git-sample\3d-bp4\
Move-Item -Path legacy_datasets\* samples\git-sample\legacy\
Move-Item -Path example-3d\hdf5\* -Destination samples\git-sample\ -Force
Move-Item -Path example-thetaMode\hdf5\* -Destination samples\git-sample\thetaMode\ -Force
Move-Item -Path example-3d-bp4\* -Destination samples\git-sample\3d-bp4\ -Force
Move-Item -Path legacy_datasets\* -Destination samples\git-sample\legacy\ -Force
Remove-Item -Recurse -Force example-3d*
Remove-Item -Recurse -Force example-thetaMode*
Remove-Item -Recurse -Force example-3d-bp4*
Expand All @@ -44,7 +44,7 @@ Remove-Item *.zip
# Ref.: https://github.com/openPMD/openPMD-viewer/issues/296
Invoke-WebRequest https://github.com/openPMD/openPMD-viewer/files/5655027/diags.zip -OutFile empty_alternate_fbpic.zip
Expand-Archive empty_alternate_fbpic.zip
Move-Item -Path empty_alternate_fbpic\diags\hdf5\data00000050.h5 samples\issue-sample\empty_alternate_fbpic_00000050.h5
Move-Item -Path empty_alternate_fbpic\diags\hdf5\data00000050.h5 -Destination samples\issue-sample\empty_alternate_fbpic_00000050.h5 -Force
Remove-Item -Recurse -Force empty_alternate_fbpic*

cd $orgdir
15 changes: 3 additions & 12 deletions src/IO/ADIOS/ADIOS2PreloadAttributes.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,22 +330,13 @@ AttributeWithShapeAndResource<T>::operator bool() const
}

#define OPENPMD_INSTANTIATE_GETATTRIBUTE(type) \
template struct AttributeWithShape<type>; \
template struct AttributeWithShapeAndResource<type>; \
template AttributeWithShape<type> PreloadAdiosAttributes::getAttribute( \
std::string const &name) const; \
template auto AdiosAttributes::getAttribute( \
size_t step, adios2::IO &IO, std::string const &name) const \
-> AttributeWithShapeAndResource<type>; \
template AttributeWithShapeAndResource< \
type>::AttributeWithShapeAndResource(AttributeWithShape<type> parent); \
template AttributeWithShapeAndResource<type>:: \
AttributeWithShapeAndResource( \
size_t len_in, \
type const \
*data_in, /* NOLINTNEXTLINE(bugprone-macro-parentheses) */ \
std::optional<std::vector<type>> resource_in); \
template AttributeWithShapeAndResource< \
type>::AttributeWithShapeAndResource(adios2::Attribute<type> attr); \
template AttributeWithShapeAndResource<type>::operator bool() const;
-> AttributeWithShapeAndResource<type>;
ADIOS2_FOREACH_TYPE_1ARG(OPENPMD_INSTANTIATE_GETATTRIBUTE)
#undef OPENPMD_INSTANTIATE_GETATTRIBUTE
} // namespace openPMD::detail
Expand Down
2 changes: 1 addition & 1 deletion test/python/unittest/API/APITest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1692,7 +1692,7 @@ def testDataset(self):
extent = [1, 1, 1]
obj = io.Dataset(data_type, extent)
if found_numpy:
d = np.array((1, 1, 1, ), dtype=np.int_)
d = np.array((1, 1, 1, ), dtype="l")
obj2 = io.Dataset(d.dtype, d.shape)
assert data_type == io.determine_datatype(d.dtype)
assert obj2.dtype == obj.dtype
Expand Down