Skip to content

Add an optional workflow that runs the HIP unit tests on a simulated MI300X - #440

Open
saqibkh wants to merge 1 commit into
llnl:masterfrom
saqibkh:add-gpu-sim-ci
Open

saqibkh wants to merge 1 commit into
llnl:masterfrom
saqibkh:add-gpu-sim-ci

Conversation

@saqibkh

@saqibkh saqibkh commented Nov 10, 2026

Copy link
Copy Markdown

The existing HIP job in CI.yml builds Kokkos (HIP) and ExaCA but does not run anything, because the runners have no GPU. This adds a separate, optional workflow .github/workflows/gpu-sim.yml that builds the same way (Kokkos 4.2.00, nlohmann_json 3.11.2, hipcc, Release) for gfx942 and runs the unit_test/ gtests on a simulated AMD Instinct MI300X using PantheonSim (no GPU needed). I am the author of PantheonSim, so please weigh this accordingly.

It touches no source file, does not change CI.yml and is not meant to be a required check. A simulator cannot reproduce hardware-specific behavior or timing, so it complements real-GPU testing and does not replace it.

What it does:

  • installs ROCm 7.1 and the simulated MI300X, then rocthrust-dev, rocprim-dev, OpenMPI and GoogleTest with apt
  • builds nlohmann_json and Kokkos with -DKokkos_ENABLE_HIP=ON -DKokkos_ARCH_AMD_GFX942=ON, then ExaCA with ExaCA_ENABLE_TESTING=ON, hipcc and the same MPI flags as the HIP job (without -Werror)
  • runs vgpu run ctest -R _HIP_ with up to 2 MPI ranks
  • runs on manual dispatch and on pull requests to master that touch src, unit_test, examples, cmake or CMakeLists.txt, with a concurrency group that cancels superseded runs

Measured:

  • Run on my fork: https://github.com/saqibkh/ExaCA/actions/runs/42828646659 (this is the verification; I did not run it locally, the machine I used has no ROCm).
  • The job passed: 14 of 14 selected ctest tests (HIP variants, 1 and 2 ranks), about 18 minutes in total, most of it simulator and ROCm setup; the tests themselves take about 17 seconds. An earlier run without the exclusions had 15 of 40 failing, for the reasons below.

Not covered (excluded in the workflow, with the reason in a comment). These HIP tests fail on the current sources and do not look like simulator problems, but I have not confirmed them on real hardware:

  • Orientation: the test builds grain_id_converted_host with create_mirror_view(HostSpace(), ...), which does not copy the device data, then compares it.
  • Interface (cell_update_tests): Kokkos aborts with "attempt to access inaccessible memory space (label=GrainID)", i.e. the test reads a device view on the host.
  • Nucleation: the test reassigns its local grain_id view to a copy and checks that, instead of the cell data the kernel modified.
  • Update on 2 ranks: MPI_Isend of a device pointer segfaults with the apt OpenMPI, which is not GPU-aware.
  • The SERIAL variants of every test (Kokkos built with HIP only for this job) and the example-input runs from CI.yml.

If you would rather fix those tests on a GPU backend first, I am happy to help; otherwise the exclusion list can be shrunk as they get fixed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant