{ai}[foss/2023a] DeePMD-kit v3.0.1, Horovod v0.28.1 w/ CUDA 12.1.1 TensorFlow 2.15.1 by pavelToman · Pull Request #22219 · easybuilders/easybuild-easyconfigs

pavelToman · 2025-01-27T16:40:00Z

(created using eb --new-pr)
resolves vscentrum/vsc-software-stack#487
requiers:

protobuf: {ai}[foss/2023a] DeePMD-kit v3.0.1, Horovod v0.28.1 w/ TensorFlow 2.13.0 #22217

…M-kit-3.0.1-foss-2023a-CUDA-12.1.1-with-LAMMPS-plugin.eb, Horovod-0.28.1-foss-2023a-CUDA-12.1.1-TensorFlow-2.15.1.eb

github-actions · 2025-01-27T16:42:19Z

Updated software `Horovod-0.28.1-foss-2023a-CUDA-12.1.1-TensorFlow-2.15.1.eb`

Diff against Horovod-0.28.1-foss-2022a-PyTorch-1.12.0.eb

easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2022a-PyTorch-1.12.0.eb

diff --git a/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2022a-PyTorch-1.12.0.eb b/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2023a-CUDA-12.1.1-TensorFlow-2.15.1.eb
index 90aa0d7222..82290c1cbf 100644
--- a/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2022a-PyTorch-1.12.0.eb
+++ b/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2023a-CUDA-12.1.1-TensorFlow-2.15.1.eb
@@ -2,33 +2,44 @@ easyblock = 'PythonBundle'
 
 name = 'Horovod'
 version = '0.28.1'
-local_pt_version = '1.12.0'
-versionsuffix = '-PyTorch-%s' % local_pt_version
+local_tf_version = '2.15.1'
+local_cuda_suffix = '-CUDA-%(cudaver)s'
+versionsuffix = local_cuda_suffix + '-TensorFlow-%s' % local_tf_version
 
 homepage = 'https://github.com/uber/horovod'
-description = """Horovod is a distributed training framework for TensorFlow, PyTorch and MXnet.
-This build only has PyTorch enabled."""
+description = "Horovod is a distributed training framework for TensorFlow."
 
-toolchain = {'name': 'foss', 'version': '2022a'}
+toolchain = {'name': 'foss', 'version': '2023a'}
 
 builddependencies = [
-    ('CMake', '3.23.1'),
+    ('CMake', '3.26.3'),
 ]
 dependencies = [
-    ('Python', '3.10.4'),
+    ('Python', '3.11.3'),
     ('PyYAML', '6.0'),
-    ('PyTorch', local_pt_version),
+    ('CUDA', '12.1.1', '', SYSTEM),
+    ('NCCL', '2.18.3', local_cuda_suffix),
+    ('TensorFlow', local_tf_version, local_cuda_suffix),
 ]
 
-preinstallopts = 'HOROVOD_WITH_MPI=1 '
-preinstallopts += 'HOROVOD_WITHOUT_TENSORFLOW=1 HOROVOD_WITH_PYTORCH=1 HOROVOD_WITHOUT_MXNET=1 '
+use_pip = True
+sanity_pip_check = True
+
+local_preinstallopts = 'module swap protobuf/3.21.9-GCCcore-12.3.0 && '
+local_preinstallopts += 'HOROVOD_WITH_MPI=1 HOROVOD_GPU_ALLREDUCE=NCCL HOROVOD_GPU_BROADCAST=NCCL '
+local_preinstallopts += 'HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITHOUT_MXNET=1 '
 
 exts_list = [
     ('cloudpickle', '2.2.1', {
         'checksums': ['d89684b8de9e34a2a43b3460fbca07d09d6e25ce858df4d5a44240403b6178f5'],
     }),
     ('horovod', version, {
-        'checksums': ['92a43f5a94c43907a56805bad15f19700c62ffc83b7ca483f9e104e229f67ef0'],
+        'patches': ['Horovod-0.28.1_support_flatbuffers_2.0.6.patch'],
+        'preinstallopts': local_preinstallopts,
+        'checksums': [
+            '92a43f5a94c43907a56805bad15f19700c62ffc83b7ca483f9e104e229f67ef0',
+            '9696ffb3b2bad1d6dd5a9f37bc58078ca7c585f933bcbec037036ad9fc0b297d',
+        ],
     }),
 ]

Diff against Horovod-0.28.1-foss-2022a-CUDA-11.7.0-PyTorch-1.13.1.eb

easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2022a-CUDA-11.7.0-PyTorch-1.13.1.eb

diff --git a/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2022a-CUDA-11.7.0-PyTorch-1.13.1.eb b/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2023a-CUDA-12.1.1-TensorFlow-2.15.1.eb
index 744c678169..82290c1cbf 100644
--- a/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2022a-CUDA-11.7.0-PyTorch-1.13.1.eb
+++ b/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2023a-CUDA-12.1.1-TensorFlow-2.15.1.eb
@@ -2,36 +2,44 @@ easyblock = 'PythonBundle'
 
 name = 'Horovod'
 version = '0.28.1'
-local_pt_version = '1.13.1'
+local_tf_version = '2.15.1'
 local_cuda_suffix = '-CUDA-%(cudaver)s'
-versionsuffix = local_cuda_suffix + '-PyTorch-%s' % local_pt_version
+versionsuffix = local_cuda_suffix + '-TensorFlow-%s' % local_tf_version
 
 homepage = 'https://github.com/uber/horovod'
-description = """Horovod is a distributed training framework for TensorFlow, PyTorch and MXnet.
-This build only has PyTorch enabled."""
+description = "Horovod is a distributed training framework for TensorFlow."
 
-toolchain = {'name': 'foss', 'version': '2022a'}
+toolchain = {'name': 'foss', 'version': '2023a'}
 
 builddependencies = [
-    ('CMake', '3.23.1'),
+    ('CMake', '3.26.3'),
 ]
 dependencies = [
-    ('Python', '3.10.4'),
+    ('Python', '3.11.3'),
     ('PyYAML', '6.0'),
-    ('CUDA', '11.7.0', '', SYSTEM),
-    ('NCCL', '2.12.12', local_cuda_suffix),
-    ('PyTorch', local_pt_version, local_cuda_suffix),
+    ('CUDA', '12.1.1', '', SYSTEM),
+    ('NCCL', '2.18.3', local_cuda_suffix),
+    ('TensorFlow', local_tf_version, local_cuda_suffix),
 ]
 
-preinstallopts = 'HOROVOD_WITH_MPI=1 HOROVOD_GPU_ALLREDUCE=NCCL HOROVOD_GPU_BROADCAST=NCCL '
-preinstallopts += 'HOROVOD_WITHOUT_TENSORFLOW=1 HOROVOD_WITH_PYTORCH=1 HOROVOD_WITHOUT_MXNET=1 '
+use_pip = True
+sanity_pip_check = True
+
+local_preinstallopts = 'module swap protobuf/3.21.9-GCCcore-12.3.0 && '
+local_preinstallopts += 'HOROVOD_WITH_MPI=1 HOROVOD_GPU_ALLREDUCE=NCCL HOROVOD_GPU_BROADCAST=NCCL '
+local_preinstallopts += 'HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITHOUT_MXNET=1 '
 
 exts_list = [
     ('cloudpickle', '2.2.1', {
         'checksums': ['d89684b8de9e34a2a43b3460fbca07d09d6e25ce858df4d5a44240403b6178f5'],
     }),
     ('horovod', version, {
-        'checksums': ['92a43f5a94c43907a56805bad15f19700c62ffc83b7ca483f9e104e229f67ef0'],
+        'patches': ['Horovod-0.28.1_support_flatbuffers_2.0.6.patch'],
+        'preinstallopts': local_preinstallopts,
+        'checksums': [
+            '92a43f5a94c43907a56805bad15f19700c62ffc83b7ca483f9e104e229f67ef0',
+            '9696ffb3b2bad1d6dd5a9f37bc58078ca7c585f933bcbec037036ad9fc0b297d',
+        ],
     }),
 ]

Diff against Horovod-0.28.1-foss-2022a-CUDA-11.7.0-PyTorch-1.12.1.eb

easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2022a-CUDA-11.7.0-PyTorch-1.12.1.eb

diff --git a/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2022a-CUDA-11.7.0-PyTorch-1.12.1.eb b/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2023a-CUDA-12.1.1-TensorFlow-2.15.1.eb
index 8485defceb..82290c1cbf 100644
--- a/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2022a-CUDA-11.7.0-PyTorch-1.12.1.eb
+++ b/easybuild/easyconfigs/h/Horovod/Horovod-0.28.1-foss-2023a-CUDA-12.1.1-TensorFlow-2.15.1.eb
@@ -2,36 +2,44 @@ easyblock = 'PythonBundle'
 
 name = 'Horovod'
 version = '0.28.1'
-local_pt_version = '1.12.1'
+local_tf_version = '2.15.1'
 local_cuda_suffix = '-CUDA-%(cudaver)s'
-versionsuffix = local_cuda_suffix + '-PyTorch-%s' % local_pt_version
+versionsuffix = local_cuda_suffix + '-TensorFlow-%s' % local_tf_version
 
 homepage = 'https://github.com/uber/horovod'
-description = """Horovod is a distributed training framework for TensorFlow, PyTorch and MXnet.
-This build only has PyTorch enabled."""
+description = "Horovod is a distributed training framework for TensorFlow."
 
-toolchain = {'name': 'foss', 'version': '2022a'}
+toolchain = {'name': 'foss', 'version': '2023a'}
 
 builddependencies = [
-    ('CMake', '3.23.1'),
+    ('CMake', '3.26.3'),
 ]
 dependencies = [
-    ('Python', '3.10.4'),
+    ('Python', '3.11.3'),
     ('PyYAML', '6.0'),
-    ('CUDA', '11.7.0', '', SYSTEM),
-    ('NCCL', '2.12.12', local_cuda_suffix),
-    ('PyTorch', local_pt_version, local_cuda_suffix),
+    ('CUDA', '12.1.1', '', SYSTEM),
+    ('NCCL', '2.18.3', local_cuda_suffix),
+    ('TensorFlow', local_tf_version, local_cuda_suffix),
 ]
 
-preinstallopts = 'HOROVOD_WITH_MPI=1 HOROVOD_GPU_ALLREDUCE=NCCL HOROVOD_GPU_BROADCAST=NCCL '
-preinstallopts += 'HOROVOD_WITHOUT_TENSORFLOW=1 HOROVOD_WITH_PYTORCH=1 HOROVOD_WITHOUT_MXNET=1 '
+use_pip = True
+sanity_pip_check = True
+
+local_preinstallopts = 'module swap protobuf/3.21.9-GCCcore-12.3.0 && '
+local_preinstallopts += 'HOROVOD_WITH_MPI=1 HOROVOD_GPU_ALLREDUCE=NCCL HOROVOD_GPU_BROADCAST=NCCL '
+local_preinstallopts += 'HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITHOUT_MXNET=1 '
 
 exts_list = [
     ('cloudpickle', '2.2.1', {
         'checksums': ['d89684b8de9e34a2a43b3460fbca07d09d6e25ce858df4d5a44240403b6178f5'],
     }),
     ('horovod', version, {
-        'checksums': ['92a43f5a94c43907a56805bad15f19700c62ffc83b7ca483f9e104e229f67ef0'],
+        'patches': ['Horovod-0.28.1_support_flatbuffers_2.0.6.patch'],
+        'preinstallopts': local_preinstallopts,
+        'checksums': [
+            '92a43f5a94c43907a56805bad15f19700c62ffc83b7ca483f9e104e229f67ef0',
+            '9696ffb3b2bad1d6dd5a9f37bc58078ca7c585f933bcbec037036ad9fc0b297d',
+        ],
     }),
 ]

pavelToman · 2025-01-28T09:47:41Z

Test report by @pavelToman
SUCCESS
Build succeeded for 3 out of 3 (3 easyconfigs in total)
node4012.donphan.os - Linux RHEL 8.8, x86_64, Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz, 1 x NVIDIA NVIDIA A2, 545.23.08, Python 3.6.8
See https://gist.github.com/pavelToman/17efd63f20d8113a8751102975af2595 for a full test report.

boegel · 2025-02-18T12:15:53Z

+        'easyblock': 'PythonPackage',
+        'source_urls': ['https://pypi.python.org/packages/source/d/deepmd-kit/'],
+        'sources': ['deepmd_kit-%(version)s.tar.gz'],
+        'preinstallopts': "export DP_VARIANT=cuda && module swap protobuf/3.21.9-GCCcore-12.3.0 && ",


see also #22217, would be nice if we could avoid having to use this...

laraPPr

small typo

…-kit301

…-plugin.eb to DeePMD-kit-3.0.1-foss-2023a-CUDA-12.1.1-with-LAMMPS-plugin.eb - fix name

…D-kit-3.0.1-foss-2023a-CUDA-12.1.1.eb - fix name

boegel · 2025-06-25T13:07:32Z

+        'easyblock': 'PythonPackage',
+        'source_urls': ['https://pypi.python.org/packages/source/d/deepmd-kit/'],
+        'sources': ['deepmd_kit-%(version)s.tar.gz'],
+        'preinstallopts': "export DP_VARIANT=cuda && module swap protobuf/3.21.9-GCCcore-12.3.0 && ",


Just to be clear: we need to get rid of this nasty hack by implementing support for swap_dependencies in framework, see also #22217 (comment)

…-CUDA-12.1.1-with-LAMMPS-plugin.eb

adding easyconfigs: DeePDM-kit-3.0.1-foss-2023a-CUDA-12.1.1.eb, DeePD…

c92058b

…M-kit-3.0.1-foss-2023a-CUDA-12.1.1-with-LAMMPS-plugin.eb, Horovod-0.28.1-foss-2023a-CUDA-12.1.1-TensorFlow-2.15.1.eb

pavelToman added new update labels Jan 27, 2025

pavelToman mentioned this pull request Jan 27, 2025

DeePMD-kit vscentrum/vsc-software-stack#487

Open

github-actions Bot removed the new label Jan 27, 2025

pavelToman changed the title ~~{ai}[foss/2023a] DeePDM-kit v3.0.1, Horovod v0.28.1 w/ CUDA 12.1.1, CUDA 12.1.1 TensorFlow 2.15.1, ...~~ {ai}[foss/2023a] DeePDM-kit v3.0.1, Horovod v0.28.1 w/ CUDA 12.1.1 TensorFlow 2.15.1 Jan 27, 2025

pavelToman added new and removed update labels Jan 27, 2025

boegel requested changes Feb 18, 2025

View reviewed changes

boegel self-assigned this May 20, 2025

laraPPr requested changes Jun 19, 2025

View reviewed changes

Comment thread ...build/easyconfigs/d/DeePDM-kit/DeePDM-kit-3.0.1-foss-2023a-CUDA-12.1.1-with-LAMMPS-plugin.eb Outdated

Merge branch 'easybuilders:develop' into 20250127173956_new_pr_DeePDM…

d15b433

…-kit301

github-actions Bot added update and removed new labels Jun 19, 2025

pavelToman changed the title ~~{ai}[foss/2023a] DeePDM-kit v3.0.1, Horovod v0.28.1 w/ CUDA 12.1.1 TensorFlow 2.15.1~~ {ai}[foss/2023a] DeePMD-kit v3.0.1, Horovod v0.28.1 w/ CUDA 12.1.1 TensorFlow 2.15.1 Jun 19, 2025

Update and rename DeePDM-kit-3.0.1-foss-2023a-CUDA-12.1.1-with-LAMMPS…

167431a

…-plugin.eb to DeePMD-kit-3.0.1-foss-2023a-CUDA-12.1.1-with-LAMMPS-plugin.eb - fix name

github-actions Bot added the new label Jun 19, 2025

Update and rename DeePDM-kit-3.0.1-foss-2023a-CUDA-12.1.1.eb to DeePM…

0222bd6

…D-kit-3.0.1-foss-2023a-CUDA-12.1.1.eb - fix name

github-actions Bot removed the new label Jun 19, 2025

laraPPr mentioned this pull request Jun 20, 2025

{chem}[foss/2023a] LAMMPS v2Aug2023_update2 w/ kokkos CUDA 12.1.1 with deepmd plugin #23167

Open

2 tasks

boegel requested changes Jun 25, 2025

View reviewed changes

Comment thread ...build/easyconfigs/d/DeePMD-kit/DeePMD-kit-3.0.1-foss-2023a-CUDA-12.1.1-with-LAMMPS-plugin.eb Outdated

boegel reviewed Jun 25, 2025

View reviewed changes

Delete easybuild/easyconfigs/d/DeePMD-kit/DeePMD-kit-3.0.1-foss-2023a…

bdbc89a

…-CUDA-12.1.1-with-LAMMPS-plugin.eb

github-actions Bot added the new label Jun 25, 2025

Thyre added the 2023a label Aug 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{ai}[foss/2023a] DeePMD-kit v3.0.1, Horovod v0.28.1 w/ CUDA 12.1.1 TensorFlow 2.15.1#22219

{ai}[foss/2023a] DeePMD-kit v3.0.1, Horovod v0.28.1 w/ CUDA 12.1.1 TensorFlow 2.15.1#22219
pavelToman wants to merge 5 commits into
easybuilders:developfrom
pavelToman:20250127173956_new_pr_DeePDM-kit301

pavelToman commented Jan 27, 2025 •

edited

Loading

Uh oh!

github-actions Bot commented Jan 27, 2025 •

edited

Loading

Uh oh!

pavelToman commented Jan 28, 2025

Uh oh!

boegel Feb 18, 2025

Uh oh!

laraPPr left a comment

Uh oh!

Uh oh!

Uh oh!

boegel Jun 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

pavelToman commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Updated software Horovod-0.28.1-foss-2023a-CUDA-12.1.1-TensorFlow-2.15.1.eb

Uh oh!

pavelToman commented Jan 28, 2025

Uh oh!

boegel Feb 18, 2025

Choose a reason for hiding this comment

Uh oh!

laraPPr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

boegel Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pavelToman commented Jan 27, 2025 •

edited

Loading

github-actions Bot commented Jan 27, 2025 •

edited

Loading

Updated software `Horovod-0.28.1-foss-2023a-CUDA-12.1.1-TensorFlow-2.15.1.eb`