{lib}[foss/2024a] dm-haiku v0.0.13, tensorstore v0.1.69 w/ CUDA 12.6.0 #22028

ThomasHoffmann77 · 2024-12-13T16:04:33Z

(created using eb --new-pr)

{ai}[gfbf/2024a] jax v0.4.34, ml_dtypes v0.5.0 w/ CUDA 12.6.0 WIP #21924

…store-0.1.69-foss-2024a.eb and patches: tensorstore-0.1.69_disable_third_party_downloads.patch

github-actions · 2024-12-13T16:05:25Z

Updated software `dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb`

Diff against dm-haiku-0.0.13-foss-2023a.eb

easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.13-foss-2023a.eb

diff --git a/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.13-foss-2023a.eb b/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb
index 532f9092e7..ff6c06f4ba 100644
--- a/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.13-foss-2023a.eb
+++ b/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb
@@ -1,31 +1,53 @@
+# update 0.0.12: Thomas Hoffmann (EMBL) 
 easyblock = 'PythonBundle'
 
 name = 'dm-haiku'
 version = '0.0.13'
+versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://github.com/deepmind/dm-haiku'
 description = """Haiku is a simple neural network library for JAX developed by some of the authors of Sonnet, a neural
 network library for TensorFlow."""
 
-toolchain = {'name': 'foss', 'version': '2023a'}
+toolchain = {'name': 'foss', 'version': '2024a'}
+
+builddependencies = [
+    ('poetry', '1.8.3')
+]
 
 dependencies = [
-    ('Python', '3.11.3'),
-    ('SciPy-bundle', '2023.07'),
-    ('jax', '0.4.25'),  # required by jmp, also provides absl-py
+    ('Python', '3.12.3'),
+    ('SciPy-bundle', '2024.05'),
+    ('jax', '0.4.34', versionsuffix),  # required by jmp, also provides absl-py
+    ('PyYAML', '6.0.2'),
+    ('CUDA', '12.6.0', '', SYSTEM),
+    ('tensorstore', '0.1.69'),
+    ('protobuf-python', '5.28.0'),
+    ('Optax', '0.2.4', versionsuffix),
 ]
 
 use_pip = True
 
 exts_list = [
     ('jmp', '0.0.4', {
-        'source_tmpl': SOURCE_PY3_WHL,
-        'checksums': ['6aa7adbddf2bd574b28c7faf6e81a735eb11f53386447896909c6968dc36807d'],
+        'checksums': ['5dfeb0fd7c7a9f72a70fff0aab9d0cbfae32a809c02f4037ff3485ceb33e1730'],
+    }),
+    ('flax', '0.10.2', {
+        'checksums': ['6f831350026ad48182ba6588bb4dd72dc1084985d9aca923254cb3e4c78d75f3'],
+    }),
+    ('nest_asyncio', '1.6.0', {
+        'checksums': ['6f172d5449aca15afd6c646851f4e31e02c598d553a667e38cafa997cfec55fe'],
+    }),
+    ('humanize', '4.11.0', {
+        'checksums': ['e66f36020a2d5a974c504bd2555cf770621dbdbb6d82f94a6857c0b1ea2608be'],
+    }),
+    ('orbax_checkpoint', '0.10.2', {
+        'modulename': 'orbax.checkpoint',
+        'checksums': ['e575ebe1f94e5cb6353ab8c9df81de0ca7cddc118645c3bfc17b8344f19d42f1'],
     }),
     ('dm_haiku', version, {
         'modulename': 'haiku',
-        'source_tmpl': SOURCE_PY3_WHL,
-        'checksums': ['ee9562c68a059f146ad07f555ca591cb8c11ef751afecc38353863562bd23f43'],
+        'checksums': ['029bb91b5b1edb0d3fe23304d3bf12a545ea6e485041f7f5d8c8d85ebcf6e17d'],
     }),
 ]

Diff against dm-haiku-0.0.12-foss-2023a-CUDA-12.1.1.eb

easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.12-foss-2023a-CUDA-12.1.1.eb

diff --git a/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.12-foss-2023a-CUDA-12.1.1.eb b/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb
index f66f10ec55..ff6c06f4ba 100644
--- a/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.12-foss-2023a-CUDA-12.1.1.eb
+++ b/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb
@@ -2,24 +2,28 @@
 easyblock = 'PythonBundle'
 
 name = 'dm-haiku'
-version = '0.0.12'
+version = '0.0.13'
 versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://github.com/deepmind/dm-haiku'
 description = """Haiku is a simple neural network library for JAX developed by some of the authors of Sonnet, a neural
 network library for TensorFlow."""
 
-toolchain = {'name': 'foss', 'version': '2023a'}
+toolchain = {'name': 'foss', 'version': '2024a'}
+
+builddependencies = [
+    ('poetry', '1.8.3')
+]
 
 dependencies = [
-    ('Python', '3.11.3'),
-    ('SciPy-bundle', '2023.07'),
-    ('jax', '0.4.25', versionsuffix),  # required by jmp, also provides absl-py
-    ('PyYAML', '6.0'),
-    ('CUDA', '12.1.1', '', SYSTEM),
-    ('tensorstore', '0.1.65'),
-    ('protobuf-python', '4.24.0'),
-    ('Optax', '0.2.2', versionsuffix),
+    ('Python', '3.12.3'),
+    ('SciPy-bundle', '2024.05'),
+    ('jax', '0.4.34', versionsuffix),  # required by jmp, also provides absl-py
+    ('PyYAML', '6.0.2'),
+    ('CUDA', '12.6.0', '', SYSTEM),
+    ('tensorstore', '0.1.69'),
+    ('protobuf-python', '5.28.0'),
+    ('Optax', '0.2.4', versionsuffix),
 ]
 
 use_pip = True
@@ -28,20 +32,22 @@ exts_list = [
     ('jmp', '0.0.4', {
         'checksums': ['5dfeb0fd7c7a9f72a70fff0aab9d0cbfae32a809c02f4037ff3485ceb33e1730'],
     }),
-    ('flax', '0.8.4', {
-        'checksums': ['968683f850198e1aa5eb2d9d1e20bead880ef7423c14f042db9d60848cb1c90b'],
+    ('flax', '0.10.2', {
+        'checksums': ['6f831350026ad48182ba6588bb4dd72dc1084985d9aca923254cb3e4c78d75f3'],
     }),
     ('nest_asyncio', '1.6.0', {
         'checksums': ['6f172d5449aca15afd6c646851f4e31e02c598d553a667e38cafa997cfec55fe'],
     }),
-    ('orbax_checkpoint', '0.5.18', {
+    ('humanize', '4.11.0', {
+        'checksums': ['e66f36020a2d5a974c504bd2555cf770621dbdbb6d82f94a6857c0b1ea2608be'],
+    }),
+    ('orbax_checkpoint', '0.10.2', {
         'modulename': 'orbax.checkpoint',
-        'preinstallopts': """sed -i 's/jax >= 0.4.25/&\\*/g' pyproject.toml &&""",
-        'checksums': ['29f5d311b412760bd6a2fecab3bdbf75407bc00dc6d0457d19478258ecc8fa6d'],
+        'checksums': ['e575ebe1f94e5cb6353ab8c9df81de0ca7cddc118645c3bfc17b8344f19d42f1'],
     }),
-    (name, version, {
+    ('dm_haiku', version, {
         'modulename': 'haiku',
-        'checksums': ['ba0b3acf71433156737fe342c486da11727e5e6c9e054245f4f9b8f0b53eb608'],
+        'checksums': ['029bb91b5b1edb0d3fe23304d3bf12a545ea6e485041f7f5d8c8d85ebcf6e17d'],
     }),
 ]

Diff against dm-haiku-0.0.9-foss-2022a-CUDA-11.7.0.eb

easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.9-foss-2022a-CUDA-11.7.0.eb

diff --git a/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.9-foss-2022a-CUDA-11.7.0.eb b/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb
index b2e1c44658..ff6c06f4ba 100644
--- a/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.9-foss-2022a-CUDA-11.7.0.eb
+++ b/easybuild/easyconfigs/d/dm-haiku/dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb
@@ -1,20 +1,29 @@
+# update 0.0.12: Thomas Hoffmann (EMBL) 
 easyblock = 'PythonBundle'
 
 name = 'dm-haiku'
-version = '0.0.9'
+version = '0.0.13'
 versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://github.com/deepmind/dm-haiku'
 description = """Haiku is a simple neural network library for JAX developed by some of the authors of Sonnet, a neural
 network library for TensorFlow."""
 
-toolchain = {'name': 'foss', 'version': '2022a'}
+toolchain = {'name': 'foss', 'version': '2024a'}
+
+builddependencies = [
+    ('poetry', '1.8.3')
+]
 
 dependencies = [
-    ('CUDA', '11.7.0', '', SYSTEM),
-    ('Python', '3.10.4'),
-    ('SciPy-bundle', '2022.05'),
-    ('jax', '0.3.25', versionsuffix),  # required by jmp, also provides absl-py
+    ('Python', '3.12.3'),
+    ('SciPy-bundle', '2024.05'),
+    ('jax', '0.4.34', versionsuffix),  # required by jmp, also provides absl-py
+    ('PyYAML', '6.0.2'),
+    ('CUDA', '12.6.0', '', SYSTEM),
+    ('tensorstore', '0.1.69'),
+    ('protobuf-python', '5.28.0'),
+    ('Optax', '0.2.4', versionsuffix),
 ]
 
 use_pip = True
@@ -23,9 +32,22 @@ exts_list = [
     ('jmp', '0.0.4', {
         'checksums': ['5dfeb0fd7c7a9f72a70fff0aab9d0cbfae32a809c02f4037ff3485ceb33e1730'],
     }),
-    (name, version, {
+    ('flax', '0.10.2', {
+        'checksums': ['6f831350026ad48182ba6588bb4dd72dc1084985d9aca923254cb3e4c78d75f3'],
+    }),
+    ('nest_asyncio', '1.6.0', {
+        'checksums': ['6f172d5449aca15afd6c646851f4e31e02c598d553a667e38cafa997cfec55fe'],
+    }),
+    ('humanize', '4.11.0', {
+        'checksums': ['e66f36020a2d5a974c504bd2555cf770621dbdbb6d82f94a6857c0b1ea2608be'],
+    }),
+    ('orbax_checkpoint', '0.10.2', {
+        'modulename': 'orbax.checkpoint',
+        'checksums': ['e575ebe1f94e5cb6353ab8c9df81de0ca7cddc118645c3bfc17b8344f19d42f1'],
+    }),
+    ('dm_haiku', version, {
         'modulename': 'haiku',
-        'checksums': ['97752b32cdbe5a3e2d1c60ea884d33eb4b36e7d410000f0d61b571417c925435'],
+        'checksums': ['029bb91b5b1edb0d3fe23304d3bf12a545ea6e485041f7f5d8c8d85ebcf6e17d'],
     }),
 ]

Updated software `Optax-0.2.4-gfbf-2024a-CUDA-12.6.0.eb`

Diff against Optax-0.2.2-gfbf-2023a-CUDA-12.1.1.eb

easybuild/easyconfigs/o/Optax/Optax-0.2.2-gfbf-2023a-CUDA-12.1.1.eb

diff --git a/easybuild/easyconfigs/o/Optax/Optax-0.2.2-gfbf-2023a-CUDA-12.1.1.eb b/easybuild/easyconfigs/o/Optax/Optax-0.2.4-gfbf-2024a-CUDA-12.6.0.eb
index 61000b1388..b883757d20 100644
--- a/easybuild/easyconfigs/o/Optax/Optax-0.2.2-gfbf-2023a-CUDA-12.1.1.eb
+++ b/easybuild/easyconfigs/o/Optax/Optax-0.2.4-gfbf-2024a-CUDA-12.6.0.eb
@@ -1,32 +1,32 @@
 easyblock = 'PythonBundle'
 
 name = 'Optax'
-version = '0.2.2'
+version = '0.2.4'
 versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://github.com/deepmind/optax'
 description = """Optax is a gradient processing and optimization library for JAX."""
 
-toolchain = {'name': 'gfbf', 'version': '2023a'}
+toolchain = {'name': 'gfbf', 'version': '2024a'}
 
 dependencies = [
-    ('CUDA', '12.1.1', '', SYSTEM),
-    ('Python', '3.11.3'),
-    ('SciPy-bundle', '2023.07'),
-    ('jax', '0.4.25', versionsuffix),
+    ('CUDA', '12.6.0', '', SYSTEM),
+    ('Python', '3.12.3'),
+    ('SciPy-bundle', '2024.05'),
+    ('jax', '0.4.34', versionsuffix),
 ]
 
 use_pip = True
 
 exts_list = [
-    ('toolz', '0.12.1', {
-        'checksums': ['ecca342664893f177a13dac0e6b41cbd8ac25a358e5f215316d43e2100224f4d'],
+    ('chex', '0.1.87', {
+        'checksums': ['0096d89cc8d898bb521ef4bfbf5c24549022b0e5b301f529ab57238896fe6c5d'],
     }),
-    ('chex', '0.1.86', {
-        'checksums': ['e8b0f96330eba4144659e1617c0f7a57b161e8cbb021e55c6d5056c7378091d1'],
+    ('toolz', '1.0.0', {
+        'checksums': ['2c86e3d9a04798ac556793bced838816296a2f085017664e4995cb40a1047a02'],
     }),
     ('optax', version, {
-        'checksums': ['f09bf790ef4b09fb9c35f79a07594c6196a719919985f542dc84b0bf97812e0e'],
+        'checksums': ['4e05d3d5307e6dde4c319187ae36e6cd3a0c035d4ed25e9e992449a304f47336'],
     }),
 ]

Diff against Optax-0.2.2-gfbf-2023a.eb

easybuild/easyconfigs/o/Optax/Optax-0.2.2-gfbf-2023a.eb

diff --git a/easybuild/easyconfigs/o/Optax/Optax-0.2.2-gfbf-2023a.eb b/easybuild/easyconfigs/o/Optax/Optax-0.2.4-gfbf-2024a-CUDA-12.6.0.eb
index d447a77bbd..b883757d20 100644
--- a/easybuild/easyconfigs/o/Optax/Optax-0.2.2-gfbf-2023a.eb
+++ b/easybuild/easyconfigs/o/Optax/Optax-0.2.4-gfbf-2024a-CUDA-12.6.0.eb
@@ -1,30 +1,32 @@
 easyblock = 'PythonBundle'
 
 name = 'Optax'
-version = '0.2.2'
+version = '0.2.4'
+versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://github.com/deepmind/optax'
 description = """Optax is a gradient processing and optimization library for JAX."""
 
-toolchain = {'name': 'gfbf', 'version': '2023a'}
+toolchain = {'name': 'gfbf', 'version': '2024a'}
 
 dependencies = [
-    ('Python', '3.11.3'),
-    ('SciPy-bundle', '2023.07'),
-    ('jax', '0.4.25'),
+    ('CUDA', '12.6.0', '', SYSTEM),
+    ('Python', '3.12.3'),
+    ('SciPy-bundle', '2024.05'),
+    ('jax', '0.4.34', versionsuffix),
 ]
 
 use_pip = True
 
 exts_list = [
-    ('toolz', '0.12.1', {
-        'checksums': ['ecca342664893f177a13dac0e6b41cbd8ac25a358e5f215316d43e2100224f4d'],
+    ('chex', '0.1.87', {
+        'checksums': ['0096d89cc8d898bb521ef4bfbf5c24549022b0e5b301f529ab57238896fe6c5d'],
     }),
-    ('chex', '0.1.86', {
-        'checksums': ['e8b0f96330eba4144659e1617c0f7a57b161e8cbb021e55c6d5056c7378091d1'],
+    ('toolz', '1.0.0', {
+        'checksums': ['2c86e3d9a04798ac556793bced838816296a2f085017664e4995cb40a1047a02'],
     }),
     ('optax', version, {
-        'checksums': ['f09bf790ef4b09fb9c35f79a07594c6196a719919985f542dc84b0bf97812e0e'],
+        'checksums': ['4e05d3d5307e6dde4c319187ae36e6cd3a0c035d4ed25e9e992449a304f47336'],
     }),
 ]

Diff against Optax-0.1.7-foss-2022a-CUDA-11.7.0.eb

easybuild/easyconfigs/o/Optax/Optax-0.1.7-foss-2022a-CUDA-11.7.0.eb

diff --git a/easybuild/easyconfigs/o/Optax/Optax-0.1.7-foss-2022a-CUDA-11.7.0.eb b/easybuild/easyconfigs/o/Optax/Optax-0.2.4-gfbf-2024a-CUDA-12.6.0.eb
index 4fd9f0da8a..b883757d20 100644
--- a/easybuild/easyconfigs/o/Optax/Optax-0.1.7-foss-2022a-CUDA-11.7.0.eb
+++ b/easybuild/easyconfigs/o/Optax/Optax-0.2.4-gfbf-2024a-CUDA-12.6.0.eb
@@ -1,39 +1,32 @@
 easyblock = 'PythonBundle'
 
 name = 'Optax'
-version = '0.1.7'
+version = '0.2.4'
 versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://github.com/deepmind/optax'
 description = """Optax is a gradient processing and optimization library for JAX."""
 
-toolchain = {'name': 'foss', 'version': '2022a'}
+toolchain = {'name': 'gfbf', 'version': '2024a'}
 
 dependencies = [
-    ('CUDA', '11.7.0', '', SYSTEM),
-    ('Python', '3.10.4'),
-    ('dm-haiku', '0.0.9', versionsuffix),
-    ('dm-tree', '0.1.8'),
-    ('typing-extensions', '4.3.0'),
-]
-
-builddependencies = [
-    ('pytest-xdist', '2.5.0'),
+    ('CUDA', '12.6.0', '', SYSTEM),
+    ('Python', '3.12.3'),
+    ('SciPy-bundle', '2024.05'),
+    ('jax', '0.4.34', versionsuffix),
 ]
 
 use_pip = True
 
 exts_list = [
-    ('toolz', '0.12.0', {
-        'checksums': ['88c570861c440ee3f2f6037c4654613228ff40c93a6c25e0eba70d17282c6194'],
+    ('chex', '0.1.87', {
+        'checksums': ['0096d89cc8d898bb521ef4bfbf5c24549022b0e5b301f529ab57238896fe6c5d'],
     }),
-    ('chex', '0.1.6', {
-        'checksums': ['adb5d2352b5f0d248ccf594be1b1bf9ee7a2bee2a57f0eac78547538d479b0e7'],
+    ('toolz', '1.0.0', {
+        'checksums': ['2c86e3d9a04798ac556793bced838816296a2f085017664e4995cb40a1047a02'],
     }),
     ('optax', version, {
-        'checksums': ['6a5a848bc5e55e619b187c749fdddc4a5443ea14be85cc769f995779865c110d'],
-        # ignore equivalence_test.py which imports flax, which depends on optax
-        'runtest': 'python -m pytest -n %(parallel)s --pyargs --ignore-glob="**/equivalence_test.py" optax',
+        'checksums': ['4e05d3d5307e6dde4c319187ae36e6cd3a0c035d4ed25e9e992449a304f47336'],
     }),
 ]

Updated software `tensorstore-0.1.69-foss-2024a.eb`

Diff against tensorstore-0.1.65-foss-2023a.eb

easybuild/easyconfigs/t/tensorstore/tensorstore-0.1.65-foss-2023a.eb

diff --git a/easybuild/easyconfigs/t/tensorstore/tensorstore-0.1.65-foss-2023a.eb b/easybuild/easyconfigs/t/tensorstore/tensorstore-0.1.69-foss-2024a.eb
index 5dd271c0d8..e9412298ce 100644
--- a/easybuild/easyconfigs/t/tensorstore/tensorstore-0.1.65-foss-2023a.eb
+++ b/easybuild/easyconfigs/t/tensorstore/tensorstore-0.1.69-foss-2024a.eb
@@ -1,59 +1,391 @@
-# Thomas Hoffmann, EMBL Heidlelberg, [email protected], 2024/10
+# Thomas Hoffmann, EMBL Heidlelberg, [email protected], 2024/12
 easyblock = 'PythonBundle'
 
 name = 'tensorstore'
-version = '0.1.65'
+version = '0.1.69'
 
 homepage = 'https://github.com/google/tensorstore'
 description = """TensorStore is an open-source C++ and Python software library designed for      
 storage and manipulation of large multi-dimensional arrays."""
 
-toolchain = {'name': 'foss', 'version': '2023a'}
-builddependencies = [
-    ('NASM', '2.16.01'),
-    ('pybind11', '2.11.1'),
-    ('Bazel', '6.3.1'),
+toolchain = {'name': 'foss', 'version': '2024a'}
 
+use_pip = True
 
+builddependencies = [
+    ('pybind11', '2.12.0'),
+    ('Bazel', '7.4.1'),  # .bazelversion has 7.4.0
+    ('PyYAML', '6.0.2'),
+    # ('NASM', '2.16.03'),  # TODO syslib
+    # ('Abseil', '20240722.0'),  # TODO syslib
+    # ('RE2', '2024-07-02'),  # TODO syslib
+    # ('protobuf', '28.0'),  # TODO syslib
+    # ('c-ares', '1.34.3'),  # TODO syslib
+    # ('OpenSSL','3', '', SYSTEM)]
+]
 
-    ('PyYAML', '6.0'),
-    ('zlib', '1.2.13'),
-    ('LibTIFF', '4.5.0'),
-    ('snappy', '1.1.10'),
-    ('Brotli', '1.0.9'),
-    ('protobuf', '24.0'),
+dependencies = [
+    ('Python', '3.12.3'),
+    ('SciPy-bundle', '2024.05'),
+    ('ml_dtypes', '0.5.0'),
+    ('zlib', '1.3.1'),
+    ('LibTIFF', '4.6.0'),
+    ('libpng', '1.6.43'),
+    # ('libavif', '1.1.1'),
+    ('cURL', '8.7.1'),
     ('bzip2', '1.0.8'),
-    ('zstd', '1.5.5'),
-    ('libwebp', '1.3.1'),
-    ('nlohmann_json', '3.11.2'),
-    ('Blosc', '1.21.5'),
+    # ('snappy', '1.1.10'),  # TODO syslib
+    # ('Brotli', '1.1.0'), # TODO syslib
+    # ('libwebp', '1.4.0'),  # TODO syslib
+    # ('nlohmann_json', '3.11.3'),  # TODO syslib
+    # ('Blosc', '1.21.6'),  # TODO  # TODO syslib
+    # ('tinyxml2', '10.0.0'), TODO  # TODO syslib
 ]
 
-dependencies = [
-    ('Python', '3.11.3'),
-    ('SciPy-bundle', '2023.07'),
-    ('ml_dtypes', '0.3.2'),
+_systemlibs = [
+    'numpy',
+    'com_github_pybind_pybind11',
+    'net_zlib',
+    'jpeg',
+    'png',
+    'org_lz4',
+    'libtiff',
+    'se_curl',
+    'net_zstd',
+    'org_tukaani_xz',
+    'com_google_brotli',
+    'org_sourceware_bzip2',
+    'org_nghttp2',
+    # TODO: fix include paths for related system.BUILD.bzl
+    # 'com_github_nlohmann_json',  #TODO
+    # 'blake3',  #TODO
+    # 'org_aomedia_avif',  #TODO
+    # 'com_github_cares_cares', # TODO
+    # 'libwebp',  # TODO
+    # 'nasm',  # TODO
+    # 'com_google_absl', TODO
+    # 'com_google_snappy',  # TODO
+    # 'tinyxml2', TODO
+    # 'org_blosc_cblosc',  #TODO
 ]
 
-use_pip = True
+# Generate list of includ paths and link path for syslibs to be passed to Bazel as --copt and --linkopt
+_bzl_include_copts = ' '.join(['--copt=-I$EBROOT%s/include' % x for x in [
+    'PYBIND11',
+    'LIBTIFF',
+    'LIBJPEGMINTURBO/',
+    'ZLIB',
+    'LIBPNG',
+    'LZ4',
+    'ZSTD',
+    'CURL',
+    'XZ',
+    'BZIP2',
+    # '$EBROOTLIBAVIF',
+    # '$EBROOTTINYXML2',
+    # '$EBROOTRE2',
+    # '$EBROOTCMINARES',
+    # '$EBROOTABSEIL',
+    # '$EBROOTSNAPPY',
+    # '$EBROOTBLOSC',
+    # '$EBROOTLIBWEBP',
+    # '$EBROOTNLOHMANN_JSON'
+]])
+_bzl_linkopts = _bzl_include_copts.replace('copt', 'linkopt').replace('include', 'lib').replace('-I', '-L')
+
+# Download all bazel third_party artifacts by components and provide them in _archives_dir.
+# Third_party urls are modified to use file:// instead of https:// by patch _disable_third_party_downloads.
+_archives_dir = '%(builddir)s/archives'
+_extract_cmd = 'mkdir -p %(builddir)s/archives && cp %s ' + _archives_dir
+default_easyblock = 'Binary'
+# templates for Bazel sources download:
+_sources = [{
+    'download_filename': '%(version)s.tar.gz',
+    'filename': '%(name)s-%(version)s.tar.gz',
+    'extract_cmd': _extract_cmd,
+}]
+_sources_v = [{
+    'download_filename': 'v%(version)s.tar.gz',
+    'filename': '%(name)s-%(version)s.tar.gz',
+    'extract_cmd': _extract_cmd,
+}]
+_sources_vzip = [{
+    'download_filename': 'v%(version)s.zip',
+    'filename': '%(name)s-%(version)s.zip',
+    'extract_cmd': _extract_cmd,
+}]
+_sources_namedotversion = [{
+    'download_filename': '%(name)s.%(version)s.tar.gz',
+    'filename': '%(name)s-%(version)s.tar.gz',
+    'extract_cmd': _extract_cmd,
+}]
+_sources_nameminvversion = [{
+    'download_filename': '%(name)s-v%(version)s.tar.gz',
+    'filename': '%(name)s-%(version)s.tar.gz',
+    'extract_cmd': _extract_cmd,
+}]
+_sources_nameminversion = [{
+    'filename': '%(name)s-%(version)s.tar.gz',
+    'extract_cmd': _extract_cmd,
+}]
+_sources_nameminversionbz2 = [{
+    'filename': '%(name)s-%(version)s.tar.bz2',
+    'extract_cmd': _extract_cmd,
+}]
+_sources_nameminversionzip = [{
+    'filename': '%(name)s-%(version)s.zip',
+    'extract_cmd': _extract_cmd,
+}]
 
+default_component_specs = {
+    'install_cmd': '#',
+    'source_urls': [
+        "https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/bazelbuild/%(name)s/archive/",
+        "https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/bazelbuild/%(name)s/releases" +
+        "/download/%(version)s/",
+        "https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/bazel-contrib/%(name)s/archive/",
+        "https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/bazel-contrib/%(name)s/releases/download/" +
+        "v%(version)s/",
+        "https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/%(name)s/%(name)s/archive/",
+        "https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/google/%(name)s/archive",
+        'https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/google/%(name)s/releases/download/' +
+        '%(version)s/',
+        'https://storage.googleapis.com/tensorstore-bazel-mirror/sourceforge.net/projects/%(name)s/files/%(name)s/' +
+        '%(version)s'
+    ],
+    'sources': _sources,
+}
+components = [
+    ('bazel_features', '1.17.0', {
+        'sources': _sources_v,
+        'checksums': [{'bazel_features-1.17.0.tar.gz':
+                       'bdc12fcbe6076180d835c9dd5b3685d509966191760a0eb10b276025fcb76158'}],
+    }),
+    ('rules_proto', '6.0.2', {
+        'checksums': [{'rules_proto-6.0.2.tar.gz': '6fb6767d1bef535310547e03247f7518b03487740c11b6c6adb7952033fe1295'}],
+    }),
+    ('apple_support', '1.17.1', {
+        'sources': _sources_namedotversion,
+        'checksums': [{'apple_support-1.17.1.tar.gz':
+                       'b53f6491e742549f13866628ddffcc75d1f3b2d6987dc4f14a16b242113c890b'}],
+    }),
+    ('toolchains_llvm', '1.2.0', {
+        'sources': _sources_nameminvversion,
+        'checksums': [{'toolchains_llvm-1.2.0.tar.gz':
+                       'e3fb6dc6b77eaf167cb2b0c410df95d09127cbe20547e5a329c771808a816ab4'}],
+    }),
+    ('bazel-skylib', '1.7.1', {
+        'checksums': [{'bazel-skylib-1.7.1.tar.gz':
+                       'e3fea03ff75a9821e84199466799ba560dbaebb299c655b5307f4df1e5970696'}],
+    }),
+    ('platforms', '0.0.10', {
+        'sources': _sources_nameminversion,
+        'checksums': [{'platforms-0.0.10.tar.gz': '218efe8ee736d26a3572663b374a253c012b716d8af0c07e842e82f238a0a7ee'}],
+    }),
+    ('rules_cc', '0.0.13', {
+        'sources': _sources_nameminversion,
+        'checksums': [{'rules_cc-0.0.13.tar.gz': 'd9bdd3ec66b6871456ec9c965809f43a0901e692d754885e89293807762d3d80'}],
+    }),
+    ('protobuf', '28.2', {
+        'source_urls': [
+            "https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/protocolbuffers/protobuf/archive/"
+        ],
+        'sources': _sources_v,
+        'checksums': [{'protobuf-28.2.tar.gz': 'b2340aa47faf7ef10a0328190319d3f3bee1b24f426d4ce8f4253b6f27ce16db'}],
+    }),
+    ('boringssl', '20220720', {
+        'sources': [{
+            'download_filename': '098695591f3a2665fccef83a3732ecfc99acdcdd.tar.gz',
+            'filename': '%(name)s-%(version)s-098695591f3a2665fccef83a3732ecfc99acdcdd.tar.gz',
+            'extract_cmd': _extract_cmd,
+        }],
+        'checksums': [{'boringssl-20220720-098695591f3a2665fccef83a3732ecfc99acdcdd.tar.gz':
+                       'e141448cf6f686b6e9695f6b6459293fd602c8d51efe118a83106752cf7e1280'}],
+    }),
+    ('riegeli', '20231212', {
+        'sources': [{
+            'download_filename': 'e54e4300f61cc113e5da816952d20ae070fe7219.tar.gz',
+            'filename': '%(name)s-%(version)s-e54e4300f61cc113e5da816952d20ae070fe7219.tar.gz',
+            'extract_cmd': _extract_cmd,
+        }],
+        'checksums': [{'riegeli-20231212-e54e4300f61cc113e5da816952d20ae070fe7219.tar.gz':
+                      'c2a40a1a422fe513faa198b325865022ccfbb1ca6317ef5f51a8b60d7c2bdc4c'}],
+    }),
+    ('grpc', '1.66.1', {
+        'sources': _sources_v,
+        'checksums': [{'grpc-1.66.1.tar.gz': '79ed4ab72fa9589b20f8b0b76c16e353e4cfec1d773d33afad605d97b5682c61'}],
+    }),
+    ('abseil-cpp', '20240722.0', {
+        'source_urls': ['https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/abseil/abseil-cpp/archive'],
+        'checksums': [{'abseil-cpp-20240722.0.tar.gz':
+                       'f50e5ac311a81382da7fa75b97310e4b9006474f9560ac46f54a9967f07d4ae3'}],
+    }),
+    ('json', '3.11.3', {
+        'source_urls': ['https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/nlohmann/json/archive'],
+        'sources': _sources_vzip,
+        'checksums': [{'json-3.11.3.zip': '04022b05d806eb5ff73023c280b68697d12b93e1b7267a0b22a1a39ec7578069'}],
+    }),
+    ('half', '2.1.0', {
+        'sources': _sources_nameminversionzip,
+        'checksums': [{'half-2.1.0.zip': 'ad1788afe0300fa2b02b0d1df128d857f021f92ccf7c8bddd07812685fa07a25'}],
+    }),
+    ('BLAKE3', '1.5.4', {
+        'source_urls': [
+            'https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/BLAKE3-team/BLAKE3/archive/'
+        ],
+        'checksums': [{'BLAKE3-1.5.4.tar.gz': 'ddd24f26a31d23373e63d9be2e723263ac46c8b6d49902ab08024b573fd2a416'}],
+    }),
+    ('re2', '2024-07-02', {
+        'sources': _sources_nameminversion,
+        'checksums': [{'re2-2024-07-02.tar.gz': 'eb2df807c781601c14a260a507a5bb4509be1ee626024cb45acbd57cb9d4032b'}],
+    }),
+    ('rules_pkg', '1.0.1', {
+        'checksums': [{'rules_pkg-1.0.1.tar.gz': '23005750a27aabfd5975a3d5aeac9542371cbfa24d3ad74e47f80b84547754da'}],
+    }),
+    ('tinyxml2', '10.0.0', {
+        'source_urls': [
+            'https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/leethomason/tinyxml2/archive/'
+        ],
+        'checksums': [{'tinyxml2-10.0.0.tar.gz': '3bdf15128ba16686e69bce256cc468e76c7b94ff2c7f391cc5ec09e40bff3839'}],
+    }),
+    ('c-blosc', '1.21.6', {
+        'source_urls': ['https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/Blosc/c-blosc/archive/'],
+        'sources': _sources_vzip,
+        'checksums': [{'c-blosc-1.21.6.zip': '1919c97d55023c04aa8771ea8235b63e9da3c22e3d2a68340b33710d19c2a2eb'}],
+    }),
+    ('googleapis', '20240910', {
+        'sources': [{
+            'download_filename': '32bc03653260356351854429bd7e2dfbf670d352.tar.gz',
+            'filename': '%(name)s-%(version)s-32bc03653260356351854429bd7e2dfbf670d352.tar.gz',
+            'extract_cmd': _extract_cmd,
+        }],
+        'checksums': [{'googleapis-20240910-32bc03653260356351854429bd7e2dfbf670d352.tar.gz':
+                       '46ca6d9a6349c3845334dde2d55d482a11e7c1072a9085b89b6c1e94cdeb2d3e'}],
+    }),
+    ('snappy', '20240910', {
+        'sources': [{
+            'download_filename': '32ded457c0b1fe78ceb8397632c416568d6714a0.tar.gz',
+            'filename': '%(name)s-%(version)s-32ded457c0b1fe78ceb8397632c416568d6714a0.tar.gz',
+            'extract_cmd': _extract_cmd,
+        }],
+        'checksums': [{'snappy-20240910-32ded457c0b1fe78ceb8397632c416568d6714a0.tar.gz':
+                       '677d1dd8172bac1862e6c8d7bbe1fe9fb2320cfd11ee04756b1ef8b3699c6135'}],
+    }),
+    ('libavif', '1.0.4', {
+        'source_urls': [
+            'https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/AOMediaCodec/libavif/archive'
+        ],
+        'sources': _sources_v,
+        'checksums': [{'libavif-1.0.4.tar.gz': 'dc56708c83a4b934a8af2b78f67f866ba2fb568605c7cf94312acf51ee57d146'}],
+    }),
+    ('libwebp', '1.4.0', {
+        'source_urls': [
+            'https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/webmproject/libwebp/archive'
+        ],
+        'sources': _sources_v,
+        'checksums': [{'libwebp-1.4.0.tar.gz': '12af50c45530f0a292d39a88d952637e43fb2d4ab1883c44ae729840f7273381'}],
+    }),
+    ('c-ares', '1.33.1', {
+        'sources': _sources_v,
+        'checksums': [{'c-ares-1.33.1.tar.gz': 'b3d127d8357863eb465053ce9308b79d9b00314f92ec09df056221a1a45c2fef'}],
+    }),
+    ('aomedia', '20221101', {
+        'source_urls': [
+            'https://storage.googleapis.com/tensorstore-bazel-mirror/aomedia.googlesource.com/aom/+archive/'
+        ],
+        'sources': [{
+            'download_filename': 'd730cef03ac754f2b6a233e926cd925d8ce8de81.tar.gz',
+            'filename': '%(name)s-%(version)s-d730cef03ac754f2b6a233e926cd925d8ce8de81.tar.gz',
+            'extract_cmd': _extract_cmd,
+        }],
+        'checksums': [{'aomedia-20221101-d730cef03ac754f2b6a233e926cd925d8ce8de81.tar.gz':
+                       '7f87960be61edb846e25b7d10e4e41aef6d652f62270e19172e7cafae9b536ad'}],
+    }),
+    ('videolan-dav1d', '1.4.3', {
+        'source_urls': ['https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/videolan/dav1d/archive'],
+        'checksums': [{'videolan-dav1d-1.4.3.tar.gz':
+                       '88a023e58d955e0886faf49c72940e0e90914a948a8e60c1326ce3e09e7a6099'}],
+    }),
+    ('libyuv', '20240925', {
+        'source_urls': [
+            'https://storage.googleapis.com/tensorstore-bazel-mirror/chromium.googlesource.com/libyuv/libyuv/+archive'
+        ],
+        'sources': [{
+            'download_filename': '61bf0b61f7799c02eddc87b65c0fc542011cfd2b.tar.gz',
+            'filename': '%(name)s-%(version)s-61bf0b61f7799c02eddc87b65c0fc542011cfd2b.tar.gz',
+            'extract_cmd': _extract_cmd,
+        }],
+        'checksums': [{'libyuv-20240925-61bf0b61f7799c02eddc87b65c0fc542011cfd2b.tar.gz':
+                       'a869a841acb0579d85354a227562b15a603e35f5112394a48955eda20d736f46'}],
+    }),
+    ('nasm', '2.15.05', {
+        'source_urls': [
+            'https://storage.googleapis.com/tensorstore-bazel-mirror/www.nasm.us/pub/%(name)s/releasebuilds/%(version)s'
+        ],
+        'sources': _sources_nameminversionbz2,
+        'checksums': [{'nasm-2.15.05.tar.bz2': '3c4b8339e5ab54b1bcb2316101f8985a5da50a3f9e504d43fa6f35668bee2fd0'}],
+
+    }),
+    ('xds', '20240911', {
+        'source_urls': ['https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/cncf/xds/archive/'],
+        'sources': [{
+            'download_filename': 'b4127c9b8d78b77423fd25169f05b7476b6ea932.tar.gz',
+            'filename': '%(name)s-%(version)s-b4127c9b8d78b77423fd25169f05b7476b6ea932.tar.gz',
+            'extract_cmd': _extract_cmd,
+        }],
+        'checksums': [{'xds-20240911-b4127c9b8d78b77423fd25169f05b7476b6ea932.tar.gz':
+                       'aa5f1596bbef3f277dcf4700e4c1097b34301ae66f3b79cd731e3adfbaff2f8f'}],
+    }),
+    ('data-plane-api', '20240911', {
+        'source_urls': [
+            'https://storage.googleapis.com/tensorstore-bazel-mirror/github.com/envoyproxy/data-plane-api/archive/'
+        ],
+        'sources': [{
+            'download_filename': '19420fea29dd86aa3b25172151f0311476a66599.tar.gz',
+            'filename': '%(name)s-%(version)s-19420fea29dd86aa3b25172151f0311476a66599.tar.gz',
+            'extract_cmd': _extract_cmd,
+        }],
+        'checksums': [{'data-plane-api-20240911-19420fea29dd86aa3b25172151f0311476a66599.tar.gz':
+                       'd84dcb4d88a961fcf6aa5ea1ff479180bd9e58e41ad82d8dca3beacac2fb3bdc'}],
+
+    }),
+    ('cel-spec', '0.16.2', {
+        'sources': _sources_vzip,
+        'checksums': [{'cel-spec-0.16.2.zip': '13234d9622ef4b7660b6259498a6f02f001fd260d42fa21e67b28fd4c4e0d344'}],
+
+    }),
+]
 
-local_ts_useebbazel = """sed -i 's/bazel_path =.*/"""
-local_ts_useebbazel += """bazel_path = os.path.join(os.getenv("EBROOTBAZEL"),"bin", "bazel")/g'"""
-local_ts_useebbazel += " bazelisk.py&& "  # TODO: patch?
-local_ts_version = """sed -i "s/use_scm_version=/version='%(version)s',&/g" setup.py&&"""
-local_ts_bzl_exp = """export TENSORSTORE_BAZEL_STARTUP_OPTIONS='--output_user_root %(builddir)s/cache' &&"""
-# inject CFLAGS:
-local_ts_bzl_exp += """export TENSORSTORE_BAZEL_BUILD_OPTIONS="$(for i in  $CFLAGS;do echo --copt=$i; done)" &&"""
+# generate third_party/archives.bzl in order to set EB_OFFLINE_ARCHIVES to _archives_dir
+# (see patch _disable_third_party_downloads.patch):
+_prebuild_gen_archivesbzl = """echo 'EB_OFFLINE_ARCHIVES="%s"' > third_party/archives.bzl &&""" % _archives_dir
+# set system libs:
+_prebuild_bazel_exports_syslibs = """export TENSORSTORE_SYSTEM_LIBS="%s"  && """ % ','.join(_systemlibs)
+# set output_user_root:
+_prebuild_bazel_exports_startup = "export TENSORSTORE_BAZEL_STARTUP_OPTIONS="
+_prebuild_bazel_exports_startup += "'--output_user_root %(builddir)s/cache' &&"
+# setup Bazel build options (pass CFLAGS, extra include paths and extra link paths for systemlibs:
+_prebuild_bazel_exports_buildopts = 'export TENSORSTORE_BAZEL_BUILD_OPTIONS="'
+_prebuild_bazel_exports_buildopts += '$(for i in  $CFLAGS;do echo --copt=$i; done) '  # pass CFLAGS to Bazel as copts
+_prebuild_bazel_exports_buildopts += '%s %s' % (_bzl_linkopts, _bzl_include_copts)
+_prebuild_bazel_exports_buildopts += '" &&'  # end prebuild_bazel_exports_buildopts
 
 
-local_ts_preinstall = local_ts_version + local_ts_useebbazel + local_ts_bzl_exp
+_preinstall_opts = _prebuild_gen_archivesbzl
+_preinstall_opts += _prebuild_bazel_exports_startup
+_preinstall_opts += _prebuild_bazel_exports_buildopts
+_preinstall_opts += _prebuild_bazel_exports_syslibs
 
 exts_list = [
     (name, version, {
         'installopts': '-v',
-        'preinstallopts': local_ts_preinstall,
-        'checksums': ['65cbe5a600c32569bb0b9f597ea318cc298a13b42d5fc98168c97bb11f320eae'],
+        'patches': ['tensorstore-0.1.69_disable_third_party_downloads.patch'],
+        'preinstallopts': _preinstall_opts,
+        'checksums': [
+            {'tensorstore-0.1.69.tar.gz': '150cdc7e2044b7629ea466bfc8425ab50e52c8300950dbdd4a64445a2d4fbab1'},
+            {'tensorstore-0.1.69_disable_third_party_downloads.patch':
+             'cd6ac01da452ae936ee1b64cf71174a42d6c1ed5bbd6bc6421dcd27fe636d374'},
+        ],
     }),
 ]

test checksum

downgrad dependency jax 0.4.35 -> 0.4.34

move checksums

add checksum for humanize

adding easyconfigs: dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb, tensor…

e280005

…store-0.1.69-foss-2024a.eb and patches: tensorstore-0.1.69_disable_third_party_downloads.patch

github-actions bot added the update label Dec 13, 2024

ThomasHoffmann77 mentioned this pull request Dec 13, 2024

{bio}[foss/2024a] AlphaFold3 WIP #21864

Open

Update tensorstore-0.1.69-foss-2024a.eb

2dd6ef2

test checksum

ThomasHoffmann77 mentioned this pull request Dec 19, 2024

{ai,bio}[foss/2024a] AlphaFold3 v3.0.0, jax-triton v0.2.0, jaxtyping v0.2.36, dm-tree v0.1.8 w/ CUDA 12.6.0 DRAFT! #22067

Draft

8 tasks

ThomasHoffmann77 added 2 commits December 20, 2024 14:32

Update dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb

d426a8f

downgrad dependency jax 0.4.35 -> 0.4.34

Update tensorstore-0.1.69-foss-2024a.eb

1a7968c

boegel mentioned this pull request Jan 7, 2025

AlphaFold vscentrum/vsc-software-stack#477

Open

2 tasks

ThomasHoffmann77 and others added 4 commits January 7, 2025 12:24

Update tensorstore-0.1.69-foss-2024a.eb

5e6da54

move checksums

fix style

7e7e2fc

Update dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb

4d5d70e

add checksum for humanize

add EC for Optax v0.2.4

b9be48a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{lib}[foss/2024a] dm-haiku v0.0.13, tensorstore v0.1.69 w/ CUDA 12.6.0 #22028

{lib}[foss/2024a] dm-haiku v0.0.13, tensorstore v0.1.69 w/ CUDA 12.6.0 #22028

ThomasHoffmann77 commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

{lib}[foss/2024a] dm-haiku v0.0.13, tensorstore v0.1.69 w/ CUDA 12.6.0 #22028

Are you sure you want to change the base?

{lib}[foss/2024a] dm-haiku v0.0.13, tensorstore v0.1.69 w/ CUDA 12.6.0 #22028

Conversation

ThomasHoffmann77 commented Dec 13, 2024 • edited Loading

github-actions bot commented Dec 13, 2024 • edited Loading

Updated software dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb

Updated software Optax-0.2.4-gfbf-2024a-CUDA-12.6.0.eb

Updated software tensorstore-0.1.69-foss-2024a.eb

ThomasHoffmann77 commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

Updated software `dm-haiku-0.0.13-foss-2024a-CUDA-12.6.0.eb`

Updated software `Optax-0.2.4-gfbf-2024a-CUDA-12.6.0.eb`

Updated software `tensorstore-0.1.69-foss-2024a.eb`