chore: support allocating testnets to the local DC by basvandijk · Pull Request #10122 · dfinity/ic

basvandijk · 2026-05-07T13:35:35Z

What

Support optionally allocating Farm testnets to the same DC as were the GitHub runner is running.

Why

The nested system-tests are quite flaky:

$ bazel run //ci/githubstats:query -- top 100 flaky --gt 0 --columns=label,total,flaky,flaky% --include=%nested%
...
┍━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━┑
│    │ label                                                       │   total │   flaky │   flaky% │
┝━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━┿━━━━━━━━━┿━━━━━━━━━━┥
│  0 │ //rs/tests/nested/nns_recovery:nr_all_broken_seq_np_actions │      37 │       3 │      8.1 │
│  1 │ //rs/tests/nested:hostos_upgrade_smoke_test                 │      45 │       2 │      4.4 │
│  2 │ //rs/tests/nested/nns_recovery:nr_broken_dfinity_node       │      37 │       1 │      2.7 │
│  3 │ //rs/tests/nested:hostos_upgrade_smoke_test_head_nns        │      37 │       1 │      2.7 │
│  4 │ //rs/tests/nested:registration                              │      45 │       1 │      2.2 │
┕━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━┙

It's because these are the only tests that use the SetupOS disk images which are very large (2.6G). Downloading these images on Farm hosts often times out. Especially if the transfer has to cross the Atlantic, i.e. when the Farm host is in zh1 and the image was built in dm1 or vice versa.

We should avoid these cross-DC transfers. One way of doing that is forcing a testnet to be allocated to the same DC as were the image was created which is the DC where the the GitHub runner is running.

How

This introduces the new function on SystemTestGroup: .allocate_testnet_to_local_dc(). When set the Farm group is created with required_host_features set to the DC of the GitHub runner.
The DC of the GitHub runner is determined via the NODE_NAME environment variable. This will have a value on CI like dm1-spm34 where the part before the - denotes the DC.
The DC is outputted as a bazel volatile status variable in bazel/workspace_status.sh. It has to be volatile because we don't want a different node to invalidate a previously cached test.
A new //bazel:volatile-status.txt target is introduced which declares the volatile status file as the rule's default output.
system_tests will use //bazel:volatile-status.txt as a runtime dependency.

Farm metadata

The FARM_METADATA used to be a stable variable meaning that a change in user or CI job would invalidate a previously cached system-test. We should work towards having the property that when CI has ran a system-test running the same test locally on the same commit should cause the test to be fetched from the cache instead of being run again. For this reason the FARM_METADATA has also been made volatile.

basvandijk · 2026-05-07T18:09:00Z

Before merging this we need to add a new feature to Farm. The problem is that when the bazel-test-all job is run from dm1 (which is often the case since we have more runners there than in zh1) we quickly allocate all available resources on the dm1 Farm hosts. This is because these Farm hosts were originally meant for performance tests and hence only allow a maximum of 64 vCPUs per host while regular hosts allow 256 vCPUs.

We can't simply increase the 64 to 256 since that will cause performance tests to allocate multiple VMs per host which makes their measurements / benchmarks unreliable.

What I think we need is a new VmAllocationMode that is specifically designed for performance tests. Let's call it PerformanceAllocation for now. Then we would increase the max vCPUs to 256 but then the semantics of PerformanceAllocation is that it would behave like the default MinIntraDistanceLoadBalanceAllocation but remove its first ordering property:

the number of VMs of the group that the host has already allocated in
descending order (VMs of a testnet are grouped together on a host as
much as possible).

This will cause VMs of performance tests to be load-balanced over the dm1 hosts instead of being colocated together on a single host before spilling over to other hosts.

…ts.bzl and not in bazel/BUILD.bazel anymore

nmattia · 2026-05-11T11:48:21Z

 )

+def _write_version_file_var_impl(ctx):
+    """Helper rule that creates a file with the content of the provided var from the version file."""


Can you add a sentence explaining what we empirically discovered about when those targets will get rebuilt? Namely (IIRC) if the target (modulo volatile status content) does not exist then it will be built; otherwise the existing one will be used.

Writing this now actually I wonder if there are security implications (since you'll be blindly accepting cached artifacts from a different build) but I guess we anyway assume the cache is not malicious.

nmattia · 2026-05-11T11:48:49Z

 echo "STABLE_FARM_METADATA $STABLE_FARM_METADATA"
+
+# Used for allocating a Farm testnet to the local DC in CI (Search for allocate_testnet_to_local_dc)
+NODE_NAME="${NODE_NAME:-}"


Suggested change

NODE_NAME="${NODE_NAME:-}"

NODE_NAME="${NODE_NAME:-unknown}"

nmattia · 2026-05-11T11:49:55Z

    icos_images |= icos_config.icos_images

    env_var_files["FARM_METADATA"] = "//rs/tests:farm_metadata.txt"
+    env_var_files["DC"] = "//rs/tests:DC.txt"


is this DC or NODE_NAME? I thought it'd read what's set by workspace_status

This is outdated.

…stem_tests.bzl and not in bazel/BUILD.bazel anymore" This reverts commit 57abc75.

This reverts commit a896b76.

This reverts commit 4e15a93.

This reverts commit a415d64.

nmattia · 2026-05-12T14:03:40Z

    visibility = ["//visibility:public"],
 )

+# This is the contents of the ctx.version_file, i.e. all non-STABLE_,


I think this should be compacted a bit, it's a lot to read. I think it's enough to point out that volatile-status.txt has to be a direct dependency, and when it is so then it behaves somewhat magically.

nmattia · 2026-05-12T14:09:07Z

+    # VOLATILE_STATUS_FILE is read during group creation time to determine the required host features and Farm metadata.
+    # In colocated tests the group is created in the non-colocated wrapper driver, we shouldn't use runtime_deps
+    # to depend on the VOLATILE_STATUS_FILE since that will cause the variable to be repointed to a path on the colocated VM
+    # which won't exist during group creaton time on the non-colocated wrapper. Instead we pass it as a regular environment variable


Can you prefix it with RUN_SCRIPT_ and give it the same treatment as the other files read by the wrapper?

nmattia · 2026-05-12T14:09:46Z


 exec \
    env -C "$TEST_TMPDIR" \
+    VOLATILE_STATUS_FILE="$(realpath "$VOLATILE_STATUS_FILE")" \


At this point could we already read out the farm metadata and only set that, instead of silently passing the whole volatile status? Same with DC. It keeps everything tight and the inputs declarative.

Sure, I guess we could do the parsing in bash rather than in Rust.

nmattia · 2026-05-12T14:13:46Z

+# BUILD_TIMESTAMP 1778503130
+# DC zh1
+# FORMATTED_DATE 2026 May 11 12 38 50 Mon
+volatile_status(


I think we should keep write_info_file_var and this consistent: either we use volatile/non-volatile for both, or we use info_file/version_file for both

I'm not sure what you mean here. WDYM with "both"? Note that we use the volatile status for both the DC and the FARM_METADATA.

nmattia · 2026-05-12T14:15:37Z

+# Used for allocating a Farm testnet to the local DC in CI (Search for allocate_testnet_to_local_dc)
+NODE_NAME="${NODE_NAME:-unknown}"
+echo "DC ${NODE_NAME%%-*}"


This assumes that the node name is always prefixed with <DC>-, correct? worth adding a note. Even better: I'd export NODE_NAME and do the transformation for the one file that needs the DC

Actually, the assumption breaks for unknown though I think then DC will happen to be unknown

Yeah I can add a node that the DC will be everything up to the first -.

chore: support allocating testnets to the local DC

e9c2625

github-actions Bot added the chore label May 7, 2026

IDX GitHub Automation and others added 4 commits May 7, 2026 13:41

Automatically fixing code for linting and formatting issues

843e0c9

fix clippy

9a878ba

Use .allocate_testnet_to_local_dc() on all nested tests

a4157cf

fix clippy

47e971b

basvandijk marked this pull request as ready for review May 7, 2026 18:09

basvandijk requested review from a team as code owners May 7, 2026 18:09

github-actions Bot added @idx @node @core-protocol labels May 7, 2026

Bownairo approved these changes May 7, 2026

View reviewed changes

pierugo-dfinity approved these changes May 8, 2026

View reviewed changes

basvandijk and others added 4 commits May 8, 2026 09:10

just pass DC to volatile

b71a39f

Automatically fixing code for linting and formatting issues

fb2094a

fix

3a63084

Use write_version_file_var

a415d64

basvandijk added the CI_ALL_BAZEL_TARGETS Runs all bazel targets label May 8, 2026

basvandijk added 3 commits May 8, 2026 12:58

doc

4e15a93

refactor

a896b76

drop visibility for DC.txt, as it is only used in rs/tests/system_tes…

57abc75

…ts.bzl and not in bazel/BUILD.bazel anymore

nmattia reviewed May 11, 2026

View reviewed changes

basvandijk added 5 commits May 11, 2026 12:31

Revert "drop visibility for DC.txt, as it is only used in rs/tests/sy…

2751444

…stem_tests.bzl and not in bazel/BUILD.bazel anymore" This reverts commit 57abc75.

Revert "refactor"

afe5159

This reverts commit a896b76.

Revert "doc"

65896d0

This reverts commit 4e15a93.

Revert "Use write_version_file_var"

700dedd

This reverts commit a415d64.

doc

22dc1c3

basvandijk added 2 commits May 11, 2026 13:22

Use DC unknown by default

f4ce2ab

Make Farm metadata volatile

4fbfba9

basvandijk mentioned this pull request May 12, 2026

chore: move FARM_METADATA to volatile status file #10136

Closed

Merge branch 'master' into basvandijk/allocate_testnet_to_local_dc

4f02028

nmattia reviewed May 12, 2026

View reviewed changes

Conversation

basvandijk commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

How

Farm metadata

Uh oh!

basvandijk commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

basvandijk commented May 7, 2026 •

edited

Loading

basvandijk commented May 7, 2026 •

edited

Loading