Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/guides/remote-deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ Set these on the local machine. The upload variables are read here, since
| `ISV_CLIENT_SECRET` | Required for result upload to ISV Lab Service | locally |
| `NGC_API_KEY` | Required for NIM model benchmarks | forwarded |
| `ISVTEST_INCLUDE_UNRELEASED` | Include checks not yet in `released_tests.json` | forwarded |
| `ISVTEST_BREAKFIX_ALLOW_MUTATION` | Explicitly allow a mutating breakfix validation | forwarded |
| `ISVTEST_BREAKFIX_NODE` | Dedicated Kubernetes node selected for breakfix validation | forwarded |

Anything else the tests need has to reach the target another way - a config file
under `isvctl/` travels in the deployment archive, so `-f` overrides are the
Expand Down Expand Up @@ -85,6 +87,23 @@ Pass extra pytest arguments after `--`:
uv run isvctl deploy run <target-ip> -f isvctl/configs/suites/slurm.yaml -- -v -s -k "test_name"
```

### Running the Kubernetes Cordon Breakfix Check

The cordon reference requires explicit mutation consent and an exact dedicated
node on multi-node clusters. It restores the node's schedulability before the
run exits.

```bash
ISVTEST_INCLUDE_UNRELEASED=1 \
ISVTEST_BREAKFIX_ALLOW_MUTATION=1 \
ISVTEST_BREAKFIX_NODE=<dedicated-test-node> \
uv run isvctl deploy run <target-ip> \
-j <jumphost> -u ubuntu \
-f isvctl/configs/providers/kubernetes-breakfix.yaml \
--phase test --no-upload \
-- -v -s -k CordonNodeCheck
```

### With ISV Lab Service Integration

Upload results to the ISV Lab Service:
Expand Down
37 changes: 37 additions & 0 deletions isvctl/configs/providers/kubernetes-breakfix.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

# Explicitly mutating Kubernetes BFX01-04 reference configuration.
#
# This uses the current kubectl context and deliberately has no cluster setup
# or teardown lifecycle. A single-node Minikube context can be selected
# automatically. Multi-node contexts, including DSX, require an explicit node:
#
# export ISVTEST_INCLUDE_UNRELEASED=1
# export ISVTEST_BREAKFIX_ALLOW_MUTATION=1
# export ISVTEST_BREAKFIX_NODE=<dedicated-test-node>
# isvctl test run -f isvctl/configs/providers/kubernetes-breakfix.yaml \
# --label breakfix -- -v -s -k CordonNodeCheck

import: ../suites/k8s.yaml

version: "1.0"

commands:
kubernetes:
phases: ["test"]
steps:
- name: cordon_node
phase: test
command: "python shared/breakfix/cordon_node.py"
args:
- "--node={{ env.ISVTEST_BREAKFIX_NODE | default('', true) }}"
timeout: 1200
requires_available_validations:
- CordonNodeCheck

tests:
description: "Explicit Kubernetes BFX01-04 cordon validation"

settings:
show_skipped_tests: true
Loading