Skip to content

add a smoke test to ensure server images start correctly - #1646

Open
tmckayus wants to merge 2 commits into
NVIDIA:mainfrom
tmckayus:smokeimages
Open

add a smoke test to ensure server images start correctly#1646
tmckayus wants to merge 2 commits into
NVIDIA:mainfrom
tmckayus:smokeimages

Conversation

@tmckayus

Copy link
Copy Markdown
Contributor

This adds a simple smoke test after image builds to make sure the server instances start. The goal is to detect defects in the Dockerfile used to build the image (like faulty configuration of shared library paths, etc).

@tmckayus tmckayus added this to the 26.10 milestone Jul 31, 2026
@tmckayus
tmckayus requested a review from a team as a code owner July 31, 2026 16:49
@tmckayus
tmckayus requested a review from jakirkham July 31, 2026 16:49
@tmckayus tmckayus added non-breaking Introduces a non-breaking change improvement Improves an existing functionality labels Jul 31, 2026
@tmckayus
tmckayus requested a review from ramakrishnap-nv July 31, 2026 16:49
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f13d66c7-3fce-4a3a-bc94-8f02bb6c96d2

📥 Commits

Reviewing files that changed from the base of the PR and between 0cccfd3 and 6ccc937.

📒 Files selected for processing (3)
  • .github/workflows/test_images.yaml
  • ci/docker/README.md
  • ci/docker/smoke_image.sh
🚧 Files skipped from review as they are similar to previous changes (2)
  • .github/workflows/test_images.yaml
  • ci/docker/smoke_image.sh

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Walkthrough

Walkthrough

The pull request adds a Docker smoke-test script for REST and gRPC image startup. It integrates Ubuntu and conditional CUDA 13 UBI10 checks into the image workflow and documents the commands.

Changes

Image startup smoke testing

Layer / File(s) Summary
Smoke script execution
ci/docker/smoke_image.sh
Adds image acquisition with local fallback, REST and gRPC container startup checks, readiness polling, startup failure detection, log reporting, cleanup, and configurable timeout and GPU arguments.
CI workflow and documentation
.github/workflows/test_images.yaml, ci/docker/README.md
Runs Ubuntu REST/gRPC smoke tests and conditional CUDA 13 UBI10 tests. Documents the smoke-test commands and CI job.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 6ccc9

This change adds container startup smoke checks and related CI documentation for REST and gRPC images. No current merge-readiness risk has been identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 1 files. (2 skipped: 2 … Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the primary change: adding a smoke test to verify that server images start correctly.
Description check ✅ Passed The description explains that the smoke test runs after image builds and detects Dockerfile defects that prevent server instances from starting.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 1 files. (2 skipped: 2 unsupported.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ci/docker/README.md`:
- Line 42: Update the reference in the smoke job note to use the official
product name “GitHub” instead of “GITHUB,” without changing the referenced
workflow or job identifiers.

In `@ci/docker/smoke_image.sh`:
- Around line 40-79: Update smoke_one’s cleanup flow so every failure path
performs cleanup before terminating: handle a failed docker run, replace
fail/exit-based paths with nonzero returns followed by cleanup, or use an EXIT
trap that safely removes the container and temporary log. Preserve log output
and failure status for loader errors, early exits, and timeouts.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 521d87c9-d77d-49a0-8e5f-2bc77bf28770

📥 Commits

Reviewing files that changed from the base of the PR and between 2842c1b and 3b63567.

📒 Files selected for processing (3)
  • .github/workflows/test_images.yaml
  • ci/docker/README.md
  • ci/docker/smoke_image.sh

Comment thread ci/docker/README.md
Comment thread ci/docker/smoke_image.sh Outdated
@github-actions

github-actions Bot commented Jul 31, 2026

Copy link
Copy Markdown

CI Test Summary

⏭️ All 5 test job(s) skipped.

# CUOPT_SERVER_TYPE=grpc. The jobs above run *inside* the image as a GHA
# container and never launch the servers, so they cannot catch packaging
# gaps such as UBI10's RHEL lib/lib64 NCCL path miss.
smoke:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we add the test script as part of test_image.sh ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't put this in test_image.sh as wired today — that job runs inside the image as a GHA container, so it never exercises ENTRYPOINT/CMD, and it already patches LD_LIBRARY_PATH for the nvidia wheels (which is what masked the UBI10 NCCL packaging bug). The smoke test has to be host-side docker run.

The bug we hit was invoking the ubi10 image with just a docker command, as a user would.

@github-actions

Copy link
Copy Markdown

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

3 similar comments
@github-actions

Copy link
Copy Markdown

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

@github-actions

Copy link
Copy Markdown

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown

🔔 Hi @anandhkb, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

@copy-pr-bot

copy-pr-bot Bot commented Sep 3, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Sep 3, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@tmckayus

tmckayus commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 6ccc937

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Improves an existing functionality non-breaking Introduces a non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants