Skip to content

Support storage backend integration test suite #284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shanyl9 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Support storage backend integration test suite #284

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4e4d46f
added support for rapid env with storage be
shanyl9 Jan 14, 2026
d6c3cf0
tests.sh changes only for internal use
shanyl9 Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions .github/workflows/manual-integration-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,19 @@ on:
description: 'Extra Helm --set arguments (comma-separated, e.g. foo=bar,bar=baz)'
required: false
default: ''
use_storage_backend:
description: 'Use storage backend (Rapid7) instead of local storage'
required: false
type: boolean
default: false
customer_guid:
description: 'Customer GUID (Account ID) for storage backend'
required: false
default: ''
access_key:
description: 'Access key for storage backend'
required: false
default: ''
Comment on lines +26 to +38
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Sensitive credentials exposed as workflow dispatch inputs.

customer_guid and access_key are defined as plaintext workflow_dispatch inputs. These values will be visible in the GitHub Actions UI (workflow run parameters) and potentially in logs. Credentials should be stored as GitHub repository or environment secrets and referenced via ${{ secrets.BACKEND_ACCESS_KEY }}, not passed through the dispatch form.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/manual-integration-tests.yml around lines 26 - 38, The
workflow currently exposes sensitive values via workflow_dispatch inputs
(customer_guid and access_key); remove these plaintext inputs and instead read
credentials from GitHub Secrets (e.g., secrets.BACKEND_CUSTOMER_GUID and
secrets.BACKEND_ACCESS_KEY) in the workflow, keep only non-sensitive flags like
use_storage_backend, and update any steps that referenced customer_guid or
access_key to use the secrets via the secrets context (e.g., replace uses of the
customer_guid/access_key inputs with secrets.BACKEND_CUSTOMER_GUID and
secrets.BACKEND_ACCESS_KEY).


jobs:
integration-tests:
Expand Down Expand Up @@ -50,6 +63,7 @@ jobs:
version: v3.14.4

- name: Install Kubescape via Helm
if: ${{ github.event.inputs.use_storage_backend != 'true' }}
run: |
EXTRA_SET_ARGS=""
if [ -n "${{ github.event.inputs.extra_helm_set_args }}" ]; then
Expand Down Expand Up @@ -78,6 +92,36 @@ jobs:
--wait
popd

- name: Install Rapid7 release via Helm
if: ${{ github.event.inputs.use_storage_backend == 'true' }}
run: |
EXTRA_SET_ARGS=""
if [ -n "${{ github.event.inputs.extra_helm_set_args }}" ]; then
IFS=',' read -ra ADDR <<< "${{ github.event.inputs.extra_helm_set_args }}"
for arg in "${ADDR[@]}"; do
EXTRA_SET_ARGS="$EXTRA_SET_ARGS --set $arg"
done
fi
git clone https://github.com/kubescape/helm-charts.git
pushd helm-charts
git checkout ${{ github.event.inputs.branch_helm_chart }}
echo "Building Helm dependencies"
helm dependency build charts/kubescape-operator
echo "Installing Rapid7 release with Helm"
set -x
helm upgrade --install rapid7 charts/kubescape-operator \
-n kubescape --create-namespace \
--set nodeAgent.image.repository=$(echo "${{ github.event.inputs.node_agent_image }}" | cut -d: -f1) \
--set nodeAgent.image.tag=$(echo "${{ github.event.inputs.node_agent_image }}" | cut -d: -f2-) \
--set storage.image.repository=$(echo "${{ github.event.inputs.storage_image }}" | cut -d: -f1) \
--set storage.image.tag=$(echo "${{ github.event.inputs.storage_image }}" | cut -d: -f2-) \
--set nodeAgent.config.learningPeriod=30s \
--set nodeAgent.config.updatePeriod=1m \
--set storage.cleanupInterval=1m \
$EXTRA_SET_ARGS \
--wait
popd

- name: Verify all Pods are running
run: |
kubectl logs -n kubescape -l app=node-agent
Expand All @@ -86,7 +130,16 @@ jobs:
- name: Install gotestsum
run: go install gotest.tools/gotestsum@latest

- name: Set storage backend environment variables
if: ${{ github.event.inputs.use_storage_backend == 'true' }}
run: |
echo "BACKEND_ACCOUNT_ID=${{ github.event.inputs.customer_guid }}" >> $GITHUB_ENV
echo "BACKEND_ACCESS_KEY=${{ github.event.inputs.access_key }}" >> $GITHUB_ENV

- name: Run non-failover tests in parallel
env:
BACKEND_ACCOUNT_ID: ${{ github.event.inputs.use_storage_backend == 'true' && github.event.inputs.customer_guid || '' }}
BACKEND_ACCESS_KEY: ${{ github.event.inputs.use_storage_backend == 'true' && github.event.inputs.access_key || '' }}
run: |
cd tests/integration-test-suite
echo "Running non-failover tests"
Expand Down Expand Up @@ -173,6 +226,7 @@ jobs:
version: v3.14.4

- name: Install Kubescape via Helm
if: ${{ github.event.inputs.use_storage_backend != 'true' }}
run: |
EXTRA_SET_ARGS=""
if [ -n "${{ github.event.inputs.extra_helm_set_args }}" ]; then
Expand Down Expand Up @@ -201,6 +255,36 @@ jobs:
--wait
popd

- name: Install Rapid7 release via Helm
if: ${{ github.event.inputs.use_storage_backend == 'true' }}
run: |
EXTRA_SET_ARGS=""
if [ -n "${{ github.event.inputs.extra_helm_set_args }}" ]; then
IFS=',' read -ra ADDR <<< "${{ github.event.inputs.extra_helm_set_args }}"
for arg in "${ADDR[@]}"; do
EXTRA_SET_ARGS="$EXTRA_SET_ARGS --set $arg"
done
fi
git clone https://github.com/kubescape/helm-charts.git
pushd helm-charts
git checkout ${{ github.event.inputs.branch_helm_chart }}
echo "Building Helm dependencies"
helm dependency build charts/kubescape-operator
echo "Installing Rapid7 release with Helm"
set -x
helm upgrade --install rapid7 charts/kubescape-operator \
-n kubescape --create-namespace \
--set nodeAgent.image.repository=$(echo "${{ github.event.inputs.node_agent_image }}" | cut -d: -f1) \
--set nodeAgent.image.tag=$(echo "${{ github.event.inputs.node_agent_image }}" | cut -d: -f2-) \
--set storage.image.repository=$(echo "${{ github.event.inputs.storage_image }}" | cut -d: -f1) \
--set storage.image.tag=$(echo "${{ github.event.inputs.storage_image }}" | cut -d: -f2-) \
--set nodeAgent.config.learningPeriod=30s \
--set nodeAgent.config.updatePeriod=1m \
--set storage.cleanupInterval=1m \
$EXTRA_SET_ARGS \
--wait
popd

- name: Verify all Pods are running
run: |
kubectl logs -n kubescape -l app=node-agent
Expand All @@ -209,7 +293,16 @@ jobs:
- name: Install gotestsum
run: go install gotest.tools/gotestsum@latest

- name: Set storage backend environment variables
if: ${{ github.event.inputs.use_storage_backend == 'true' }}
run: |
echo "BACKEND_ACCOUNT_ID=${{ github.event.inputs.customer_guid }}" >> $GITHUB_ENV
echo "BACKEND_ACCESS_KEY=${{ github.event.inputs.access_key }}" >> $GITHUB_ENV

- name: Run failover test
env:
BACKEND_ACCOUNT_ID: ${{ github.event.inputs.use_storage_backend == 'true' && github.event.inputs.customer_guid || '' }}
BACKEND_ACCESS_KEY: ${{ github.event.inputs.use_storage_backend == 'true' && github.event.inputs.access_key || '' }}
run: |
cd tests/integration-test-suite
gotestsum --format standard-verbose --junitfile "junit-${{ matrix.test_name }}.xml" -- -count=1 -timeout 30m -v -run "IntegrationTestSuite/${{ matrix.test_name }}$" ./... -- --skip-ensure-helm --update-helm-if-present=false
Expand Down
14 changes: 9 additions & 5 deletions tests/integration-test-suite/case_crashloop_and_stable_profile_incomplete_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,10 @@ import (
"context"
"time"

"github.com/kubescape/storage/pkg/apis/softwarecomposition/v1beta1"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

containerwatcher "github.com/kubescape/node-agent/pkg/containerwatcher/v1"
)

func (s *IntegrationTestSuite) TestCrashLoopAndStableProfileIncomplete() {
Expand All @@ -31,8 +30,8 @@ Goal: Verify that a deployment with one crashlooping and one stable container do
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{
"app": "crashloop-stable-test-deployment",
containerwatcher.MaxSniffingTimeLabel: "2m",
"app": "crashloop-stable-test-deployment",
MaxSniffingTimeLabel: "2m",
},
},
Spec: corev1.PodSpec{
Expand Down Expand Up @@ -62,7 +61,12 @@ Goal: Verify that a deployment with one crashlooping and one stable container do
time.Sleep(3 * time.Minute)

s.LogWithTimestamp("Fetching application profile and verifying it is partial/completed")
applicationProfile, err := fetchApplicationProfile(s.ksObjectConnection, s.testNamespace, "deployment", "crashloop-stable-test-deployment")
var applicationProfile *v1beta1.ApplicationProfile
if s.isRapid7 {
applicationProfile, err = fetchApplicationProfileFromStorageBackend(s.testNamespace, "deployment", "crashloop-stable-test-deployment", s.accountID, s.accessKey)
} else {
applicationProfile, err = fetchApplicationProfileFromCluster(s.ksObjectConnection, s.testNamespace, "deployment", "crashloop-stable-test-deployment")
}
s.Require().NoError(err)
s.Require().NotNil(applicationProfile)
// The profile should be marked as partial/completed
Expand Down
14 changes: 9 additions & 5 deletions tests/integration-test-suite/case_crashloop_profile_incomplete_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,10 @@ import (
"context"
"time"

"github.com/kubescape/storage/pkg/apis/softwarecomposition/v1beta1"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

containerwatcher "github.com/kubescape/node-agent/pkg/containerwatcher/v1"
)

func (s *IntegrationTestSuite) TestCrashLoopProfileIncomplete() {
Expand All @@ -31,8 +30,8 @@ Goal: Verify that a crashlooping container does not result in a completed applic
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{
"app": "crashloop-test-deployment",
containerwatcher.MaxSniffingTimeLabel: "2m",
"app": "crashloop-test-deployment",
MaxSniffingTimeLabel: "2m",
},
},
Spec: corev1.PodSpec{
Expand All @@ -58,7 +57,12 @@ Goal: Verify that a crashlooping container does not result in a completed applic
time.Sleep(3 * time.Minute)

s.LogWithTimestamp("Fetching application profile and verifying it is partial/completed")
applicationProfile, err := fetchApplicationProfile(s.ksObjectConnection, s.testNamespace, "deployment", "crashloop-test-deployment")
var applicationProfile *v1beta1.ApplicationProfile
if s.isRapid7 {
applicationProfile, err = fetchApplicationProfileFromStorageBackend(s.testNamespace, "deployment", "crashloop-test-deployment", s.accountID, s.accessKey)
} else {
applicationProfile, err = fetchApplicationProfileFromCluster(s.ksObjectConnection, s.testNamespace, "deployment", "crashloop-test-deployment")
}
s.Require().NoError(err)
s.Require().NotNil(applicationProfile)
// The profile should be marked as partial/completed
Expand Down
10 changes: 4 additions & 6 deletions tests/integration-test-suite/case_cronjob_profile_create_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ import (
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

containerwatcher "github.com/kubescape/node-agent/pkg/containerwatcher/v1"
)

func (s *IntegrationTestSuite) TestCronJobProfileCreate() {
Expand All @@ -28,8 +26,8 @@ func (s *IntegrationTestSuite) TestCronJobProfileCreate() {
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{
"app": "test-cronjob-profile",
containerwatcher.MaxSniffingTimeLabel: "2m",
"app": "test-cronjob-profile",
MaxSniffingTimeLabel: "2m",
},
},
Spec: corev1.PodSpec{
Expand Down Expand Up @@ -57,8 +55,8 @@ func (s *IntegrationTestSuite) TestCronJobProfileCreate() {
time.Sleep(3 * time.Minute)

s.LogWithTimestamp("Verifying profiles are complete")
verifyApplicationProfileCompleted(s.T(), s.ksObjectConnection, "complete", s.testNamespace, "cronjob", "test-cronjob-profile")
verifyNetworkNeighborProfileCompleted(s.T(), s.ksObjectConnection, false, false, "complete", s.testNamespace, "cronjob", "test-cronjob-profile")
verifyApplicationProfileCompleted(s.T(), s.ksObjectConnection, "complete", s.testNamespace, "cronjob", "test-cronjob-profile", s.accountID, s.accessKey, s.isRapid7)
verifyNetworkNeighborProfileCompleted(s.T(), s.ksObjectConnection, false, false, "complete", s.testNamespace, "cronjob", "test-cronjob-profile", s.accountID, s.accessKey, s.isRapid7)

s.LogWithTimestamp("TestCronJobProfileCreate completed successfully")
}
29 changes: 17 additions & 12 deletions tests/integration-test-suite/case_init_container_profile_create_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,10 @@ import (
"context"
"time"

"github.com/kubescape/storage/pkg/apis/softwarecomposition/v1beta1"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

containerwatcher "github.com/kubescape/node-agent/pkg/containerwatcher/v1"
)

func (s *IntegrationTestSuite) TestInitContainerProfileCreate() {
Expand All @@ -32,8 +31,8 @@ The profile should be complete after the main container's learning period finish
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{
"app": "init-container-test-deployment",
containerwatcher.MaxSniffingTimeLabel: "2m",
"app": "init-container-test-deployment",
MaxSniffingTimeLabel: "5m",
},
},
Spec: corev1.PodSpec{
Expand All @@ -60,12 +59,18 @@ The profile should be complete after the main container's learning period finish
s.LogWithTimestamp("Waiting for pod to be ready (init container should be running)")
WaitForPodWithLabelReady(s.T(), s.clientset, s.testNamespace, "app=init-container-test-deployment")

s.LogWithTimestamp("Waiting 90 seconds for init container to complete in storage (main container should be running)")
time.Sleep(90 * time.Second)

// Verify that profile exists but is not complete yet
s.LogWithTimestamp("Verifying profile exists but is not complete after init container")
applicationProfile, err := fetchApplicationProfile(s.ksObjectConnection, s.testNamespace, "deployment", "init-container-test-deployment")
var applicationProfile *v1beta1.ApplicationProfile
if s.isRapid7 {
s.LogWithTimestamp("Waiting 3 minutes for init container to complete in storage backend (main container should be running)")
time.Sleep(3 * time.Minute)
applicationProfile, err = fetchApplicationProfileFromStorageBackend(s.testNamespace, "deployment", "init-container-test-deployment", s.accountID, s.accessKey)
} else {
s.LogWithTimestamp("Waiting 90 seconds for init container to complete in cluster (main container should be running)")
time.Sleep(90 * time.Second)
applicationProfile, err = fetchApplicationProfileFromCluster(s.ksObjectConnection, s.testNamespace, "deployment", "init-container-test-deployment")
}
s.Require().NoError(err)
s.Require().NotNil(applicationProfile)
s.Require().Equal("complete", applicationProfile.Annotations["kubescape.io/completion"])
Expand All @@ -84,12 +89,12 @@ The profile should be complete after the main container's learning period finish
}
s.Require().True(initContainerFound, "Init container should be present in the profile")

s.LogWithTimestamp("Waiting 3 minutes for main container learning period")
time.Sleep(3 * time.Minute)
s.LogWithTimestamp("Waiting 4 minutes for main container learning period")
time.Sleep(4 * time.Minute)

s.LogWithTimestamp("Verifying profiles are complete")
verifyApplicationProfileCompleted(s.T(), s.ksObjectConnection, "complete", s.testNamespace, "deployment", "init-container-test-deployment")
verifyNetworkNeighborProfileCompleted(s.T(), s.ksObjectConnection, false, false, "complete", s.testNamespace, "deployment", "init-container-test-deployment")
verifyApplicationProfileCompleted(s.T(), s.ksObjectConnection, "complete", s.testNamespace, "deployment", "init-container-test-deployment", s.accountID, s.accessKey, s.isRapid7)
verifyNetworkNeighborProfileCompleted(s.T(), s.ksObjectConnection, false, false, "complete", s.testNamespace, "deployment", "init-container-test-deployment", s.accountID, s.accessKey, s.isRapid7)

s.LogWithTimestamp("TestInitContainerProfileCreate completed successfully")
}
20 changes: 13 additions & 7 deletions tests/integration-test-suite/case_init_sidecar_profile_create_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,10 @@ import (
"context"
"time"

"github.com/kubescape/storage/pkg/apis/softwarecomposition/v1beta1"
appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

containerwatcher "github.com/kubescape/node-agent/pkg/containerwatcher/v1"
)

func (s *IntegrationTestSuite) TestInitSidecarProfileCreate() {
Expand All @@ -34,8 +33,8 @@ The profile should be complete after the learning period finishes and include bo
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{
"app": "init-sidecar-test-deployment",
containerwatcher.MaxSniffingTimeLabel: "2m",
"app": "init-sidecar-test-deployment",
MaxSniffingTimeLabel: "5m",
},
},
Spec: corev1.PodSpec{
Expand Down Expand Up @@ -74,7 +73,14 @@ The profile should be complete after the learning period finishes and include bo

// Verify that profile exists and is in ready state
s.LogWithTimestamp("Verifying profile exists and is in ready state")
applicationProfile, err := fetchApplicationProfile(s.ksObjectConnection, s.testNamespace, "deployment", "init-sidecar-test-deployment")
var applicationProfile *v1beta1.ApplicationProfile
if s.isRapid7 {
s.LogWithTimestamp("Waiting 3 minutes for sidecar container to complete in storage backend while main container keeps running")
time.Sleep(3 * time.Minute)
applicationProfile, err = fetchApplicationProfileFromStorageBackend(s.testNamespace, "deployment", "init-sidecar-test-deployment", s.accountID, s.accessKey)
} else {
applicationProfile, err = fetchApplicationProfileFromCluster(s.ksObjectConnection, s.testNamespace, "deployment", "init-sidecar-test-deployment")
}
s.Require().NoError(err)
s.Require().NotNil(applicationProfile)
s.Require().Equal("complete", applicationProfile.Annotations["kubescape.io/completion"])
Expand All @@ -101,8 +107,8 @@ The profile should be complete after the learning period finishes and include bo
time.Sleep(3 * time.Minute)

s.LogWithTimestamp("Verifying profiles are complete")
verifyApplicationProfileCompleted(s.T(), s.ksObjectConnection, "complete", s.testNamespace, "deployment", "init-sidecar-test-deployment")
verifyNetworkNeighborProfileCompleted(s.T(), s.ksObjectConnection, false, false, "complete", s.testNamespace, "deployment", "init-sidecar-test-deployment")
verifyApplicationProfileCompleted(s.T(), s.ksObjectConnection, "complete", s.testNamespace, "deployment", "init-sidecar-test-deployment", s.accountID, s.accessKey, s.isRapid7)
verifyNetworkNeighborProfileCompleted(s.T(), s.ksObjectConnection, false, false, "complete", s.testNamespace, "deployment", "init-sidecar-test-deployment", s.accountID, s.accessKey, s.isRapid7)

s.LogWithTimestamp("TestInitSidecarProfileCreate completed successfully")
}
10 changes: 4 additions & 6 deletions tests/integration-test-suite/case_job_profile_create_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ import (
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

containerwatcher "github.com/kubescape/node-agent/pkg/containerwatcher/v1"
)

func (s *IntegrationTestSuite) TestJobProfileCreate() {
Expand All @@ -24,8 +22,8 @@ func (s *IntegrationTestSuite) TestJobProfileCreate() {
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: map[string]string{
"app": "test-job-profile",
containerwatcher.MaxSniffingTimeLabel: "2m",
"app": "test-job-profile",
MaxSniffingTimeLabel: "2m",
},
},
Spec: corev1.PodSpec{
Expand All @@ -51,8 +49,8 @@ func (s *IntegrationTestSuite) TestJobProfileCreate() {
time.Sleep(3 * time.Minute)

s.LogWithTimestamp("Verifying profiles are complete")
verifyApplicationProfileCompleted(s.T(), s.ksObjectConnection, "complete", s.testNamespace, "job", "test-job-profile")
verifyNetworkNeighborProfileCompleted(s.T(), s.ksObjectConnection, false, false, "complete", s.testNamespace, "job", "test-job-profile")
verifyApplicationProfileCompleted(s.T(), s.ksObjectConnection, "complete", s.testNamespace, "job", "test-job-profile", s.accountID, s.accessKey, s.isRapid7)
verifyNetworkNeighborProfileCompleted(s.T(), s.ksObjectConnection, false, false, "complete", s.testNamespace, "job", "test-job-profile", s.accountID, s.accessKey, s.isRapid7)

s.LogWithTimestamp("TestJobProfileCreate completed successfully")
}
Loading
Loading