THREESCALE-10236 Custom CA certificate support for operator deployment by borisurbanik · Pull Request #1181 · 3scale/3scale-operator

borisurbanik · 2026-06-25T22:25:13Z

https://redhat.atlassian.net/browse/THREESCALE-10236

The 3scale operator connects to the 3scale Admin API (and, via the Tenant controller, the Master API) to reconcile capabilities CRs (Backend, Product, Application, ActiveDoc, etc.). These connections use HTTPS.

In environments where 3scale is deployed with internal or self-signed CA, the operator's HTTP client rejects the connection because the CA is not in the system trust store. The existing workarounds — the insecure_skip_verify annotation and patching the operator Subscription to mount a CA via SSL_CERT_FILE (Red Hat Solution 7049968) — are either insecure or fragile (overwritten on upgrades).

Customers need a way to provide a custom CA bundle so the operator can trust internal CAs without disabling verification.

Sources of 3scale connections (each with different credential configuration):

Explicit providerAccountRef: the CR references a Secret containing adminURL and token.
Default threescale-provider-account secret: a well-known Secret name. Same schema as source 1 but discovered by convention.
Local APIManager: the operator discovers an APIManager CR in the namespace and derives the admin URL and token from system-seed.
Tenant controller / Master API: the Tenant CR references a masterCredentialsRef secret (defaulting to system-seed) containing master API credentials.

This PR adds ability to configure a custom CA bundle when connecting to any of the above endpoints. The bundle will be loaded from a well-known configmap from the operator namespace (threescale-ca-bundle) and will be reloaded automatically if a change is detected.

Testing:

The testing follows 3 scenarios when using a standard deployment of APIManager on creating a backend resource without secret -> the backend reconciler will fall back to endpoint for local APIManager.

make sure the request to 3scale admin URL is failing initially:

oc apply -n $NAMESPACE -f - <<EOF
apiVersion: capabilities.3scale.net/v1beta1
kind: Backend
metadata:
  name: test-backend-noconfig
spec:
  name: "Test Backend No Config"
  systemName: "test-backend-noconfig"
  privateBaseURL: "https://httpbin.org"
EOF

# Wait a moment for reconcile, then check status
oc get backend test-backend-noconfig -n $NAMESPACE -o jsonpath='{.status.conditions}' | jq

# Check for the Warning event
oc get events -n $NAMESPACE --field-selector reason=ReconcileError,involvedObject.name=test-backend-noconfig

Result:

borisurbanik@MacBookPro rook_ceph_deploy % oc get backend test-backend-noconfig -n $NAMESPACE -o jsonpath='{.status.conditions}' | jq
[
  {
    "lastTransitionTime": "2026-06-30T22:28:51Z",
    "message": "Get \"https://3scale-admin.apps.burbanik-3scale.cp.fyre.ibm.com:443/admin/api/backend_apis.json?page=1&per_page=500\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
    "status": "True",
    "type": "Failed"
  },
  {
    "lastTransitionTime": "2026-06-30T22:28:51Z",
    "status": "False",
    "type": "Invalid"
  },
  {
    "lastTransitionTime": "2026-06-30T22:28:51Z",
    "status": "False",
    "type": "Synced"
  }
]
borisurbanik@MacBookPro rook_ceph_deploy % oc get events -n $NAMESPACE --field-selector reason=ReconcileError,involvedObject.name=test-backend-noconfig
LAST SEEN   TYPE      REASON           OBJECT                          MESSAGE
5s          Warning   ReconcileError   backend/test-backend-noconfig   Get "https://3scale-admin.apps.burbanik-3scale.cp.fyre.ibm.com:443/admin/api/backend_apis.json?page=1&per_page=500": tls: failed to verify certificate: x509: certificate signed by unknown authority

Check that the annotation insecure_skip_verify would bypass the problem:

oc apply -n $NAMESPACE -f - <<EOF
apiVersion: capabilities.3scale.net/v1beta1
kind: Backend
metadata:
  name: test-backend-insecure
  annotations:
    insecure_skip_verify: "true"
spec:
  name: "Test Backend Insecure"
  systemName: "test-backend-insecure"
  privateBaseURL: "https://httpbin.org"
EOF

oc wait --for=condition=Synced --timeout=60s backend/test-backend-insecure -n $NAMESPACE
oc get backend test-backend-insecure -n $NAMESPACE -o jsonpath='{.status.conditions}' | jq

Result:

borisurbanik@MacBookPro rook_ceph_deploy % oc wait --for=condition=Synced --timeout=60s backend/test-backend-insecure -n $NAMESPACE
backend.capabilities.3scale.net/test-backend-insecure condition met
borisurbanik@MacBookPro rook_ceph_deploy % oc get backend test-backend-insecure -n $NAMESPACE -o jsonpath='{.status.conditions}' | jq
[
  {
    "lastTransitionTime": "2026-06-30T22:31:26Z",
    "status": "False",
    "type": "Failed"
  },
  {
    "lastTransitionTime": "2026-06-30T22:31:26Z",
    "status": "False",
    "type": "Invalid"
  },
  {
    "lastTransitionTime": "2026-06-30T22:31:26Z",
    "status": "True",
    "type": "Synced"
  }
]

Check that custom CA will work as well

# Extract the ingress CA and create the ConfigMap
oc get secret router-ca -n openshift-ingress-operator \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/ingress-ca.crt

oc create configmap threescale-ca-bundle \
  --from-file=ca-bundle.crt=/tmp/ingress-ca.crt \
  -n $NAMESPACE

# Wait for the CABundleWatcher to reconcile (it's non-leader-election so picks up immediately)
sleep 3

oc apply -n $NAMESPACE -f - <<EOF
apiVersion: capabilities.3scale.net/v1beta1
kind: Backend
metadata:
  name: test-backend-ca
spec:
  name: "Test Backend With CA"
  systemName: "test-backend-ca"
  privateBaseURL: "https://httpbin.org"
EOF

oc wait --for=condition=Synced --timeout=60s backend/test-backend-ca -n $NAMESPACE
oc get backend test-backend-ca -n $NAMESPACE -o jsonpath='{.status.conditions}' | jq

Result:

borisurbanik@MacBookPro rook_ceph_deploy % oc wait --for=condition=Synced --timeout=60s backend/test-backend-ca -n $NAMESPACE
backend.capabilities.3scale.net/test-backend-ca condition met
borisurbanik@MacBookPro rook_ceph_deploy % oc get backend test-backend-ca -n $NAMESPACE -o jsonpath='{.status.conditions}' | jq
[
  {
    "lastTransitionTime": "2026-06-30T22:34:32Z",
    "status": "False",
    "type": "Failed"
  },
  {
    "lastTransitionTime": "2026-06-30T22:34:32Z",
    "status": "False",
    "type": "Invalid"
  },
  {
    "lastTransitionTime": "2026-06-30T22:34:32Z",
    "status": "True",
    "type": "Synced"
  }
]

Test that invalid bundle will surface errors on the config map itself

oc apply -n $NAMESPACE -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: threescale-ca-bundle
data:
  ca-bundle.crt: "this is not a valid PEM certificate"
EOF

sleep 3

oc get events -n $NAMESPACE \
  --field-selector reason=InvalidCABundle,involvedObject.name=threescale-ca-bundle

Result:

borisurbanik@MacBookPro rook_ceph_deploy % oc get events -n $NAMESPACE \
  --field-selector reason=InvalidCABundle,involvedObject.name=threescale-ca-bundle
LAST SEEN   TYPE      REASON            OBJECT                           MESSAGE
17s         Warning   InvalidCABundle   configmap/threescale-ca-bundle   InvalidCAFormat: No valid PEM-encoded certificates found in CA bundle

Test bring-your-own-ca + Test bundle with multiple CAs

echo "=== Step 1: Generate self-signed CA ==="
openssl genrsa -out /tmp/tc2-ca.key 2048
openssl req -x509 -new -nodes -key /tmp/tc2-ca.key -sha256 -days 365 \
  -out /tmp/tc2-ca.crt -subj "/CN=TestCA-TC2"

echo ""
echo "=== Step 1b: Generate server cert signed by TestCA-TC2 and patch route ==="
openssl genrsa -out /tmp/tc2-server.key 2048
openssl req -new -key /tmp/tc2-server.key \
  -out /tmp/tc2-server.csr \
  -subj "/CN=${ROUTE_HOST}" \
  -addext "subjectAltName=DNS:${ROUTE_HOST}"
openssl x509 -req -in /tmp/tc2-server.csr \
  -CA /tmp/tc2-ca.crt -CAkey /tmp/tc2-ca.key -CAcreateserial \
  -out /tmp/tc2-server.crt -days 365 -sha256 \
  -extfile <(echo "subjectAltName=DNS:${ROUTE_HOST}")
openssl verify -CAfile /tmp/tc2-ca.crt /tmp/tc2-server.crt

oc patch route "${ROUTE_NAME}" -n "${APIMANAGER_NS}" \
  --type=merge -p "{
    \"spec\": {
      \"tls\": {
        \"termination\": \"edge\",
        \"insecureEdgeTerminationPolicy\": \"Redirect\",
        \"certificate\": $(cat /tmp/tc2-server.crt | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))'),
        \"key\": $(cat /tmp/tc2-server.key | python3 -c 'import sys,json; print(json.dumps(sys.stdin.read()))')
      }
    }
  }"

echo "=== Step 2: Concatenate second CA and update ConfigMap in-place ==="
openssl genrsa -out /tmp/tc2-ca2.key 2048
openssl req -x509 -new -nodes -key /tmp/tc2-ca2.key -sha256 -days 365 \
  -out /tmp/tc2-ca2.crt -subj "/CN=TestCA-TC2-Second"
cat /tmp/tc2-ca.crt /tmp/tc2-ca2.crt > /tmp/tc2-bundle.crt
oc create configmap threescale-ca-bundle \
  --from-file=ca-bundle.crt=/tmp/tc2-bundle.crt -n "${OPERATOR_NS}" \
  --dry-run=client -o yaml | oc apply -f -

echo "=== Step 3: Verify no warnings after update ==="
sleep 10
echo "--- Certificate blocks in ConfigMap ---"
oc get configmap threescale-ca-bundle \
  -o jsonpath='{.data.ca-bundle\.crt}' -n "${OPERATOR_NS}" | grep -c "BEGIN CERTIFICATE"
echo "--- InvalidCABundle events ---"
oc get events -n "${OPERATOR_NS}" --field-selector reason=InvalidCABundle --sort-by='.lastTimestamp'

echo ""
echo "=== Step 4: Trigger reconcile and verify Synced=True ==="
oc annotate backend "${BACKEND_NAME}" -n "${APIMANAGER_NS}" \
  test-reconcile-trigger="$(date -u +%Y%m%dT%H%M%SZ)" --overwrite
sleep 15
echo "--- Backend conditions ---"
oc get backend "${BACKEND_NAME}" -n "${APIMANAGER_NS}" \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}'

Result:

=== Step 3: Verify no warnings after update ===

--- Certificate blocks in ConfigMap ---
2
--- InvalidCABundle events ---
LAST SEEN   TYPE      REASON            OBJECT                           MESSAGE
84m         Warning   InvalidCABundle   configmap/threescale-ca-bundle   InvalidCAFormat: No valid PEM-encoded certificates found in CA bundle
76m         Warning   InvalidCABundle   configmap/threescale-ca-bundle   InvalidCAFormat: No valid PEM-encoded certificates found in CA bundle
34m         Warning   InvalidCABundle   configmap/threescale-ca-bundle   InvalidCAFormat: No valid PEM-encoded certificates found in CA bundle
32m         Warning   InvalidCABundle   configmap/threescale-ca-bundle   InvalidCAFormat: No valid PEM-encoded certificates found in CA bundle
27m         Warning   InvalidCABundle   configmap/threescale-ca-bundle   InvalidCAFormat: No valid PEM-encoded certificates found in CA bundle
24m         Warning   InvalidCABundle   configmap/threescale-ca-bundle   InvalidCAFormat: No valid PEM-encoded certificates found in CA bundle
21m         Warning   InvalidCABundle   configmap/threescale-ca-bundle   InvalidCAFormat: No valid PEM-encoded certificates found in CA bundle

=== Step 4: Trigger reconcile and verify Synced=True ===
backend.capabilities.3scale.net/test-backend-ca annotated
--- Backend conditions ---
Failed	False
Invalid	False
Synced	True

Test setting up ca bundle after upgrade from 2.16.3 using OLM

Tested using live cluster with following configuration (with catalog image built from the PR branch):

   oc apply -f - <<'EOF'
   apiVersion: operators.coreos.com/v1alpha1
   kind: CatalogSource
   metadata:
     name: threescale-dev
     namespace: openshift-marketplace
   spec:
     sourceType: grpc
     image: quay.io/burbanik/3scale-operator-catalog:v0.14.0
   EOF

   oc apply -f - <<'EOF'
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: threescale-og
     namespace: 3scale-test
   spec:
     targetNamespaces:
     - 3scale-test
   EOF

   oc apply -f - <<'EOF'
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: 3scale-operator
     namespace: 3scale-test
   spec:
     source: threescale-dev
     sourceNamespace: openshift-marketplace
     name: 3scale-operator
     channel: threescale-2.16
     startingCSV: 3scale-operator.v0.13.3
     installPlanApproval: Manual
   EOF


cat <<'EOF' | oc apply -n 3scale-test -f -
  apiVersion: capabilities.3scale.net/v1beta1
  kind: Backend
  metadata:
    name: test-backend
    namespace: 3scale-test
  spec:
    name: "Test Backend"
    privateBaseURL: "https://echo-api.3scale.net:443"
  EOF)

# at this point the sync should be failing

# upgrade to 2.17 and create router-ca bundle

oc get secret router-ca -n openshift-ingress-operator -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/ingress-ca.crt
oc create configmap threescale-ca-bundle --from-file=ca-bundle.crt=/tmp/ingress-ca.crt -n 3scale-test

Result:

Backend status before upgrade:

borisurbanik@MacBookPro 3scale-operator % oc get backend test-backend -o json | jq -r '.status.conditions'
[
  {
    "lastTransitionTime": "2026-07-02T21:31:39Z",
    "message": "Get \"https://3scale-admin.apps.burbanik-3scale.cp.fyre.ibm.com:443/admin/api/backend_apis.json?page=1&per_page=500\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
    "status": "True",
    "type": "Failed"
  },
  {
    "lastTransitionTime": "2026-07-02T21:31:39Z",
    "status": "False",
    "type": "Invalid"
  },
  {
    "lastTransitionTime": "2026-07-02T21:31:39Z",
    "status": "False",
    "type": "Synced"
  }
]

After upgrade with custom CA:

borisurbanik@MacBookPro 3scale-operator % oc get backend test-backend -o json | jq -r '.status.conditions'
[
  {
    "lastTransitionTime": "2026-07-02T21:53:44Z",
    "status": "False",
    "type": "Failed"
  },
  {
    "lastTransitionTime": "2026-07-02T21:31:39Z",
    "status": "False",
    "type": "Invalid"
  },
  {
    "lastTransitionTime": "2026-07-02T21:53:44Z",
    "status": "True",
    "type": "Synced"
  }
]

openshift-ci · 2026-06-25T22:25:16Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

codecov-commenter · 2026-06-25T22:28:32Z

Codecov Report

❌ Patch coverage is 78.78788% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.65%. Comparing base (c59a4c8) to head (4690e33).
⚠️ Report is 12 commits behind head on master.

Files with missing lines	Patch %	Lines
controllers/configuration/ca_bundle_watcher.go	82.50%	7 Missing ⚠️
main.go	0.00%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1181      +/-   ##
==========================================
+ Coverage   44.03%   44.65%   +0.62%     
==========================================
  Files         204      208       +4     
  Lines       20960    21235     +275     
==========================================
+ Hits         9230     9483     +253     
- Misses      10933    10952      +19     
- Partials      797      800       +3

Flag	Coverage Δ
unit	`44.65% <78.78%> (+0.62%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
apis/apps/v1alpha1 (u)	`63.56% <ø> (ø)`
apis/capabilities/v1alpha1 (u)	`3.50% <ø> (ø)`
apis/capabilities/v1beta1 (u)	`20.21% <ø> (ø)`
controllers (i)	`12.59% <84.00%> (+0.51%)`	⬆️
pkg (u)	`64.26% <92.77%> (+0.56%)`	⬆️

Files with missing lines	Coverage Δ
controllers/configuration/tls_config.go	`100.00% <100.00%> (ø)`
pkg/controller/helper/threescale_api.go	`84.00% <100.00%> (+2.18%)`	⬆️
pkg/testhelper/tls.go	`100.00% <100.00%> (ø)`
controllers/configuration/ca_bundle_watcher.go	`82.50% <82.50%> (ø)`
main.go	`0.00% <0.00%> (ø)`

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

borisurbanik · 2026-07-07T21:59:19Z

/retest

briangallagher · 2026-07-27T05:08:58Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

tkan145

Do we really need all of those CR tests and mock just to test API call?

The most important thing is that ca_bundler_watcher is not tested anywhere.

borisurbanik · 2026-08-04T22:16:24Z

Do we really need all of those CR tests and mock just to test API call?

The most important thing is that ca_bundler_watcher is not tested anywhere.

As discussed, added the ca_bundler_watcher reconciler test and removed the mocked tests to be replaced in functional test in the future to eliminate the mocks.

borisurbanik · 2026-08-10T13:58:19Z

/retest

borisurbanik · 2026-08-10T16:40:14Z

/retest

borisurbanik · 2026-08-10T20:57:43Z

/retest

…ents Introduce a new CABundleWatcher controller that watches the threescale-ca-bundle ConfigMap and atomically publishes its parsed x509.CertPool to a package-level variable. The CA changes are watched on all replicas of operator, not just leader to have warmed up cache in case of a switchover. Use the pool in all PortaClient* constructors in pkg/controller/helper so that outbound TLS connections to the 3scale Admin API automatically trust operator-configured CAs; insecureSkipVerify bypasses this when set. Co-Authored-By: IBM Bob <AskBob@ibm.com>

tkan145 · 2026-08-13T23:45:07Z

/lgtm
/approve

openshift-ci Bot added the do-not-merge/work-in-progress label Jun 25, 2026

borisurbanik force-pushed the THREESCALE-10236 branch 8 times, most recently from 4c92127 to 1ef25a4 Compare June 30, 2026 21:56

borisurbanik marked this pull request as ready for review June 30, 2026 22:45

borisurbanik requested a review from a team as a code owner June 30, 2026 22:45

openshift-ci Bot removed the do-not-merge/work-in-progress label Jun 30, 2026

tkan145 reviewed Jul 3, 2026

View reviewed changes

borisurbanik force-pushed the THREESCALE-10236 branch from 15d9757 to 7284dae Compare July 6, 2026 21:37

tkan145 reviewed Jul 7, 2026

View reviewed changes

Comment thread controllers/configuration/ca_bundle_watcher.go

Comment thread controllers/configuration/tls_config.go Outdated

Comment thread pkg/controller/helper/threescale_api.go Outdated

Comment thread doc/operator-ca-bundle.md Outdated

Comment thread doc/operator-ca-bundle.md Outdated

tkan145 reviewed Jul 21, 2026

View reviewed changes

Comment thread pkg/controller/helper/threescale_api_test.go Outdated

Comment thread test/unitcontrollers/activedoc_controller_test.go Outdated

borisurbanik force-pushed the THREESCALE-10236 branch 3 times, most recently from 77d65d3 to 06f126f Compare July 27, 2026 22:30

tkan145 reviewed Aug 4, 2026

View reviewed changes

Comment thread pkg/controller/helper/threescale_api_test.go Outdated

borisurbanik force-pushed the THREESCALE-10236 branch 2 times, most recently from 0163710 to 18c6492 Compare August 4, 2026 22:11

borisurbanik force-pushed the THREESCALE-10236 branch from 18c6492 to 8f91974 Compare August 5, 2026 11:34

borisurbanik requested a review from tkan145 August 5, 2026 11:52

tkan145 reviewed Aug 10, 2026

View reviewed changes

borisurbanik force-pushed the THREESCALE-10236 branch 2 times, most recently from b335c16 to 7caedce Compare August 10, 2026 11:08

borisurbanik mentioned this pull request Aug 11, 2026

Disable synthetic test for deployment update #1187

Merged

borisurbanik force-pushed the THREESCALE-10236 branch from 7caedce to dc80df8 Compare August 11, 2026 08:04

borisurbanik force-pushed the THREESCALE-10236 branch from dc80df8 to 4690e33 Compare August 13, 2026 18:26

tkan145 approved these changes Aug 13, 2026

View reviewed changes

borisurbanik merged commit a862087 into 3scale:master Aug 14, 2026
22 checks passed

borisurbanik deleted the THREESCALE-10236 branch August 14, 2026 04:55

Uh oh!

Conversation

borisurbanik commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing:

Uh oh!

openshift-ci Bot commented Jun 25, 2026

Uh oh!

codecov-commenter commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

borisurbanik commented Jul 7, 2026

Uh oh!

Uh oh!

Uh oh!

briangallagher commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

tkan145 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

borisurbanik commented Aug 4, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

borisurbanik commented Aug 10, 2026

Uh oh!

borisurbanik commented Aug 10, 2026

Uh oh!

borisurbanik commented Aug 10, 2026

Uh oh!

tkan145 commented Aug 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

borisurbanik commented Jun 25, 2026 •

edited

Loading

codecov-commenter commented Jun 25, 2026 •

edited

Loading

briangallagher commented Jul 27, 2026 •

edited

Loading