OpenShift follow-up for NVBugs 6218358 (26.05)

docs/docs/extraction/deployment-options.md

@@ -27,8 +27,11 @@ so the service container installs `ffmpeg` and `ffprobe` at startup. This
 runtime install requires package-repository network egress, a writable root
 filesystem, and security policy that allows the image's scoped sudo use. If
 your cluster blocks startup package installation (for example air-gapped
-environments), use a custom service image that already contains `ffmpeg` and
-`ffprobe`, then set `service.image.repository` and `service.image.tag`.
+environments or **OpenShift restricted-v2**), use a custom service image that
+already contains `ffmpeg` and `ffprobe`, then set `service.image.repository`
+and `service.image.tag`. For OpenShift overrides, see
+[OpenShift deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#openshift-deployment)
+in the Helm chart README.
 
 ### I want examples and notebooks
 

docs/docs/extraction/prerequisites-support-matrix.md

@@ -11,8 +11,11 @@ Before you begin using [NeMo Retriever Library](overview.md), confirm your softw
 - For audio and video, `ffmpeg` and `ffprobe` must be on `PATH` (for example
   `sudo apt-get install -y --no-install-recommends ffmpeg` on Debian/Ubuntu).
   `ffmpeg-python` and `nemo-retriever[multimedia]` do not install these binaries.
-  On Helm with package-repo access, set `service.installFfmpeg=true`. For
-  air-gapped clusters, see [Air-gapped and disconnected deployment](deployment-options.md#air-gapped-deployment).
+  On Helm with package-repo access, set `service.installFfmpeg=true`. On
+  OpenShift restricted-v2, use a prebuilt service image instead — see
+  [OpenShift deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#audio-and-video-ffmpeg-on-restricted-openshift)
+  in the Helm chart README. For air-gapped clusters, see
+  [Air-gapped and disconnected deployment](deployment-options.md#air-gapped-deployment).
 
 !!! note
 
@@ -98,6 +101,8 @@ These NIM microservices are **optional** for the default extraction pipeline. Th
 
 For 26.05, use **`nemotron_3_nano_omni_30b_a3b_reasoning`** when you enable the caption stage (hosted model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`). The Helm key is in the [optional NIMs](#optional-helm-nims-not-auto-wired-by-default) table above.
 
+For direct `/v1/chat/completions` smoke tests against the Omni NIM (outside the retriever service), set `chat_template_kwargs.enable_thinking=false` so the caption appears in `message.content`. The service caption profile already applies this during ingest. See [OpenShift deployment — Omni caption manual smoke testing](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#omni-caption-manual-smoke-testing) for a request example.
+
 Optional features listed in the table above require additional GPU support, disk space, and feature-specific system dependencies beyond the four default NIMs.
 
 For published NIM model IDs and deployment-specific constraints, use the product support matrices linked under [Related Topics](#related-topics) below.

nemo_retriever/helm/README.md

@@ -297,7 +297,10 @@ short list of knobs you'll touch first.
 | `service.gpu.enabled`         | `false`                            | The service does **not** need a GPU. |
 
 For audio and video extraction, set `service.installFfmpeg=true` when your
-cluster allows runtime package installation. For air-gapped clusters, see
+cluster allows runtime package installation. **OpenShift restricted-v2** blocks
+that path — use a prebuilt service image instead; see
+[Audio and video on restricted OpenShift](#audio-and-video-ffmpeg-on-restricted-openshift).
+For air-gapped clusters, see
 [Deployment options — Air-gapped and disconnected deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/deployment-options/#air-gapped-deployment).
 
 ### Audio and video (Parakeet ASR) { #audio-video-parakeet }
@@ -939,6 +942,308 @@ sanity check before opening Grafana.
 
 ---
 
+## OpenShift deployment { #openshift-deployment }
+
+The chart defaults target generic Kubernetes clusters that allow fixed numeric
+UIDs (`runAsUser` / `runAsGroup` / `fsGroup` **1000**). **OpenShift 4.x**
+namespaces under the default **restricted-v2** Security Context Constraint (SCC)
+and **Pod Security Admission (PSA) `restricted`** profile assign a per-namespace
+UID/GID range instead. A stock `helm install` without overrides therefore fails
+SCC validation, emits PSA warnings, or crashes on log paths the random UID cannot
+write.
+
+We do **not** change chart defaults for OpenShift-only behavior (that would affect
+other platforms). Use the overrides below on OpenShift, or save the YAML block
+into a local values file and pass `-f <file>`.
+
+### Cluster posture (typical QA / hardened namespaces)
+
+| Control | Typical default on a new OpenShift project |
+| --- | --- |
+| SCC | **restricted-v2** (first match in priority order) |
+| PSA | `pod-security.kubernetes.io/warn=restricted` (and often `audit=restricted`; `enforce` may be unset on dev clusters) |
+| UID assignment | SCC injects `runAsUser` / `fsGroup` from the namespace range (for example `1000750000–1000759999`) |
+
+On clusters with **PSA `enforce=restricted`**, missing container `securityContext`
+fields become hard rejections, not warnings.
+
+### Override reference (maps to chart limitations)
+
+| Symptom on stock install | Cause | Helm override |
+| --- | --- | --- |
+| `FailedCreate`: UID/GID **1000** not in namespace range | Hardcoded `service.podSecurityContext` UID/GID/fsGroup | Omit `runAsUser`, `runAsGroup`, and `fsGroup`; keep only `runAsNonRoot: true` |
+| PSA warning: `allowPrivilegeEscalation`, capabilities, `seccompProfile` | Empty `service.securityContext` | Set restricted baseline on `service.securityContext` (see sample below) |
+| `PermissionError` on `/var/lib/nemo-retriever/retriever-service.log` when `persistence.enabled=false` | Default log path is image-owned; random UID cannot write without a PVC | Point `serviceConfig.logging.file` at `/tmp/...` (chart mounts `emptyDir` at `/tmp`) |
+| `CreateContainerConfigError`: non-numeric image `USER nemo` on **vectordb** | Vectordb container has no `securityContext` block for SCC to annotate | Disable vectordb for smoke tests, or patch the vectordb Deployment after install (below) |
+| PSA warnings on **otel-collector** | Otel Deployment has no `securityContext` in the chart | `topology.otel.enabled=false` unless you patch that Deployment |
+| Audio/video fails or pod never gets `ffmpeg` | `service.installFfmpeg=true` runs sudo at startup; **restricted-v2** blocks privilege escalation (`no-new-privileges`) | Prebuild a service image with `ffmpeg`/`ffprobe` baked in (see [Audio and video on restricted OpenShift](#audio-and-video-ffmpeg-on-restricted-openshift)); leave `service.installFfmpeg=false` |
+| `ImagePullBackOff` for a service image in the **internal OpenShift registry** | Chart `imagePullSecrets` lists only `ngc-secret`; the namespace ServiceAccount `dockercfg` secret is not merged automatically | Add the SA pull secret under `imagePullSecrets` (see [Internal registry pull secrets](#internal-registry-pull-secrets)) |
+| Optional NIM `CrashLoopBackOff` with missing `.so` in logs | GPU/CUDA libraries not on `LD_LIBRARY_PATH` for some NIM Operator stacks on OCP | Append paths via `nimOperator.<key>.env` (see [Optional NIM runtime environment](#optional-nim-runtime-environment)) |
+
+### Recommended value overrides
+
+Save the block below as a local values file (for example `openshift-restricted.yaml`)
+and pass `-f openshift-restricted.yaml` on every `helm install` / `helm upgrade`.
+This is the profile QA used on OpenShift 4.20 with the NIM Operator: core PDF
+pipeline, Parakeet ASR (with a prebuilt `ffmpeg` service image), and Omni captioning
+all reached Ready when combined with the NGC secrets and any NIM-specific env
+overrides noted in later subsections.
+
+```yaml
+# OpenShift overrides for nemo-retriever Helm chart (restricted-v2 / PSA restricted).
+# helm install retriever ./nemo_retriever/helm -f openshift-restricted.yaml ...
+
+service:
+  podSecurityContext:
+    runAsNonRoot: true
+    # Do NOT set runAsUser, runAsGroup, or fsGroup — OpenShift SCC assigns them.
+  securityContext:
+    runAsNonRoot: true
+    allowPrivilegeEscalation: false
+    capabilities:
+      drop: ["ALL"]
+    seccompProfile:
+      type: RuntimeDefault
+  # Leave installFfmpeg false on restricted-v2; use a custom image for audio/video.
+  installFfmpeg: false
+
+serviceConfig:
+  logging:
+    # Writable without persistence PVC (chart always mounts emptyDir at /tmp).
+    file: /tmp/retriever-service.log
+  vectordb:
+    # Set false for minimal service-only validation; see vectordb patch below if enabled.
+    enabled: false
+
+topology:
+  otel:
+    enabled: false
+```
+
+When **`persistence.enabled=true`**, you can keep the default log path under
+`persistence.mountPath` (`/var/lib/nemo-retriever`) because the PVC is mounted and
+SCC-assigned `fsGroup` applies. When persistence is off, always relocate logs to
+`/tmp` (or another path backed by `service.extraVolumes`).
+
+### Audio and video (`ffmpeg`) on restricted OpenShift { #audio-and-video-ffmpeg-on-restricted-openshift }
+
+Elsewhere this README documents `service.installFfmpeg=true`, which installs
+`ffmpeg`/`ffprobe` at container startup via passwordless `sudo`. On OpenShift
+**restricted-v2** SCC, that path is blocked: PSA restricted sets
+`allowPrivilegeEscalation: false` and the SCC enforces **no-new-privileges**, so
+the entrypoint cannot elevate to install packages.
+
+For audio and video extraction on OpenShift, **do not** set `service.installFfmpeg=true`.
+Instead, extend the service image on a connected build host and point the chart at
+that tag (same pattern as [air-gapped custom images](#1-service-image)):
+
+```dockerfile
+FROM nvcr.io/nvstaging/nim/nrl-service:<BASE_TAG>
+USER root
+RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg \
+    && rm -rf /var/lib/apt/lists/*
+USER nemo
+```
+
+Push the result to NGC, your private registry, or the
+[OpenShift internal registry](#internal-registry-pull-secrets), then set
+`service.image.repository` / `service.image.tag` accordingly.
+
+### Internal registry pull secrets { #internal-registry-pull-secrets }
+
+When you rebuild the service image into the OpenShift internal registry
+(`image-registry.openshift-image-registry.svc:5000/...`), pods pull through the
+namespace ServiceAccount's automatic `kubernetes.io/dockercfg` secret. The chart
+renders an explicit `imagePullSecrets` list on every Pod; that list **replaces**
+the implicit SA default rather than merging with it. If you set only `ngc-secret`
+(for NGC NIM images) but point `service.image` at the internal registry, the
+service Pod can fail with `ImagePullBackOff`.
+
+**Internal-registry service image only** — omit the chart-managed NGC pull secret
+name so the helper does not inject `ngc-secret` alone:
+
+```yaml
+ngcImagePullSecret:
+  create: false
+  # Do not set name — leaves ngc-secret out of the rendered Pod spec.
+
+imagePullSecrets:
+  - name: default-dockercfg-xxxxx   # replace with your SA secret (see below)
+```
+
+**Both NGC NIMs and an internal-registry service image** — list every secret the
+pods need:
+
+```yaml
+ngcImagePullSecret:
+  create: false
+  name: ngc-secret
+
+imagePullSecrets:
+  - name: ngc-secret
+  - name: default-dockercfg-xxxxx
+```
+
+Look up the ServiceAccount pull secret name:
+
+```bash
+oc get sa default -n nemo-retriever -o jsonpath='{.imagePullSecrets[*].name}{"\n"}'
+```
+
+Pre-create `ngc-secret` / `ngc-api` with `oc create secret` when
+`ngcImagePullSecret.create=false` and `ngcApiSecret.create=false`, as in the
+install examples below.
+
+### Optional NIM runtime environment { #optional-nim-runtime-environment }
+
+On OpenShift 4.20 with the NIM Operator, QA found that some **optional** NIM pods
+(for example Parakeet ASR or Omni caption) needed an explicit `LD_LIBRARY_PATH`
+so CUDA/driver libraries from the GPU Operator stack were visible at runtime.
+Symptoms are `CrashLoopBackOff` and log lines referencing a missing `.so` (for
+example `libcudart` or `libcudnn`).
+
+Merge additional `env` entries under the relevant `nimOperator.<key>.env` list in
+your values file (retain the chart's existing `env` entries for that NIM and
+append `LD_LIBRARY_PATH`):
+
+```yaml
+nimOperator:
+  audio:
+    env:
+      # ... chart defaults (NIM_TAGS_SELECTOR, NIM_TRITON_LOG_VERBOSE, etc.) ...
+      - name: LD_LIBRARY_PATH
+        value: "/usr/local/nvidia/lib64:/usr/local/cuda/lib64"
+  nemotron_3_nano_omni_30b_a3b_reasoning:
+    env:
+      # ... chart defaults (NIM_HTTP_API_PORT, etc.) ...
+      - name: LD_LIBRARY_PATH
+        value: "/usr/local/nvidia/lib64:/usr/local/cuda/lib64"
+```
+
+Paths vary with GPU Operator version and node image. Inspect a healthy GPU
+workload on the same cluster (`oc exec` into a CUDA sample pod) or the failing
+NIM pod's filesystem before pinning production values.
+
+### Omni caption manual smoke testing { #omni-caption-manual-smoke-testing }
+
+The retriever service caption profile already sends
+`chat_template_kwargs.enable_thinking=false` to the Omni NIM during ingest. When
+you call the Omni NIM **directly** at `/v1/chat/completions` (for example with
+`curl` against the in-cluster `NIMService`), include the same flag so the
+caption text lands in `message.content` instead of reasoning-only fields:
+
+```json
+{
+  "model": "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning",
+  "messages": [{"role": "user", "content": "..."}],
+  "chat_template_kwargs": {"enable_thinking": false}
+}
+```
+
+See [Image captioning](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/docs/docs/extraction/prerequisites-support-matrix.md#image-captioning-2605)
+for pipeline scope (chart regions are not captioned).
+
+### Example install on OpenShift 4.20 (service-only smoke test)
+
+Matches early QA repro with external NIMs disabled, no persistence, and no
+results PVC:
+
+```bash
+oc new-project nemo-retriever
+
+oc create secret docker-registry ngc-secret -n nemo-retriever \
+  --docker-server=nvcr.io --docker-username='$oauthtoken' \
+  --docker-password="$NGC_API_KEY"
+
+oc create secret generic ngc-api -n nemo-retriever \
+  --from-literal=NGC_API_KEY="$NGC_API_KEY" \
+  --from-literal=NGC_CLI_API_KEY="$NGC_API_KEY"
+
+helm install retriever ./nemo_retriever/helm -n nemo-retriever \
+  -f openshift-restricted.yaml \
+  --set ngcImagePullSecret.create=false \
+  --set ngcApiSecret.create=false \
+  --set nims.enabled=false \
+  --set persistence.enabled=false \
+  --set retrieverResults.enabled=false \
+  --set service.image.repository=nvcr.io/nvstaging/nim/nrl-service \
+  --set service.image.tag=<TAG>
+```
+
+Verify pods:
+
+```bash
+oc get pods -n nemo-retriever
+oc describe pod -l app.kubernetes.io/name=nemo-retriever -n nemo-retriever
+```
+
+You should see SCC-assigned numeric `runAsUser` on containers that declare a
+`securityContext` block, and no PSA warnings once overrides are applied.
+
+### Example install with NIM Operator (full pipeline validation)
+
+After NIM Operator and GPU Operator are installed, reuse `openshift-restricted.yaml`
+and pre-created NGC secrets. Point `service.image` at a **ffmpeg-enabled** build
+when validating [audio and video](#audio-and-video-ffmpeg-on-restricted-openshift).
+Add [optional NIM `LD_LIBRARY_PATH`](#optional-nim-runtime-environment) overrides
+if ASR or Omni pods crash on missing libraries.
+
+```bash
+helm install retriever ./nemo_retriever/helm -n nemo-retriever \
+  -f openshift-restricted.yaml \
+  --set ngcImagePullSecret.create=false \
+  --set ngcApiSecret.create=false \
+  --set service.image.repository=nvcr.io/nvstaging/nim/nrl-service \
+  --set service.image.tag=<TAG>
+```
+
+QA validated this posture on OpenShift 4.20 with default PDF, Parakeet ASR, and
+Omni caption smoke tests passing when the overrides above and any per-NIM env
+fixes were applied.
+
+### Enabling the vectordb Deployment on OpenShift
+
+`serviceConfig.vectordb.enabled=true` renders a **vectordb** container from the
+same image (`USER nemo`, non-numeric). The chart does not yet expose a
+`securityContext` value for that container. After `helm install`, patch the
+Deployment so OpenShift can inject a numeric UID into the container spec:
+
+```bash
+RELEASE=retriever
+NS=nemo-retriever
+VDB_DEPLOY="${RELEASE}-nemo-retriever-vectordb"
+
+oc patch deployment "$VDB_DEPLOY" -n "$NS" --type=json -p='[
+  {"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": {
+    "allowPrivilegeEscalation": false,
+    "capabilities": {"drop": ["ALL"]},
+    "runAsNonRoot": true,
+    "seccompProfile": {"type": "RuntimeDefault"}
+  }}
+]'
+```
+
+Re-apply the patch after `helm upgrade` if the Deployment is recreated. A future
+chart release may add first-class `topology.vectordb.securityContext` values.
+
+### Enabling the OpenTelemetry collector on OpenShift
+
+The chart’s otel-collector Deployment likewise lacks `securityContext` fields.
+Prefer `topology.otel.enabled=false` (as in the sample values) unless you operate
+your own collector or patch `*-otel` the same way as vectordb.
+
+### What we intentionally do not require on OpenShift
+
+Do **not** bind the namespace to **anyuid** SCC or set PSA `enforce=privileged`
+unless your security team explicitly approves it. The overrides above are intended
+to keep **restricted-v2** / PSA **restricted** posture.
+
+### Related documentation
+
+- [Pre-Requisites & Support Matrix](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/docs/docs/extraction/prerequisites-support-matrix.md)
+- [Deployment options](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/docs/docs/extraction/deployment-options.md)
+
+---
 ## Air-gapped deployment { #air-gapped-deployment }
 
 See [Deployment options — Air-gapped and disconnected deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/deployment-options/#air-gapped-deployment) for overview and workflow. Chart-specific reference for mirroring: