diff --git a/docs/docs/extraction/deployment-options.md b/docs/docs/extraction/deployment-options.md index 6049312343..083f69fc56 100644 --- a/docs/docs/extraction/deployment-options.md +++ b/docs/docs/extraction/deployment-options.md @@ -18,17 +18,21 @@ Use the sections below to pick documentation and deployment options that match y 3. **Published Library Helm charts (supported):** cluster install and upgrade procedures are covered in the [NeMo Retriever Library](https://docs.nvidia.com/nemo/retriever/latest/extraction/overview/) — use alongside the NeMo Retriever chart README for your release 4. [Environment variables](environment-config.md) and [Troubleshoot](troubleshoot.md) as needed -**Core NIMs for the default extraction pipeline** (26.05): `page_elements`, `table_structure`, `ocr`, and `vlm_embed` (`llama-nemotron-embed-vl-1b-v2:1.12.0`). These four are auto-wired into the retriever service. **Nemotron Parse**, **Nemotron 3 Nano Omni**, the **VL reranker**, and **Parakeet ASR** are optional and not auto-wired. For a minimal GPU footprint, disable optional keys you do not need (see [Recommended minimal install (26.05)](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). See [Pre-Requisites & Support Matrix — Default Helm NIMs](prerequisites-support-matrix.md#default-helm-nims). +**Core NIMs for the default extraction pipeline** (26.05): `page_elements`, `table_structure`, `ocr`, and `vlm_embed` (`llama-nemotron-embed-vl-1b-v2:1.12.0`). These four are auto-wired into the retriever service. **Nemotron Parse**, **Nemotron 3 Nano Omni**, the **VL reranker**, and **Parakeet ASR** are optional and not auto-wired. For a minimal GPU footprint, disable optional keys you do not need (refer to [Recommended minimal install (26.05)](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Refer to [Pre-Requisites & Support Matrix — Default Helm NIMs](prerequisites-support-matrix.md#default-helm-nims). **Docker Compose (unsupported, developer-only):** [Docker Compose for local development](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docker.md) — **not** a substitute for Helm or the published Library charts. -For audio and video extraction in Kubernetes, set `service.installFfmpeg=true` -so the service container installs `ffmpeg` and `ffprobe` at startup. This -runtime install requires package-repository network egress, a writable root -filesystem, and security policy that allows the image's scoped sudo use. If -your cluster blocks startup package installation (for example air-gapped -environments), use a custom service image that already contains `ffmpeg` and -`ffprobe`, then set `service.image.repository` and `service.image.tag`. +For audio and video extraction in Kubernetes, the Helm chart defaults to +`service.installFfmpeg=true`, which installs `ffmpeg` and `ffprobe` at container +startup. That path requires package-repository network egress, a writable root +filesystem, and security policy that allows the image's scoped sudo use. If your +cluster blocks startup package installation (air-gapped environments or +**OpenShift restricted-v2**), build a custom service image with `ffmpeg` and +`ffprobe` baked in, set `service.installFfmpeg=false`, and point +`service.image.repository` / `service.image.tag` at that image. For OpenShift +value overrides, refer to +[OpenShift deployment](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#openshift-deployment) +in the Helm chart README. ### I want examples and notebooks diff --git a/docs/docs/extraction/prerequisites-support-matrix.md b/docs/docs/extraction/prerequisites-support-matrix.md index 9f97db16f6..2ce60fcec6 100644 --- a/docs/docs/extraction/prerequisites-support-matrix.md +++ b/docs/docs/extraction/prerequisites-support-matrix.md @@ -11,8 +11,7 @@ Before you begin using [NeMo Retriever Library](overview.md), confirm your softw - For audio and video, `ffmpeg` and `ffprobe` must be on `PATH` (for example `sudo apt-get install -y --no-install-recommends ffmpeg` on Debian/Ubuntu). `ffmpeg-python` and `nemo-retriever[multimedia]` do not install these binaries. - On Helm with package-repo access, set `service.installFfmpeg=true`. For - air-gapped clusters, see [Air-gapped and disconnected deployment](deployment-options.md#air-gapped-deployment). + For container and Kubernetes guidance, refer to [Audio and video](audio-video.md). - For PDF extraction with `extract_method="nemotron_parse"`, install the Nemotron Parse client dependencies with `pip install "nemo-retriever[nemotron-parse]"` (pulls `open-clip-torch`, which provides the `open_clip` module required by the Nemotron Parse @@ -66,7 +65,7 @@ The NeMo Retriever Library extraction core pipeline features run on a single A10 ### Default Helm NIMs { #default-helm-nims } -The production Helm chart enables these NIM microservices **by default** (for example via `nimOperator.*.enabled=true`): +The production Helm chart enables these NIM microservices **by default** (for example with `nimOperator.*.enabled=true`): | Helm flag | NIM | Role | |-----------|-----|------| @@ -79,9 +78,9 @@ The production Helm chart enables these NIM microservices **by default** (for ex !!! note - **Helm / NIM:** The production chart deploys **Nemotron OCR v1** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). For image defaults and upgrade notes, see [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. + **Helm / NIM:** The production chart deploys **Nemotron OCR v1** under `nimOperator.ocr` (`nvcr.io/nim/nvidia/nemotron-ocr-v1:1.3.0`). For image defaults and upgrade notes, refer to [OCR NIM configuration](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#ocr-nim-configuration) in the Helm chart README. - **Local Hugging Face inference:** When you deploy locally with HuggingFace model weights (for example `pip install "nemo-retriever[local]"` and GPU inference without remote OCR NIM URLs), the default OCR engine is **Nemotron OCR v2**, which runs in **multilingual** mode by default. For CLI flags and API parameters, see [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). Remote OCR NIM endpoints use their own model and language behavior; local OCR language selectors are not sent on remote requests. + **Local Hugging Face inference:** When you deploy locally with HuggingFace model weights (for example `pip install "nemo-retriever[local]"` and GPU inference without remote OCR NIM URLs), the default OCR engine is **Nemotron OCR v2**, which runs in **multilingual** mode by default. For CLI flags and API parameters, refer to [Nemotron OCR v2 — language mode](https://github.com/NVIDIA/NeMo-Retriever/blob/main/nemo_retriever/docs/cli/README.md#nemotron-ocr-v2-language-mode). Remote OCR NIM endpoints use their own model and language behavior; local OCR language selectors are not sent on remote requests. Default OCR NIM container for release Helm deployments: @@ -94,7 +93,7 @@ Default VL embedder container and model for release deployments: ### Optional Helm NIMs (not auto-wired) { #optional-helm-nims-not-auto-wired-by-default } -These NIM microservices are **optional** for the default extraction pipeline. The retriever service does **not** call them until you enable the matching pipeline stage (reranker, Nemotron Parse, caption, or audio). For **26.05 production**, disable keys you do not need (see [Recommended minimal install (26.05)](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Set `nimOperator..enabled=true` when you want that NIM reconciled. Chart keys are in the [NeMo Retriever Helm chart README](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#nim-operator-sub-stack). +These NIM microservices are **optional** for the default extraction pipeline. The retriever service does **not** call them until you enable the matching pipeline stage (reranker, Nemotron Parse, caption, or audio). For **26.05 production**, disable keys you do not need (refer to [Recommended minimal install (26.05)](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#recommended-minimal-install-2605)). Set `nimOperator..enabled=true` when you want that NIM reconciled. Chart keys are in the [NeMo Retriever Helm chart README](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/nemo_retriever/helm/README.md#nim-operator-sub-stack). | Helm flag | NIM | Role | |-----------|-----|------| @@ -105,7 +104,7 @@ These NIM microservices are **optional** for the default extraction pipeline. Th ### Image captioning (26.05) { #image-captioning-2605 } -For 26.05, use **`nemotron_3_nano_omni_30b_a3b_reasoning`** when you enable the caption stage (hosted model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`). The Helm key is in the [optional NIMs](#optional-helm-nims-not-auto-wired-by-default) table above. +For 26.05, use **`nemotron_3_nano_omni_30b_a3b_reasoning`** when you enable the caption stage (hosted model ID `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`). The Helm key is in the [optional NIMs](#optional-helm-nims-not-auto-wired-by-default) table above. For pipeline scope (PDF chart regions are not captioned), refer to [Image captioning](multimodal-extraction.md#image-captioning). Optional features listed in the table above require additional GPU support, disk space, and feature-specific system dependencies beyond the four default NIMs. @@ -116,7 +115,7 @@ For published NIM model IDs and deployment-specific constraints, use the product NeMo Retriever Library supports the following GPU hardware given system constraints in the table. - **HF model weights** — approximate Hugging Face checkpoint footprint (files such as `model*.safetensors`, `weights.pth`, or other published weight bundles in the model repository). Values are rounded from the current public file listing and can change when the repository is updated. -- **NIM disk space** — approximate container and on-disk model cache for self-hosted NIM microservices (not the same as HF download size). For Nemotron 3 Nano Omni captioning, see the [NVIDIA NIM for Vision Language Models support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). +- **NIM disk space** — approximate container and on-disk model cache for self-hosted NIM microservices (not the same as HF download size). For Nemotron 3 Nano Omni captioning, refer to the [NVIDIA NIM for Vision Language Models support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). Model repositories and NIM references are linked in [Core and Advanced Pipeline Features](#core-and-advanced-pipeline-features) above. @@ -139,11 +138,11 @@ Model repositories and NIM references are linked in [Core and Advanced Pipeline ¹ On other supported GPUs, Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`) may require a runtime TensorRT engine build (no prebuilt profile in the chart image). -⁴ On **B200** and other **Blackwell** GPUs (compute capability 12.0), including RTX PRO 6000 Blackwell and RTX PRO 4500 Blackwell, self-hosted [audio/video extraction](audio-video.md) via Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`, `nimOperator.audio`) is **not supported**. Core PDF and multimodal extraction on Blackwell is unchanged. Video workflows that depend on Parakeet for speech transcription are affected the same way. `NIMService` for `nimOperator.audio` may stay not Ready or enter `CrashLoopBackOff` while building the Riva/TensorRT engine (for example ONNX Runtime IR version, cuDNN visibility, or FP8 tactic errors). Use a non-Blackwell dedicated GPU, [hosted Parakeet on build.nvidia.com](audio-video.md#parakeet-hosted-inference-build-nvidia), or set `nimOperator.audio.enabled=false`. +⁴ On **B200** and other **Blackwell** GPUs (compute capability 12.0), including RTX PRO 6000 Blackwell and RTX PRO 4500 Blackwell, self-hosted [audio/video extraction](audio-video.md) through Parakeet ASR (`parakeet-1-1b-ctc-en-us:1.5.0`, `nimOperator.audio`) is **not supported**. Core PDF and multimodal extraction on Blackwell is unchanged. Video workflows that depend on Parakeet for speech transcription are affected the same way. `NIMService` for `nimOperator.audio` may stay not Ready or enter `CrashLoopBackOff` while building the Riva/TensorRT engine (for example ONNX Runtime IR version, cuDNN visibility, or FP8 tactic errors). Use a non-Blackwell dedicated GPU, [hosted Parakeet on build.nvidia.com](audio-video.md#parakeet-hosted-inference-build-nvidia), or set `nimOperator.audio.enabled=false`. ² Nemotron Parse fails to start on 32GB. -³ Opt-in Omni captioning uses the [nemotron-3-nano-omni-30b-a3b-reasoning](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3-nano-omni-30b-a3b-reasoning) NIM (`nvcr.io/nim/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant`). BF16 requires at least 80 GB total GPU memory; see the [VLM NIM support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). L40S requires two GPUs. A100 40GB, A10G, and RTX PRO 4500 are below the minimum. +³ Opt-in Omni captioning uses the [nemotron-3-nano-omni-30b-a3b-reasoning](https://docs.api.nvidia.com/nim/reference/nvidia-nemotron-3-nano-omni-30b-a3b-reasoning) NIM (`nvcr.io/nim/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant`). BF16 requires at least 80 GB total GPU memory; refer to the [VLM NIM support matrix](https://docs.nvidia.com/nim/vision-language-models/latest/support-matrix.html#nemotron-3-nano-omni-30b-a3b-reasoning). L40S requires two GPUs. A100 40GB, A10G, and RTX PRO 4500 are below the minimum. \* GPUs with less than 80GB VRAM cannot run the reranker concurrently with the core pipeline. To perform recall testing with the reranker on these GPUs, shut down the core pipeline NIM microservices diff --git a/nemo_retriever/helm/README.md b/nemo_retriever/helm/README.md index 9b99e5c32a..b364114799 100644 --- a/nemo_retriever/helm/README.md +++ b/nemo_retriever/helm/README.md @@ -80,7 +80,7 @@ nemo_retriever/helm/ ## Quick start -### 1. Service image +### 1. Service image { #1-service-image } The chart defaults to the staging image published to NGC: @@ -89,7 +89,7 @@ nvcr.io/nvstaging/nim/nemo-retriever-service:043020205-001 ``` Pulling from `nvcr.io/nvstaging` requires an NGC pull secret — either set -`ngcImagePullSecret.create=true` (see below) or pre-create one in the +`ngcImagePullSecret.create=true` (refer to the section below) or pre-create one in the namespace named `ngc-secret`. To run a locally built image instead, build and push it from the repo root, @@ -119,9 +119,9 @@ Do not also set `INSTALL_FFMPEG` in `service.env`; the chart fails rendering when both are configured so the rendered Pod does not contain duplicate environment variables. -When `service.installFfmpeg=false` (the default), the service still starts -normally and processes PDF, image, text and HTML uploads. Audio / video -uploads are rejected up-front with **HTTP 501**: +When `service.installFfmpeg=false`, the service still starts normally and +processes PDF, image, text and HTML uploads. Audio / video uploads are +rejected up-front with **HTTP 501**: ```text Audio and video ingestion require FFmpeg in the retriever service @@ -144,7 +144,7 @@ that allows sudo/setuid behavior. Do not set `service.securityContext.allowPrivilegeEscalation: false` or `service.securityContext.readOnlyRootFilesystem: true` for this path. -For air-gapped or locked-down clusters, see +For air-gapped or locked-down clusters, refer to [Deployment options — Air-gapped and disconnected deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/deployment-options/#air-gapped-deployment). On a connected staging host you can extend the service image, for example: @@ -190,7 +190,7 @@ NIM (the VL reranker `rerankqa`, Nemotron Parse, Omni 30B, and the Parakeet `audio` ASR NIM) is **disabled by default** to honor the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md); -see [Recommended minimal install (26.05)](#recommended-minimal-install-2605) +refer to [Recommended minimal install (26.05)](#recommended-minimal-install-2605) for the opt-in `--set` flags that turn any of them on. ```bash @@ -220,7 +220,7 @@ helm install retriever ./nemo_retriever/helm \ > * Omni 30B captioner — `--set nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning.enabled=true` > * Parakeet ASR — `--set nimOperator.audio.enabled=true` (also set `serviceConfig.nimEndpoints.audioGrpcEndpoint=audio:50051` to wire ASR into the service, plus `service.installFfmpeg=true` if your image does not bundle ffmpeg) > -> This matches the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md) and avoids silently pulling ≈ 62 GiB of Omni weights or claiming a second dedicated GPU on a "default" install. See the [model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements) table for per-NIM GPU and disk costs. +> This matches the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md) and avoids silently pulling ≈ 62 GiB of Omni weights or claiming a second dedicated GPU on a "default" install. Refer to the [model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements) table for per-NIM GPU and disk costs. The chart auto-wires the operator-managed in-cluster URLs of the four "core" NIMs into the service's `nim_endpoints` block: @@ -291,13 +291,16 @@ short list of knobs you'll touch first. | `service.image.repository` | `localhost:32000/nemo-retriever-service` | Override to a published image. | | `service.image.tag` | `latest` | | | `service.replicas` | `1` | Hard cap = 1 while SQLite is the backend. | -| `service.installFfmpeg` | `false` | Install `ffmpeg`/`ffprobe` at container startup by setting `INSTALL_FFMPEG=true`. Requires network egress, writable root filesystem, and sudo/setuid allowed. Not for air-gapped clusters — use a custom image instead. | +| `service.installFfmpeg` | `true` | Install `ffmpeg`/`ffprobe` at container startup by setting `INSTALL_FFMPEG=true`. Requires network egress, writable root filesystem, and sudo/setuid allowed. Set `false` on air-gapped or **OpenShift restricted-v2** clusters and use a [prebuilt service image](#audio-and-video-ffmpeg-on-restricted-openshift) instead. | | `service.resources.requests` | `16 / 16Gi` | Tune in tandem with `serviceConfig.pipeline.*Workers`. | | `service.resources.limits` | `96 / 96Gi` | | | `service.gpu.enabled` | `false` | The service does **not** need a GPU. | For audio and video extraction, set `service.installFfmpeg=true` when your -cluster allows runtime package installation. For air-gapped clusters, see +cluster allows runtime package installation. **OpenShift restricted-v2** blocks +that path — use a prebuilt service image instead; refer to +[Audio and video on restricted OpenShift](#audio-and-video-ffmpeg-on-restricted-openshift). +For air-gapped clusters, refer to [Deployment options — Air-gapped and disconnected deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/deployment-options/#air-gapped-deployment). ### Audio and video (Parakeet ASR) { #audio-video-parakeet } @@ -305,11 +308,11 @@ cluster allows runtime package installation. For air-gapped clusters, see To run self-hosted Parakeet for [audio and video extraction](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/audio-video.md): 1. Set `nimOperator.audio.enabled=true` (it is on by default; disable other optional NIMs you do not need per [Recommended minimal install (26.05)](#recommended-minimal-install-2605)). -2. Pin the ASR `NIMService` to a **dedicated GPU** with `nimOperator.audio.resources`, `nodeSelector`, or `tolerations` (see [NIM Operator](https://docs.nvidia.com/nim-operator/latest/index.html)). +2. Pin the ASR `NIMService` to a **dedicated GPU** with `nimOperator.audio.resources`, `nodeSelector`, or `tolerations` (refer to [NIM Operator](https://docs.nvidia.com/nim-operator/latest/index.html)). 3. Confirm the GPU SKU in [Model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements) (footnote ⁴ lists Blackwell limitations). -4. Set `service.installFfmpeg=true` when the retriever service will process audio or video (see `service.installFfmpeg` above). +4. Set `service.installFfmpeg=true` when the retriever service will process audio or video on clusters that allow runtime package install (refer to `service.installFfmpeg` above). On **OpenShift restricted-v2**, use a [prebuilt service image](#audio-and-video-ffmpeg-on-restricted-openshift) instead. -The retriever service picks up the in-cluster ASR endpoint when `nimOperator.audio` is enabled; see [NIM Operator sub-stack](#nim-operator-sub-stack). +The retriever service picks up the in-cluster ASR endpoint when `nimOperator.audio` is enabled; refer to [NIM Operator sub-stack](#nim-operator-sub-stack). ### Service configuration (rendered into `retriever-service.yaml`) @@ -317,10 +320,10 @@ The retriever service picks up the in-cluster ASR endpoint when `nimOperator.aud |---------------------------------------------------|---------|-------| | `serviceConfig.server.port` | `7670` | Container + Service port. | | `serviceConfig.pipeline.realtimeWorkers` | `24` | Per-pod realtime worker count. | -| `serviceConfig.pipeline.batchWorkers` | `48` | Per-pod batch worker count. See [Timeouts and alleviating ingest failures](#timeouts-and-alleviating-ingest-failures) if embed or pool errors appear under load. | -| `serviceConfig.nimEndpoints.*InvokeUrl` | `""` | Override the auto-resolved NIM Operator URL. Available knobs: `pageElementsInvokeUrl`, `tableStructureInvokeUrl`, `ocrInvokeUrl`, `embedInvokeUrl`, and `captionInvokeUrl` (see [Image captioning (Omni 30B)](#image-captioning-omni-30b)). | +| `serviceConfig.pipeline.batchWorkers` | `48` | Per-pod batch worker count. Refer to [Timeouts and alleviating ingest failures](#timeouts-and-alleviating-ingest-failures) if embed or pool errors appear under load. | +| `serviceConfig.nimEndpoints.*InvokeUrl` | `""` | Override the auto-resolved NIM Operator URL. Available knobs: `pageElementsInvokeUrl`, `tableStructureInvokeUrl`, `ocrInvokeUrl`, `embedInvokeUrl`, and `captionInvokeUrl` (refer to [Image captioning (Omni 30B)](#image-captioning-omni-30b)). | | `serviceConfig.nimEndpoints.captionModelName` | `""` | Model id sent to the remote VLM. Auto-set to `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning` whenever a caption URL is resolved. | -| `serviceConfig.vectordb.enabled` | `true` | Deploy the LanceDB vectordb Pod. When `true` the chart **requires** a resolvable embed endpoint (see [VectorDB and the embed endpoint](#vectordb-and-the-embed-endpoint)); `helm install` / `helm upgrade` fails fast otherwise. | +| `serviceConfig.vectordb.enabled` | `true` | Deploy the LanceDB vectordb Pod. When `true` the chart **requires** a resolvable embed endpoint (refer to [VectorDB and the embed endpoint](#vectordb-and-the-embed-endpoint)); `helm install` / `helm upgrade` fails fast otherwise. | | `serviceConfig.vectordb.lancedbUri` | `/data/vectordb` | LanceDB on the vectordb Pod's PVC. | | `serviceConfig.vectordb.embedModel` | `nvidia/llama-nemotron-embed-vl-1b-v2` | Passed to vectordb + worker `embed_model_name`. | @@ -347,7 +350,7 @@ resolved. Pick one of: 3. --set serviceConfig.vectordb.enabled=false ``` -Resolution order matches the rest of the chart (see [Mix and match NIM +Resolution order matches the rest of the chart (refer to [Mix and match NIM sources](#3-install-with-the-nim-operator-in-cluster-nims)): 1. Explicit `serviceConfig.nimEndpoints.embedInvokeUrl` always wins. @@ -378,17 +381,17 @@ pair gated on three conditions ALL holding: | `nimOperator.vlm_embed.image` | `nvcr.io/nim/nvidia/llama-nemotron-embed-vl-1b-v2:1.12.0` | Default VLM embed NIM image. | | `nimOperator.rerankqa.enabled` | `false` | VL reranker NIM (optional; not auto-wired). Set `true` to opt in. Default `false` so 26.05 installs honor the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md) and do not silently provision an extra ≈ 3.1 GiB GPU NIM. The image points at the **VL** SKU (`llama-nemotron-rerank-vl-1b-v2`) per [prerequisites-support-matrix.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#default-helm-nims) — the text-only `llama-nemotron-rerank-1b-v2` silently degrades multimodal reranking and is not the documented POR. | | `nimOperator.nemotron_parse.enabled` | `false` | Structured-parse NIM (optional). Set `true` when using `extract_method="nemotron_parse"`. Default `false` so 26.05 installs honor the "optional and disabled by default" contract in [deployment-options.md](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/deployment-options.md). Image tag follows the [image tag conventions](#image-tag-conventions). | -| `nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning.enabled` | `false` | Omni 30B caption NIM (optional). Set `true` to enable image captioning — see [Image captioning (Omni 30B)](#image-captioning-omni-30b). Default `false` so 26.05 installs do not silently pull ≈ 62 GiB of BF16 weights or claim a second dedicated GPU. Image tag follows the [image tag conventions](#image-tag-conventions). | +| `nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning.enabled` | `false` | Omni 30B caption NIM (optional). Set `true` to enable image captioning — refer to [Image captioning (Omni 30B)](#image-captioning-omni-30b). Default `false` so 26.05 installs do not silently pull ≈ 62 GiB of BF16 weights or claim a second dedicated GPU. Image tag follows the [image tag conventions](#image-tag-conventions). | | `nimOperator.audio.enabled` | `false` | Parakeet ASR NIM (optional). Set `true` for audio/video transcription; pair with `serviceConfig.nimEndpoints.audioGrpcEndpoint=audio:50051` so the retriever-service can reach it. | | `nimOperator..image.repository` | `nvcr.io/nim/nvidia/...` | Per-NIM image. | | `nimOperator..image.pullSecrets` | `[ngc-secret]` | Referenced by the NIMService CR. | | `nimOperator..authSecret` | `ngc-api` | NIM auth Secret name. | | `nimOperator..storage.pvc.size` | `25Gi` (50Gi for vlm_embed/rerankqa, 100Gi parse, 300Gi VL) | NIMCache PVC size. | | `nimOperator..replicas` | `1` | Per-NIMService replica count. | -| `nimOperator.nimServiceGpuLimit` | `1` | Default `nvidia.com/gpu` limit on every NIMService when per-NIM `resources` is `{}`. Set to `null` for operator-only reconciliation (not reliable on all NIM Operator versions — see [GPU limits and `helm upgrade`](#gpu-limits-and-helm-upgrade)). | +| `nimOperator.nimServiceGpuLimit` | `1` | Default `nvidia.com/gpu` limit on every NIMService when per-NIM `resources` is `{}`. Set to `null` for operator-only reconciliation (not reliable on all NIM Operator versions — refer to [GPU limits and `helm upgrade`](#gpu-limits-and-helm-upgrade)). | | `nimOperator..resources` | `{}` | Per-NIM override of the whole `resources` block. Empty uses `nimServiceGpuLimit`; non-empty replaces the chart default (may require `--force-conflicts` on later `helm upgrade`). | -| `nimOperator.modelProfile` | `{}` | Chart-wide NIMCache GPU/profile filter. Applied to every NIMCache that does not have its own override. See [Filtering cached GPU profiles](#filtering-cached-gpu-profiles). | -| `nimOperator..modelProfile` | `{}` | Per-NIM NIMCache GPU/profile filter. Non-empty values REPLACE the chart-wide default (no merge). See [Filtering cached GPU profiles](#filtering-cached-gpu-profiles). | +| `nimOperator.modelProfile` | `{}` | Chart-wide NIMCache GPU/profile filter. Applied to every NIMCache that does not have its own override. Refer to [Filtering cached GPU profiles](#filtering-cached-gpu-profiles). | +| `nimOperator..modelProfile` | `{}` | Per-NIM NIMCache GPU/profile filter. Non-empty values REPLACE the chart-wide default (no merge). Refer to [Filtering cached GPU profiles](#filtering-cached-gpu-profiles). | | `nimOperator..expose.service.port` | `8000` (9000 for audio) | HTTP port. | | `nimOperator..expose.service.grpcPort` | `8001` (50051 for audio) | gRPC port. | @@ -467,7 +470,7 @@ Every NIM in this chart pins an exact NGC image tag in `values.yaml` | Family | Example | Meaning | | ------ | ------- | ------- | | Plain semver | `nemotron-page-elements-v3:1.8.0` | A standard NIM release, identical bytes on every pull. Used by the four core NIMs and the reranker / ASR NIMs. | -| `-variant` | `nemotron-parse-v1.2:1.7.0-variant`, `nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant` | The Nemotron Parse and Nemotron 3 Nano Omni 30B builds that ship per-GPU TensorRT engine variants the NIM Operator selects from at reconciliation time (see the Omni and Parse rows in the [model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements) table). The `-variant` suffix is the NGC tag that ships alongside the 26.05 chart and matches footnote ³ of the support matrix. | +| `-variant` | `nemotron-parse-v1.2:1.7.0-variant`, `nemotron-3-nano-omni-30b-a3b-reasoning:1.7.0-variant` | The Nemotron Parse and Nemotron 3 Nano Omni 30B builds that ship per-GPU TensorRT engine variants the NIM Operator selects from at reconciliation time (refer to the Omni and Parse rows in the [model hardware requirements](https://github.com/NVIDIA/NeMo-Retriever/blob/main/docs/docs/extraction/prerequisites-support-matrix.md#model-hardware-requirements) table). The `-variant` suffix is the NGC tag that ships alongside the 26.05 chart and matches footnote ³ of the support matrix. | For air-gapped mirror pipelines: mirror the *exact* tag — both the plain semver and the `-variant` form — and do not substitute `:latest`. @@ -487,7 +490,7 @@ production rollout. **Charts and captioning (26.05).** Charts and infographics use **page_elements** and **ocr** (no `graphic_elements` operator NIM in this chart). For image -captioning, set `nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning.enabled=true` — see +captioning, set `nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning.enabled=true` — refer to [Image captioning (Omni 30B)](#image-captioning-omni-30b) for the chart-side wiring and [Image captioning (26.05)](https://docs.nvidia.com/nemo/retriever/latest/extraction/prerequisites-support-matrix/#image-captioning-2605) @@ -517,7 +520,7 @@ The service derives `caption_enabled=true` from a non-null `caption_invoke_url`, so the ingestion pipeline routes caption work to the in-cluster Omni Pod with no manual ConfigMap edits. -Resolution order mirrors every other NIM endpoint (see the +Resolution order mirrors every other NIM endpoint (refer to the [NIM Operator sub-stack](#nim-operator-sub-stack) section): 1. Explicit `serviceConfig.nimEndpoints.captionInvokeUrl` always wins @@ -742,7 +745,7 @@ dev clusters to free GPU memory for `vlm_embed`. Long batch jobs may exceed the gateway proxy timeout (300 s) or an Ingress `proxy-read-timeout`. Increase ingress annotations if clients disconnect -while workers are still processing; see the commented example on +while workers are still processing; refer to the commented example on `ingress.annotations` in `values.yaml`. --- @@ -752,7 +755,7 @@ while workers are still processing; see the commented example on In `topology.mode: split` deployments the realtime and batch worker pods scale horizontally based on **queue fill ratio** and **95th-percentile processing latency**. Both signals come straight out -of the pods' `/metrics` endpoint — the publisher is always on (see +of the pods' `/metrics` endpoint — the publisher is always on (refer to `nemo_retriever_pool_queue_depth_ratio` in [`prometheus.py`](../src/nemo_retriever/service/services/prometheus.py)). The only choice you have to make is **how the metrics get from @@ -939,9 +942,319 @@ sanity check before opening Grafana. --- +## OpenShift deployment { #openshift-deployment } + +The chart defaults target generic Kubernetes clusters that allow fixed numeric +UIDs (`runAsUser` / `runAsGroup` / `fsGroup` **1000**). **OpenShift 4.x** +namespaces under the default **restricted-v2** Security Context Constraint (SCC) +and **Pod Security Admission (PSA) `restricted`** profile assign a per-namespace +UID/GID range instead. A stock `helm install` without overrides therefore fails +SCC validation, emits PSA warnings, or crashes on log paths the random UID cannot +write. + +We do **not** change chart defaults for OpenShift-only behavior (that would affect +other platforms). Use the overrides below on OpenShift, or save the YAML block +into a local values file and pass `-f `. + +### Cluster posture (typical QA / hardened namespaces) + +| Control | Typical default on a new OpenShift project | +| --- | --- | +| SCC | **restricted-v2** (first match in priority order) | +| PSA | `pod-security.kubernetes.io/warn=restricted` (and often `audit=restricted`; `enforce` may be unset on dev clusters) | +| UID assignment | SCC injects `runAsUser` / `fsGroup` from the namespace range (for example `1000750000–1000759999`) | + +On clusters with **PSA `enforce=restricted`**, missing container `securityContext` +fields become hard rejections, not warnings. + +### Override reference (maps to chart limitations) + +| Symptom on stock install | Cause | Helm override | +| --- | --- | --- | +| `FailedCreate`: UID/GID **1000** not in namespace range | Hardcoded `service.podSecurityContext` UID/GID/fsGroup | Omit `runAsUser`, `runAsGroup`, and `fsGroup`; keep only `runAsNonRoot: true` | +| PSA warning: `allowPrivilegeEscalation`, capabilities, `seccompProfile` | Empty `service.securityContext` | Set restricted baseline on `service.securityContext` (refer to the sample below) | +| `PermissionError` on `/var/lib/nemo-retriever/retriever-service.log` when `persistence.enabled=false` | Default log path is image-owned; random UID cannot write without a PVC | Point `serviceConfig.logging.file` at `/tmp/...` (chart mounts `emptyDir` at `/tmp`) | +| `CreateContainerConfigError`: non-numeric image `USER nemo` on **vectordb** | Vectordb container has no `securityContext` block for SCC to annotate | Disable vectordb for smoke tests, or patch the vectordb Deployment after install (below) | +| PSA warnings on **otel-collector** | Otel Deployment has no `securityContext` in the chart | `topology.otel.enabled=false` unless you patch that Deployment | +| Audio/video fails or pod never gets `ffmpeg` | `service.installFfmpeg=true` runs sudo at startup; **restricted-v2** blocks privilege escalation (`no-new-privileges`) | Prebuild a service image with `ffmpeg`/`ffprobe` baked in (refer to [Audio and video on restricted OpenShift](#audio-and-video-ffmpeg-on-restricted-openshift)); leave `service.installFfmpeg=false` | +| `ImagePullBackOff` for a service image in the **internal OpenShift registry** | Chart-rendered `imagePullSecrets` may omit the namespace SA `dockercfg` secret required for internal-registry pulls | List every required pull secret under `imagePullSecrets` (refer to [Internal registry pull secrets](#internal-registry-pull-secrets)) | +| Optional NIM `CrashLoopBackOff` with missing `.so` in logs | GPU/CUDA libraries not on `LD_LIBRARY_PATH` for some NIM Operator stacks on OCP | Append paths through `nimOperator..env` (refer to [Optional NIM runtime environment](#optional-nim-runtime-environment)) | + +### Recommended value overrides + +Save the block below as a local values file (for example `openshift-restricted.yaml`) +and pass `-f openshift-restricted.yaml` on every `helm install` / `helm upgrade`. +This is the profile QA used on OpenShift 4.20 with the NIM Operator: core PDF +pipeline, Parakeet ASR (with a prebuilt `ffmpeg` service image), and Omni captioning +all reached Ready when combined with the NGC secrets and any NIM-specific env +overrides noted in later subsections. + +```yaml +# OpenShift overrides for nemo-retriever Helm chart (restricted-v2 / PSA restricted). +# helm install retriever ./nemo_retriever/helm -f openshift-restricted.yaml ... + +service: + podSecurityContext: + runAsNonRoot: true + # Do NOT set runAsUser, runAsGroup, or fsGroup — OpenShift SCC assigns them. + securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault + # Chart default is installFfmpeg: true; override to false on restricted-v2. + installFfmpeg: false + +serviceConfig: + logging: + # Writable without persistence PVC (chart always mounts emptyDir at /tmp). + file: /tmp/retriever-service.log + vectordb: + # Set false for minimal service-only validation; refer to the vectordb patch below if enabled. + enabled: false + +topology: + otel: + enabled: false +``` + +When **`persistence.enabled=true`**, you can keep the default log path under +`persistence.mountPath` (`/var/lib/nemo-retriever`) because the PVC is mounted and +SCC-assigned `fsGroup` applies. When persistence is off, always relocate logs to +`/tmp` (or another path backed by `service.extraVolumes`). + +### Audio and video (`ffmpeg`) on restricted OpenShift { #audio-and-video-ffmpeg-on-restricted-openshift } + +Elsewhere this README documents `service.installFfmpeg=true`, which installs +`ffmpeg`/`ffprobe` at container startup through passwordless `sudo`. On OpenShift +**restricted-v2** SCC, that path is blocked: PSA restricted sets +`allowPrivilegeEscalation: false` and the SCC enforces **no-new-privileges**, so +the entrypoint cannot elevate to install packages. + +For audio and video extraction on OpenShift, **do not** set `service.installFfmpeg=true`. +Instead, extend the service image on a connected build host and point the chart at +that tag (same pattern as [air-gapped custom images](#1-service-image)): + +```dockerfile +FROM nvcr.io/nvstaging/nim/nrl-service: +USER root +RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg \ + && rm -rf /var/lib/apt/lists/* +USER nemo +``` + +Push the result to NGC, your private registry, or the +[OpenShift internal registry](#internal-registry-pull-secrets), then set +`service.image.repository` / `service.image.tag` accordingly. + +### Internal registry pull secrets { #internal-registry-pull-secrets } + +When you push the service image to the OpenShift internal registry +(`image-registry.openshift-image-registry.svc:5000/...`), pods need credentials +for that registry **and** for NGC NIM images. The chart renders an explicit +`imagePullSecrets` list on every Pod. List every secret each Pod needs—the +namespace ServiceAccount `dockercfg` secret when `service.image` uses the +internal registry, plus `ngc-secret` when in-cluster NIMs pull from NGC. + +If the rendered list contains only `ngc-secret` while `service.image` uses the +internal registry, the service Pod can fail with `ImagePullBackOff`. + +**Internal-registry service image only** — clear the chart-managed NGC pull secret +name so the helper does not inject `ngc-secret` alone: + +```yaml +ngcImagePullSecret: + create: false + name: "" # Explicitly empty — clears the default "ngc-secret" + +imagePullSecrets: + - name: default-dockercfg-xxxxx # replace with your SA secret (below) +``` + +**Both NGC NIMs and an internal-registry service image** — list every secret the +pods need: + +```yaml +ngcImagePullSecret: + create: false + name: ngc-secret + +imagePullSecrets: + - name: ngc-secret + - name: default-dockercfg-xxxxx +``` + +Look up the ServiceAccount pull secret name: + +```bash +oc get sa default -n nemo-retriever -o jsonpath='{.imagePullSecrets[*].name}{"\n"}' +``` + +Pre-create `ngc-secret` / `ngc-api` with `oc create secret` when +`ngcImagePullSecret.create=false` and `ngcApiSecret.create=false`, as in the +install examples below. + +### Optional NIM runtime environment { #optional-nim-runtime-environment } + +On OpenShift 4.20 with the NIM Operator, QA found that some **optional** NIM pods +(for example Parakeet ASR or Omni caption) needed an explicit `LD_LIBRARY_PATH` +so CUDA/driver libraries from the GPU Operator stack were visible at runtime. +Symptoms are `CrashLoopBackOff` and log lines referencing a missing `.so` (for +example `libcudart` or `libcudnn`). + +Helm **replaces** the whole `env` list when you override `nimOperator..env` +in a values file. Copy the chart defaults from `values.yaml` for that NIM, then +append `LD_LIBRARY_PATH`: + +```yaml +nimOperator: + audio: + env: + - name: NIM_TAGS_SELECTOR + value: "name=parakeet-1-1b-ctc-en-us,mode=ofl,vad=default,diarizer=disabled" + - name: NIM_TRITON_LOG_VERBOSE + value: "1" + - name: LD_LIBRARY_PATH + value: "/usr/local/nvidia/lib64:/usr/local/cuda/lib64" + nemotron_3_nano_omni_30b_a3b_reasoning: + env: + - name: NIM_HTTP_API_PORT + value: "8000" + - name: NIM_TRITON_LOG_VERBOSE + value: "1" + - name: LD_LIBRARY_PATH + value: "/usr/local/nvidia/lib64:/usr/local/cuda/lib64" +``` + +Paths vary with GPU Operator version and node image. Inspect a healthy GPU +workload on the same cluster (`oc exec` into a CUDA sample pod) or the failing +NIM pod's filesystem before pinning production values. + +### Omni caption manual smoke testing { #omni-caption-manual-smoke-testing } + +The retriever service caption profile already sends +`chat_template_kwargs.enable_thinking=false` to the Omni NIM during ingest. When +you call the Omni NIM **directly** at `/v1/chat/completions` (for example with +`curl` against the in-cluster `NIMService`), include the same flag so the +caption text lands in `message.content` instead of reasoning-only fields: + +```json +{ + "model": "nvidia/nemotron-3-nano-omni-30b-a3b-reasoning", + "messages": [{"role": "user", "content": "..."}], + "chat_template_kwargs": {"enable_thinking": false} +} +``` + +Refer to [Image captioning](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/docs/docs/extraction/prerequisites-support-matrix.md#image-captioning-2605) +for when to enable the caption stage and which Helm key to use. + +### Example install on OpenShift 4.20 (service-only smoke test) + +Matches early QA repro with external NIMs disabled, no persistence, and no +results PVC: + +```bash +oc new-project nemo-retriever + +oc create secret docker-registry ngc-secret -n nemo-retriever \ + --docker-server=nvcr.io --docker-username='$oauthtoken' \ + --docker-password="$NGC_API_KEY" + +oc create secret generic ngc-api -n nemo-retriever \ + --from-literal=NGC_API_KEY="$NGC_API_KEY" \ + --from-literal=NGC_CLI_API_KEY="$NGC_API_KEY" + +helm install retriever ./nemo_retriever/helm -n nemo-retriever \ + -f openshift-restricted.yaml \ + --set ngcImagePullSecret.create=false \ + --set ngcApiSecret.create=false \ + --set nims.enabled=false \ + --set persistence.enabled=false \ + --set retrieverResults.enabled=false \ + --set service.image.repository=nvcr.io/nvstaging/nim/nrl-service \ + --set service.image.tag= +``` + +Verify pods: + +```bash +oc get pods -n nemo-retriever +oc describe pod -l app.kubernetes.io/name=nemo-retriever -n nemo-retriever +``` + +You should see SCC-assigned numeric `runAsUser` on containers that declare a +`securityContext` block, and no PSA warnings after you apply overrides. + +### Example install with NIM Operator (full pipeline validation) + +After NIM Operator and GPU Operator are installed, reuse `openshift-restricted.yaml` +and pre-created NGC secrets. Point `service.image` at a **ffmpeg-enabled** build +when validating [audio and video](#audio-and-video-ffmpeg-on-restricted-openshift). +Add [optional NIM `LD_LIBRARY_PATH`](#optional-nim-runtime-environment) overrides +if ASR or Omni pods crash on missing libraries. + +```bash +helm install retriever ./nemo_retriever/helm -n nemo-retriever \ + -f openshift-restricted.yaml \ + --set ngcImagePullSecret.create=false \ + --set ngcApiSecret.create=false \ + --set service.image.repository=nvcr.io/nvstaging/nim/nrl-service \ + --set service.image.tag= +``` + +QA validated this posture on OpenShift 4.20 with default PDF, Parakeet ASR, and +Omni caption smoke tests passing when the overrides above and any per-NIM env +fixes were applied. + +### Enabling the vectordb Deployment on OpenShift + +`serviceConfig.vectordb.enabled=true` renders a **vectordb** container from the +same image (`USER nemo`, non-numeric). The chart does not yet expose a +`securityContext` value for that container. After `helm install`, patch the +Deployment so OpenShift can inject a numeric UID into the container spec: + +```bash +RELEASE=retriever +NS=nemo-retriever +VDB_DEPLOY="${RELEASE}-nemo-retriever-vectordb" + +oc patch deployment "$VDB_DEPLOY" -n "$NS" --type=json -p='[ + {"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": { + "allowPrivilegeEscalation": false, + "capabilities": {"drop": ["ALL"]}, + "runAsNonRoot": true, + "seccompProfile": {"type": "RuntimeDefault"} + }} +]' +``` + +Re-apply the patch after `helm upgrade` if the Deployment is recreated. A future +chart release may add first-class `topology.vectordb.securityContext` values. + +### Enabling the OpenTelemetry collector on OpenShift + +The chart’s otel-collector Deployment likewise lacks `securityContext` fields. +Prefer `topology.otel.enabled=false` (as in the sample values) unless you operate +your own collector or patch `*-otel` the same way as vectordb. + +### What we intentionally do not require on OpenShift + +Do **not** bind the namespace to **anyuid** SCC or set PSA `enforce=privileged` +unless your security team explicitly approves it. The overrides above are intended +to keep **restricted-v2** / PSA **restricted** posture. + +### Related documentation + +- [Pre-Requisites & Support Matrix](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/docs/docs/extraction/prerequisites-support-matrix.md) +- [Deployment options](https://github.com/NVIDIA/NeMo-Retriever/blob/26.05/docs/docs/extraction/deployment-options.md) + +--- ## Air-gapped deployment { #air-gapped-deployment } -See [Deployment options — Air-gapped and disconnected deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/deployment-options/#air-gapped-deployment) for overview and workflow. Chart-specific reference for mirroring: +Refer to [Deployment options — Air-gapped and disconnected deployment](https://docs.nvidia.com/nemo/retriever/latest/extraction/deployment-options/#air-gapped-deployment) for overview and workflow. Chart-specific reference for mirroring: ### Container images to mirror (26.05 chart defaults) @@ -1006,7 +1319,7 @@ nimOperator: outside the chart. - For **offline captioning**, enable `nimOperator.nemotron_3_nano_omni_30b_a3b_reasoning` and point the pipeline - caption endpoint at the in-cluster NIM URL (see + caption endpoint at the in-cluster NIM URL (refer to [Image captioning (26.05)](https://docs.nvidia.com/nemo/retriever/latest/extraction/prerequisites-support-matrix/#image-captioning-2605)). ### Mirroring pattern @@ -1062,7 +1375,7 @@ helm template r nemo_retriever/helm \ ``` Both renders should succeed cleanly and parse as valid Kubernetes manifests -(`kubectl apply --dry-run=client -f /tmp/r.yaml`). See [VectorDB and the +(`kubectl apply --dry-run=client -f /tmp/r.yaml`). Refer to [VectorDB and the embed endpoint](#vectordb-and-the-embed-endpoint) for why `helm template r nemo_retriever/helm` without flags is rejected as a misconfiguration.