When reviewing #6916 I found the image volume approach of making the Deacon index available in the raw-reads processing pod very nice:
|
- name: Publish deacon index as an OCI image for the raw-reads-processing image volume |
|
run: | |
|
# Push under a stable filename ("deacon.idx") regardless of the dated |
|
# build filename, so the mounted path in the deployment never changes. |
|
# |
|
# This must be a real container image (FROM scratch + COPY), not a bare |
|
# OCI artifact (e.g. `oras push --artifact-type ...`): Kubernetes' |
|
# ImageVolume / containerd only support images with a proper image |
|
# config (rootfs.diff_ids matching the layers). Artifacts with an empty |
|
# config fail to mount with "mismatched image rootfs and manifest layers". |
|
# See https://github.com/containerd/containerd/issues/11907 |
|
cp "${INDEX_VERSIONED}" deacon.idx |
|
cat <<'EOF' > Dockerfile.deacon-index |
|
FROM scratch |
|
COPY deacon.idx /deacon.idx |
|
EOF |
|
docker buildx build \ |
|
--push \ |
|
--tag "${DEACON_INDEX_IMAGE}:${DATE_TAG}" \ |
|
--tag "${DEACON_INDEX_IMAGE}:latest" \ |
|
-f Dockerfile.deacon-index . |
I think we could also use this for making the taxonomy DB available to the taxonomy service. There, we currently run an init container to download the DB from S3:
|
initContainers: |
|
- name: download-taxonomy-db |
|
image: alpine:3 |
|
command: ["sh", "-c"] |
|
args: |
|
- | |
|
set -e |
|
DB_URL="https://loculus-public.hel1.your-objectstorage.com/taxonomy/{{ .Values.taxonomyService.dbVersion }}.sqlite.gz" |
|
echo "Downloading taxonomy DB from $DB_URL" |
|
wget -q "$DB_URL" -O /data/tmp.sqlite.gz |
|
gunzip /data/tmp.sqlite.gz |
|
mv /data/tmp.sqlite /data/taxonomy.sqlite |
|
echo "Taxonomy DB ready" |
|
volumeMounts: |
|
- name: taxonomy-db |
|
mountPath: /data |
From reading a bit, it seems the image volume might have much more preferable caching behabiour at the node level compare to the init container, which makes it so we don't have to download the taxonomy DB (~3GiB) every time we start a new taxonomy service.
Required changes
When reviewing #6916 I found the image volume approach of making the Deacon index available in the raw-reads processing pod very nice:
loculus/.github/workflows/deacon-index.yml
Lines 58 to 78 in 7d48300
I think we could also use this for making the taxonomy DB available to the taxonomy service. There, we currently run an init container to download the DB from S3:
loculus/kubernetes/loculus/templates/taxonomy-deployment.yaml
Lines 26 to 41 in 7d48300
From reading a bit, it seems the image volume might have much more preferable caching behabiour at the node level compare to the init container, which makes it so we don't have to download the taxonomy DB (~3GiB) every time we start a new taxonomy service.
Required changes