Skip to content

Taxonomy service should use an image volume to get DB #7149

Description

@maverbiest

When reviewing #6916 I found the image volume approach of making the Deacon index available in the raw-reads processing pod very nice:

- name: Publish deacon index as an OCI image for the raw-reads-processing image volume
run: |
# Push under a stable filename ("deacon.idx") regardless of the dated
# build filename, so the mounted path in the deployment never changes.
#
# This must be a real container image (FROM scratch + COPY), not a bare
# OCI artifact (e.g. `oras push --artifact-type ...`): Kubernetes'
# ImageVolume / containerd only support images with a proper image
# config (rootfs.diff_ids matching the layers). Artifacts with an empty
# config fail to mount with "mismatched image rootfs and manifest layers".
# See https://github.com/containerd/containerd/issues/11907
cp "${INDEX_VERSIONED}" deacon.idx
cat <<'EOF' > Dockerfile.deacon-index
FROM scratch
COPY deacon.idx /deacon.idx
EOF
docker buildx build \
--push \
--tag "${DEACON_INDEX_IMAGE}:${DATE_TAG}" \
--tag "${DEACON_INDEX_IMAGE}:latest" \
-f Dockerfile.deacon-index .

I think we could also use this for making the taxonomy DB available to the taxonomy service. There, we currently run an init container to download the DB from S3:

initContainers:
- name: download-taxonomy-db
image: alpine:3
command: ["sh", "-c"]
args:
- |
set -e
DB_URL="https://loculus-public.hel1.your-objectstorage.com/taxonomy/{{ .Values.taxonomyService.dbVersion }}.sqlite.gz"
echo "Downloading taxonomy DB from $DB_URL"
wget -q "$DB_URL" -O /data/tmp.sqlite.gz
gunzip /data/tmp.sqlite.gz
mv /data/tmp.sqlite /data/taxonomy.sqlite
echo "Taxonomy DB ready"
volumeMounts:
- name: taxonomy-db
mountPath: /data

From reading a bit, it seems the image volume might have much more preferable caching behabiour at the node level compare to the init container, which makes it so we don't have to download the taxonomy DB (~3GiB) every time we start a new taxonomy service.

Required changes

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    deploymentCode changes targetting the deployment infrastructure

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions