Spark Cluster on Kubernetes

Builds a custom Spark image and deploys a Spark standalone cluster (1 master, 3 workers) on Kubernetes (k3s).

Prerequisites

• docker logged into docker-registry.yacodata.com
• kubectl pointing at the cluster
• The image registry must be reachable by the cluster (add it as a mirror in /etc/rancher/k3s/registries.yaml if it is HTTP-only)

1. Build and push the image

./build-push-image.sh

Builds Docker/ and pushes docker-registry.yacodata.com/custom-spark:1.0.8-jdk-slim-buster.

2. Create the registry pull secret

The deployments reference the secret regcred via imagePullSecrets. Create it from literals before running the deploy script:

kubectl create secret docker-registry regcred -n spark \
  --docker-server=docker-registry.yacodata.com \
  --docker-username=$REGISTRY_USER \
  --docker-password=$REGISTRY_PASS \
  --docker-email=YOUR_EMAIL

3. Deploy the cluster

./deploy-k8s.sh

Creates the spark namespace, applies PVC, services, deployments, then waits for rollout and prints cluster status.

Local access

kubectl port-forward svc/spark-master 8080:8080 -n spark   # Spark master web UI
kubectl port-forward svc/spark-master 7077:7077 -n spark   # Spark master protocol

Notes

• Workers wait (DNS + TCP retry loop) for the master before joining.
• The PVC uses ReadWriteOnce with the default local-path storage class — this works on a single-node cluster. For multi-node, switch to an RWX-capable class (e.g. Longhorn, NFS, CephFS).
• No ingress is configured; the master service is ClusterIP only.