First steps with Apache Spark on K8s (part #2): standalone Spark auto scaling with HPA

#1.git clone https://github.com/domasjautakis/standalone-spark-on-k8s.git
  • Main Dockerfile for master/worker container
  • Deployments for master and worker
  • Services — network service for Spark Web UI
  • Spark Proxy UI — which will allow us to get access to it outside of container. Module written by Alexis Seigneurin, more: github
#2.cd standalone-spark-on-k8s/
docker build -f Dockerfile -t spark3:v1 .
#3.docker image ls
#4.cd spark-ui-proxy/
docker build -f Dockerfile -t spark3proxy:v1 .
docker image ls
#5.kubectl apply -f ../services/spark-master-service.yaml
kubectl apply -f ../deployments/spark-master-deployment.yaml
sleep 5
kubectl apply -f ../deployments/spark-worker-deployment.yaml
sleep 5
kubectl apply -f ../deployments/spark-ui-proxy/service.yaml
kubectl apply -f ../deployments/spark-ui-proxy/deployment.yaml
#6.kubectl get po
kubectl get svc
minikube ip
#7. kubectl get deployments
kubectl autoscale deployment spark-worker --max=6 --cpu-percent=10
kubectl get hpa
#8. kubectl exec -it "$(kubectl get pods | grep "spark-worker" | cut -d ' ' -f 1)" -- /bin/bash
spark-submit --master spark://spark-master:7077 --conf spark.shuffle.service.enabled=true --conf spark.dynamicAllocation.enabled=true --class org.apache.spark.examples.SparkPi /opt/spark/examples/jars/spark-examples_2.12-3.0.2.jar 100000
#9.kubectl get hpa

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store