All you will read here is personal opinion or lack of knowledge :) Please feel free to contact me for fixing incorrect parts.

In previous blogposts I have described how to setup environment for Kubernetes and how to auto scale Standalone Spark. In this post let’s have a look on GCP Spark Operator. Join my journey with K8s and Spark! Operator versions shown below:


All you will read here is personal opinion or lack of knowledge :) Please feel free to contact me for fixing incorrect parts.

If you’re in this blogpost — most probably you know what Apache Spark and K8s is. Probably the most exciting feature which is possible under Kubernetes is auto scaling which allows to utilize resources more efficient. It’s possible to scale Spark in different manner, but most probably the most known is HPA. The more details can be found at: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/


All you will read here is personal opinion or lack of knowledge :) Please feel free to contact me for fixing incorrect parts.

In order to start working with Spark on Kubernetes first step is to prepare sandbox. In my case I’m using Windows 10 based machine with Hyper-V. For this sandbox I can dedicate 16 Gigs and 8 threads.

All scripts from this article can be found at: github.com/domasjautakis/docker-minikube


All you will read here is personal opinion or lack of knowledge :) Please feel free to contact me for fixing incorrect parts.

As data engineer who is passionated about Apache Spark I decided to compare different and similar open-source projects like Delta, Hudi and Iceberg. The idea is simple: prepare environment for all three technologies and compare them from Apache Spark and consumption perspectives including Hive and Presto. e. Topic is splitted in three parts:

  • Preparation of single node cluster including: Hadoop, Spark, Hive, Presto and all dependencies.
  • Test how Delta, Hudi, Iceberg will behave with updates, deletes, time-travels…

Domas Jautakis

Data Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store