All you will read here is personal opinion or lack of knowledge :) Please feel free to contact me for fixing incorrect parts.

If you’re in this blogpost — most probably you know what Apache Spark and K8s is. Probably the most exciting feature which is possible under Kubernetes is auto scaling which allows to utilize resources more efficient. It’s possible to scale Spark in different manner, but most probably the most known is HPA. The more details can be found at: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

All you will read here is personal opinion or lack of knowledge :) Please feel free to contact me for fixing incorrect parts.

In order to start working with Spark on Kubernetes first step is to prepare sandbox. In my case I’m using Windows 10 based machine with Hyper-V. For this sandbox I can dedicate 16 Gigs and 8 threads.

All scripts from this article can be found at: github.com/domasjautakis/docker-minikube

All you will read here is personal opinion or lack of knowledge :) Please feel free to contact me for fixing incorrect parts.

As data engineer who is passionated about Apache Spark I decided to compare different and similar open-source projects like Delta, Hudi and Iceberg. The idea is simple: prepare environment for all three technologies and compare them from Apache Spark and consumption perspectives including Hive and Presto. e. Topic is splitted in three parts:

• Preparation of single node cluster including: Hadoop, Spark, Hive, Presto and all dependencies.
• Test how Delta, Hudi, Iceberg will behave with updates, deletes, time-travels…