All the code for this is available on GitHub. 3 nodes with 2GB ram and 1 CPU will cost about $30 per month.

Let’s solve that problem by setting up our own Spark/Hadoop cluster and doing some “real” data things with it.įirstly, I used a 3 node setup on Linode of course. Also, very annoyingly it seems most videos/posts on Spark about shuffling/joins blah blah that make no sense to someone who doesn’t use Spark on daily basis, or they are so “Hello World” as to be useless in the real world. Granted, most people never work on datasets large enough to warrant the use of Spark. With the continued rise os AWS Glue and GCP DataProc, running Spark scripts without managing a cluster has never been easier.

I’ve written about Spark a few times, even 2 years ago, but it still seems to be steadily increasing in popularity, albeit still missing from many companies tech stacks. I’ve never seen so many posts about Apache Spark before, not sure if it’s 3.0, or because the world is burning down.

Install spark locally