Viewing posts for the category Spark

How to build Spark source code and run its built-in example in IntelliJ

One way to dig and debug into the Spark's codebase is to build Spark source code and run its built-in examples (e.g., LogQuery and GroupByTest) in IntelliJ. This gives you full control over Spark source code. For example, you can hack into the codebase such as adding your code on top of Spark, adding breakpoints and observing code behavior. However, setting up Spark sourcecode and running its example in IntelliJ is a little bit tricky. This tutorial shows you how to build Spark source code and run its LogQuery example in local mode in IntelliJ.