How to increase Spark memory for execution

Question

I need to increase the memory for execution and storage because I am facing a lot of spills. How can I do it?

Answer 1

Probably the spill is because you have less memory allocated for execution. You can increase this as follows:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.memory.fraction=0.7