Not able to preserve shuffle files in Spark

Question

I am trying to execute MapReduce with Spark. I am facing a problem with the shuffle. I am using executors for this and when I remove the executors, I am losing all the shuffle files. Please help.

Answer 1

You lose the files because by default, there is no setting to save files created by external executors. To save the files even after removing the executors, you will have to change the configuration. The property for this is spark.shuffle.service.enabled and the command to save files even after the executor is removed will be like this:

./bin/spark-submit <all your existing options> --conf spark.shuffle.service.enabled=true