How to retain Spark jar and app jar after staging

Question

Hi. I have some staging function going on in my Spark application. The problem is that the Spark jar and app jar files are automatically getting deleted after staging. I want to retain these files to study them and learn as I am new to this concept. Can someone tell me how to do this?

Answer 1

By default, Spark jar, app jar, and cache files are configured to be deleted after the job. If you want to preserve these files you will have to change the configuration. You can do it using the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.preserve.staging.files=true