How to change commiter algorithm version in Spark

Question

I am using version 1 committer algorithm and I read in the docs that using version 2 might make the performance better. Can someone tell me how to use version 2 instead of 1?

Answer 1

To change to version 2, run the following command in the Spark shell:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2