Close file after writing write-ahead on driver

Question

I am S3 file system and it does not support flushing. I want the Spark application to close the file after writing the write-ahead log on the driver. Please tell me how to do it?

Answer 1

You can change the property to close the file after writing the write-ahead logs. You can do this from the Spark-shell. Refer to the below command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.streaming.driver.writeAheadLog.closeFileAfterWrite=true