How to make driver update metrics quickly to executor

Question

Hi. I have a Spark application running and there some tasks in progress. I want the driver to update the metrics faster to the executor. How can I do this?

Answer 1

There's a heartbeat signal sent to the driver to inform it that the executor is still alive and then the driver updates the metrics. To update the metrics, you can change the time interval of the heartbeat. By default, it set to 10s. To change it, refer to the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.executor.heartbeatInterval=<time in seconds>