Need help setting Spark yarn history server address

Question

I need some help with setting the history server address of Yarn in Spark. I am following some steps to use the history server. The steps tell that after enabling the history server, I have to set the address for it. I don't know how to do it and what address to set it to. Please help.

P.S.: I want to run history server on same node as Yarn

Answer 1

If you are running history server and Yarn on the same node, then it is suggested to set the address to yarn.resourcemanager.hostname}:18080. You can do it as follows:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.historyServer.address=yarn.resourcemanager.hostname}:18080