Spark job using kubernetes instead of yarn

Question

I am writing a spark job which uses kubernetes instead of yarn.

val spark = SparkSession.builder().appName("Demo").master(????).getOrCreate() 

What should the master part be?

Answer 1

I should be in the format of k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>

In your case use it in the following way:

val spark = SparkSession.builder().appName("Demo").master(k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>).getOrCreate()