You can dynamically set a password to protect your private key. To set password dynamically, run this command in the Spark shell:
val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.ssl.keyPassword=<password>