Set maximum reading rate from Kafka partition

Question

I want to set the maximum rate at which data is read from Kafka partition just to restrict the resources being utilized for this. How can I set the Maximum rate?

Answer 1

You can do it by using the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.streaming.kafka.maxRatePerPartition= <max rate>

This will restrict the reading rate to the rate set for this property.