How to increase wait time to launch data-local task

0 votes
I am running long tasks and I see poor locality. I was doing some research and found that this can be optimized using locality wait. But the problem is that I don't know how to increase it. Please help.
Mar 11, 2019 in Apache Spark by Kara
 • 551 views

1 answer to this question.

0 votes

You can increase the locality wait time using the following command:

val sc = new SparkContext(new SparkConf())
./bin/spark-submit <all your existing options> --spark.locality.wait= <time in seconds>
answered Mar 11, 2019 by Raj

Related Questions In Apache Spark

0 votes
1 answer

How to increase the amount of data to be transferred to shuffle service at the same time?

The amount of data to be transferred ...READ MORE

answered Mar 1, 2019 in Apache Spark by Omkar
 • 69,220 points • 1,000 views
0 votes
1 answer

How to get ID of a map task in Spark?

you can access task information using TaskContext: import org.apache.spark.TaskContext sc.parallelize(Seq[Int](), ...READ MORE

answered Nov 20, 2018 in Apache Spark by Frankie
 • 9,830 points • 3,456 views
0 votes
1 answer

How to increase Spark memory for execution?

Probably the spill is because you have ...READ MORE

answered Mar 7, 2019 in Apache Spark by Pavitra
edited Mar 8, 2019 • 1,174 views
0 votes
1 answer

How to increase Garbage Collection speed?

The time interval between Garbage Collection is ...READ MORE

answered Mar 8, 2019 in Apache Spark by Pavitra
 • 1,635 views
+1 vote
2 answers

How do I get number of columns in each line from a delimited file??

Instead of spliting on '\n'. You should ...READ MORE

answered Aug 7, 2019 in Apache Spark by ashish
 • 5,524 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
 • 11,380 points • 11,028 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
 • 11,380 points • 2,536 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
 • 11,380 points • 108,832 views
0 votes
1 answer

How to make Spark wait for more time for acknowledgement?

Use the following command to increase the ...READ MORE

answered Mar 11, 2019 in Apache Spark by Raj
 • 2,494 views
0 votes
1 answer

How to set time for task speculation?

By default, the check for task speculation ...READ MORE

answered Mar 12, 2019 in Apache Spark by Veer
 • 550 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.