Trending questions in Apache Spark

0 votes
1 answer

what are the spark real time issues ?

Some of the issues I have faced ...READ MORE

Mar 18, 2019 in Apache Spark by Sharman
 • 5,355 views
0 votes
1 answer

What does reduce action do in Spark?

Hey, Reduce action converts an RDD to a ...READ MORE

Jul 4, 2019 in Apache Spark by Gitika
 • 65,770 points • 647 views
0 votes
1 answer

What is persist() in Spark?

Hi, Spark’s RDDs are by default recomputed each ...READ MORE

Jul 2, 2019 in Apache Spark by Gitika
 • 65,770 points • 720 views
0 votes
1 answer

How to print loop with condition in scala?

Hi, Yes, in scala there is a guard condition where ...READ MORE

Jul 6, 2019 in Apache Spark by Gitika
 • 65,770 points • 570 views
0 votes
1 answer

By which components spark ecosystem libraries are composed of?

Hi, Spark ecosystem libraries are composed of various ...READ MORE

Jul 1, 2019 in Apache Spark by Gitika
 • 65,770 points • 659 views
0 votes
1 answer

Spark CLI issue

For spark.read.textFile we need spark-2.x. Please try ...READ MORE

Jun 19, 2019 in Apache Spark by Maahi
 • 760 views
+1 vote
1 answer

map vs mapValues in Spark

There is a difference between the two: mapValues ...READ MORE

Jun 29, 2018 in Apache Spark by nitinrawat895
 • 11,380 points • 16,035 views
0 votes
1 answer

Doubt in display(id, name, salary) before display function

The statement display(id, name, salary) is written before the display function ...READ MORE

Jun 19, 2019 in Apache Spark by Ritu
 • 665 views
0 votes
1 answer

How to store files in executor's working directory?

You have to specify a comma-separated list ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 4,209 views
0 votes
1 answer

How to set extra JVM options for Spark application?

You cans set extra JVM options that ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 3,964 views
0 votes
1 answer

How to set cpu cores for spark task?

By default, each task is allocated with ...READ MORE

Mar 12, 2019 in Apache Spark by Veer
 • 4,418 views
0 votes
1 answer

How to change the location of Spark event logs?

You can change the location where you ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 4,528 views
0 votes
1 answer

Spark2-submit does not generate output file.

To generate the output file, you can ...READ MORE

Feb 24, 2019 in Apache Spark by Esha
 • 4,864 views
0 votes
1 answer

Starting Spark in Windows

Run below commands spark-class org.apache.spark.deploy.master.Master spark-class org.apache.spark.deploy.worker.Worker spark://192.168.254.1:7077 NOTE: The ...READ MORE

May 22, 2019 in Apache Spark by Reshma
 • 953 views
0 votes
1 answer

Starting Spark Scala console

To get command prompt for Scala open ...READ MORE

May 24, 2019 in Apache Spark by Cassy
 • 692 views
0 votes
1 answer

How to get Spark SQL configuration?

First create a Spark session like this: val ...READ MORE

Mar 18, 2019 in Apache Spark by John
 • 3,561 views
0 votes
1 answer

What is Executor Memory in a Spark application?

Every spark application has same fixed heap ...READ MORE

Jan 5, 2019 in Apache Spark by Frankie
 • 9,830 points • 6,504 views
0 votes
1 answer

SparkContext.addFile() not able to update file.

Spark by default won't let you overwrite ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 3,593 views
0 votes
3 answers

Can anyone explain fold() operation in Spark?

Fold in spark Fold is a very powerful ...READ MORE

Aug 23, 2018 in Apache Spark by samarth295
 • 2,220 points • 12,656 views
0 votes
1 answer

What is Spark Core?

It is not like a CPU to ...READ MORE

Mar 8, 2019 in Apache Spark by Raj
 • 3,519 views
0 votes
1 answer

How to enable worker cleanup in Spark?

To enable cleanup, open the spark shell ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 2,686 views
0 votes
1 answer

Increase Yarn wait time for Sparkcontext

The default time that the Yarn application waits ...READ MORE

Mar 27, 2019 in Apache Spark by Rohit
 • 2,628 views
0 votes
1 answer

How to change commiter algorithm version in Spark?

To change to version 2, run the ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 3,314 views
0 votes
1 answer

Disable Web UI for Spark Application

You can disable it like this: val sc ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 3,449 views
0 votes
1 answer

How can we use spark shell for scala without cluster?

You can run the Spark shell for ...READ MORE

Apr 28, 2019 in Apache Spark by Giri
 • 727 views
0 votes
1 answer

Set archives to be extracted in executor directory

I don't think you can copy and ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 2,030 views
0 votes
1 answer

How to enable SSL for Spark application?

You can do it dynamically like this: val ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 2,559 views
0 votes
1 answer

Spark Yarn: Changing maximum number of time to submit application

By default, the maximum number of times ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 1,991 views
0 votes
1 answer

How to enable Spark event logging?

To make Spark store the event logs, ...READ MORE

Mar 6, 2019 in Apache Spark by Rohit
 • 2,895 views
0 votes
1 answer

Need help setting Spark yarn history server address

If you are running history server and ...READ MORE

Mar 27, 2019 in Apache Spark by Neha
 • 1,990 views
0 votes
1 answer

what are the job optimization Technics in spark and scala ?

There are different methods to achieve optimization ...READ MORE

Mar 18, 2019 in Apache Spark by Veer
 • 2,319 views
0 votes
1 answer

How to make Spark wait for more time for acknowledgement?

Use the following command to increase the ...READ MORE

Mar 11, 2019 in Apache Spark by Raj
 • 2,494 views
0 votes
1 answer

How to authenticate Spark internal connections using a secret key?

You need to set the secret key ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 2,370 views
0 votes
1 answer

How to disable executor from fetching file from cache?

When a Spark application is running, the ...READ MORE

Mar 10, 2019 in Apache Spark by Siri
 • 2,490 views
0 votes
1 answer

Apache Spark, usage of yield.

Yield is used in sequence comprehensions. It is ...READ MORE

Feb 22, 2019 in Apache Spark by Saruj
 • 3,183 views
0 votes
1 answer

How to set keystore path?

You have to set the path to ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 2,227 views
0 votes
1 answer

How do spark extra listeners work?

Yes. You can use extra listeners by setting ...READ MORE

Feb 24, 2019 in Apache Spark by Rishi
 • 3,067 views
0 votes
2 answers

In a Spark DataFrame how can I flatten the struct?

// Collect data from input avro file ...READ MORE

Jul 4, 2019 in Apache Spark by Dhara dhruve
 • 6,110 views
0 votes
1 answer

How to change scheduling mode in Spark?

You can change the scheduling mode as ...READ MORE

Mar 12, 2019 in Apache Spark by Raj
 • 2,319 views
0 votes
0 answers

what are the memory issues in spark ?

Mar 18, 2019 in Apache Spark by satish kumar
 • 180 points • 2,073 views
0 votes
1 answer

How to use Spark jars for Yarn distribution?

First, store upload this archive to hdfs and ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 1,518 views
0 votes
1 answer

How to set executors for static allocation in Spark Yarn?

Open Spark shell and run the following ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 1,494 views
0 votes
1 answer

How to increase HDFS replication level in Spark?

Hi @Raunak. You can change the replication ...READ MORE

Mar 27, 2019 in Apache Spark by Yash
 • 1,534 views
0 votes
1 answer

How to access private key password with Spark?

Spark allows you to retrieve the key ...READ MORE

Mar 15, 2019 in Apache Spark by Karan
 • 1,902 views
0 votes
1 answer

How to connect to Zookeeper after setting Spark recovery mode?

You have set Zookeeper as the recovery ...READ MORE

Mar 25, 2019 in Apache Spark by Hari
 • 1,399 views
0 votes
1 answer

Increasing memory to use for Yarn application master?

You can increase the memory dynamically by ...READ MORE

Mar 26, 2019 in Apache Spark by Tina
 • 1,311 views
0 votes
1 answer

How to set max executors for dynamic allocation?

You can set it by assigning the ...READ MORE

Mar 13, 2019 in Apache Spark by Venu
 • 1,855 views
0 votes
1 answer

How to use ftp scheme using Yarn in Spark application?

In case Yarn does not support schemes ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 1,197 views
0 votes
1 answer

Thread to use Yarn application master is limited

This is because the maximum number of ...READ MORE

Mar 28, 2019 in Apache Spark by Raj
 • 1,185 views
0 votes
1 answer

Increase cores for yarn in Spark application

By default, only one core is used for ...READ MORE

Mar 26, 2019 in Apache Spark by Bhuvan
 • 1,253 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.