Recent questions tagged pyspark

0 votes
1 answer

How can I import Elastic-Net, Lasso and Ridge regression in Pyspark?

Apr 5, 2022 in Machine Learning by Nandini
 • 5,480 points • 1,954 views
0 votes
1 answer

How to read a dataframe based on an avro schema?

Oct 30, 2020 in Apache Spark by anonymous
 • 120 points • 3,300 views
0 votes
1 answer

How to implement my clustering algorithm in pyspark (without using the ready library for example k-means)?

Oct 14, 2020 in Apache Spark by dani
 • 160 points • 1,700 views
0 votes
1 answer

I want to read .csv file from S3 and load/write the same data to cassandra.

Sep 4, 2020 in AWS by Raj
 • 120 points • 4,833 views
0 votes
1 answer

how to run spark job from EC2 to EMR?

Jun 25, 2020 in Apache Spark by Edureka
 • 120 points • 2,484 views
0 votes
1 answer

Can number of Spark task be greater than the executor core?

Jun 17, 2020 in Apache Spark by Rishi
 • 160 points • 2,075 views
0 votes
1 answer

Where can I get best spark tutorials for beginners?

May 14, 2020 in Apache Spark by akhtar
 • 38,260 points • 758 views
0 votes
1 answer

Py4JJavaError: An error occurred while calling o310.csv. : java.net.ConnectException: Call From master/192.168.56.101 to master:9000

May 7, 2020 in Apache Spark by akhtar
 • 38,260 points • 7,485 views
0 votes
1 answer

TypeError: Invalid param value given for param "inputCols". Could not convert DataFrame[R&D Spend: double, Administration: double, Marketing Spend: double]to list of strings

May 7, 2020 in Apache Spark by akhtar
 • 38,260 points • 2,677 views
0 votes
1 answer

Error: No module named 'findspark'

May 6, 2020 in Apache Spark by akhtar
 • 38,260 points • 20,552 views
0 votes
1 answer

ImportError: No module named 'pyspark'

May 6, 2020 in Apache Spark by akhtar
 • 38,260 points • 15,577 views
+2 votes
2 answers

py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getEncryptionEnabled does not exist in the JVM

Apr 7, 2020 in Apache Spark by akhtar
 • 38,260 points • 22,553 views
0 votes
1 answer

env: ‘python’: No such file or directory in pyspark.

Apr 7, 2020 in Apache Spark by akhtar
 • 38,260 points • 6,441 views
+1 vote
1 answer

How to assign a column in Spark Dataframe (PySpark) as a Primary Key?

Jan 9, 2020 in Apache Spark by anonymous
 • 150 points • 14,111 views
+1 vote
1 answer

Is there any efficient way of dealing null values during concat functionality of pyspark.sql version 2.3.4?

Nov 5, 2019 in Apache Spark by aizhar
 • 130 points • 39,979 views
+1 vote
0 answers

How to combine a nested json file, which is being partitioned on the basis of source tags, and has varying internal structure, into a single json file; ( differently sourced Tag and varying structure)

Oct 11, 2019 in Apache Spark by anonymous
 • 160 points • 1,120 views
+1 vote
1 answer

How to convert a json file structure with values in single quotes to quoteless ?

Oct 4, 2019 in Apache Spark by anonymous
 • 160 points • 4,238 views
0 votes
1 answer

How to sort json file ascending in pymongo?

Aug 14, 2019 in Python by Ahmed
 • 310 points • 1,143 views
+2 votes
1 answer

Import zip files and process the excel files ( inside the zip files ) by using pyspark connecting with pymongo

Aug 10, 2019 in Python by Ahmed
 • 310 points • 2,472 views
0 votes
1 answer

Pyspark dataframe with random values

Aug 1, 2019 in Apache Spark by Esha
 • 8,964 views
0 votes
1 answer

PySpark not starting: No active sparkcontext

Jul 30, 2019 in Apache Spark by Karan
 • 4,792 views
0 votes
1 answer

Unable to use ml library in pyspark

Jul 30, 2019 in Apache Spark by Yashita
 • 2,943 views
0 votes
1 answer

RDD word count with line numbers

Jul 25, 2019 in Apache Spark by Rishi
 • 2,543 views
0 votes
1 answer

Pyspark is taking default path

Jul 16, 2019 in Apache Spark by Will
 • 1,505 views
0 votes
1 answer

How can I remove headers from dataframe?

Feb 15, 2019 in Apache Spark by Dinesh
 • 20,276 views
0 votes
1 answer

How to perform a word count on a dataframe column?

Jan 22, 2019 in Big Data Hadoop by Karan
 • 2,346 views
0 votes
1 answer

PySpark Config ?

Jul 26, 2018 in Apache Spark by shams
 • 3,670 points • 807 views

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.