load save text file in spark

0 votes

I am using load command but it is loading only parquet 

val a = spark.read.load("employee.parquet")--works fine
val a=spark.read.load("employee.txt")--error

Why is this so?

Jul 22, 2019 in Apache Spark by Esha
3,446 views

1 answer to this question.

0 votes

The reason you are able to load employee.parquet and not employee.txt using load is that spark.read.load by default assumes that data source is in parquet format so it is able to load it but we can use format function which can be used to specify the different format and use the load function to load the data. Refer to this code:

val a =spark.read.format("csv").load("employee.txt")
answered Jul 22, 2019 by Giri

Related Questions In Apache Spark

0 votes
1 answer

Load custom delimited file in Spark

Refer to the following code: val sqlContext = ...READ MORE

answered Jul 24, 2019 in Apache Spark by Ritu
1,906 views
+1 vote
1 answer

How to read a data from text file in Spark?

Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE

answered Aug 6, 2019 in Apache Spark by Gitika
• 65,770 points
5,027 views
0 votes
1 answer

Efficient way to read specific columns from parquet file in spark

As parquet is a column based storage ...READ MORE

answered Apr 20, 2018 in Apache Spark by kurt_cobain
• 9,350 points
7,860 views
0 votes
1 answer

where can i get spark-terasort.jar and not .scala file, to do spark terasort in windows.

Hi! I found 2 links on github where ...READ MORE

answered Feb 13, 2019 in Apache Spark by Omkar
• 69,220 points
1,344 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
11,033 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,540 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
108,853 views
+1 vote
1 answer

How can I write a text file in HDFS not from an RDD, in Spark program?

Yes, you can go ahead and write ...READ MORE

answered May 29, 2018 in Apache Spark by Shubham
• 13,490 points
8,460 views
0 votes
1 answer

load/save in spark

The reason why you are able to ...READ MORE

answered Jul 5, 2019 in Apache Spark by Firoz
1,560 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP