Reading a text file through spark data frame

Question

Hi team,

val df = sc.textFile("HDFS://nameservice1/user/edureka_168049/Structure_IT/samplefile.txt")
df.show()

the above is not working and when checking my NameNode it is saying security is off and safe mode is off.

even the below is also not working

val df = spark.read
.format("csv")
.option("header","true")
.option("inferSchema","true") 
.option("nullValue","NA")
.option("timestampFormat","yyyy-MM-dd"T"HH:mm:ss")
.option("mode","failfast")
.load("hdfs://nameservice1/user/edureka_168049/Structure_IT/sparkfile.csv")

Answer 1

Try this:

val df = sc.textFile("HDFS://nameservice1/user/edureka_168049/Structure_IT/samplefile.txt")
df.collect()
val df = spark.read.option("header","true").option("inferSchema","true").csv("hdfs://localhost :9000/airports.csv")
df.show()

To know more about SQL, It's recommended to join PySpark course today.