sparkstream textfilstreaming localpathdirectory I am getting empty results

Question

I have given the local path for spark text file streaming. But it is giving empty results even after adding new files. The path exists and it is not open also. Can you please tell how to rectify this.

Below is the code:

val pathFile:Path=new Path(
  "file:\\\\\\D:\\sparklearning\\inputFileStreaming\\"
)
println(pathFile.toString())
val linesDStream = streamingContext.textFileStream(pathFile.toString)
linesDStream.foreachRDD(x=>println(x.toString()))

Answer 1

Hi, according to the documentation, you have to mention the path of hdfs. So, start the Spark job, copy the file form your local file system to hdfs and then use Spark text file streaming.

Answer 2

Hey @c.kothamasu

You should copy your file to HDFS because Spark process only those files that are copied into HDFS while the program is running.