FileStreamSink Error while looking for metadata directory java lang IllegalArgumentException java net UnknownHostException hive

+2 votes

Hi,

I tried to read one csv file in spark sql. But I am getting the below error.

val dfs = sqlcontext.read.format("com.databricks.spark.csv").option("header", "true").option("inferschema","true").load("hdfs://hive/bike")
20/02/13 09:38:05 WARN streaming.FileStreamSink: Error while looking for metadata directory.
java.lang.IllegalArgumentException: java.net.UnknownHostException: hive
  at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
  at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
  at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
  at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:668)
  at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:604)
  at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2598)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2632)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2614)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:547)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:545)
  at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
  at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
  at scala.collection.immutable.List.flatMap(List.scala:355)
  at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:545)
  at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)
  at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
  ... 49 elided
Caused by: java.net.UnknownHostException: hive
  ... 73 more
                               
Feb 13, 2020 in Big Data Hadoop by akhtar
• 38,260 points
11,175 views

1 answer to this question.

0 votes

Hi@akhtar,

You tried to read file from your hdfs cluster. So you have to give the full path of the file along with master ip and port.

$  val dfs = sqlcontext.read.format("com.databricks.spark.csv").option("header", "true").option("inferschema","true").load("hdfs://master ip:port/hive/bike")

Hope it will solve your problem.

Thank You

answered Feb 13, 2020 by MD
• 95,460 points

Related Questions In Big Data Hadoop

0 votes
1 answer

Hadoop: Format namenode gives error: Shutting down NameNode at java.net.UnknownHostException

UnknownHostException is thrown when hadoop tries to resolve ...READ MORE

answered Nov 12, 2018 in Big Data Hadoop by Omkar
• 69,220 points
5,815 views
0 votes
1 answer

Error while connecting to Hive using Java JDBC

Use ​org.apache.hive.jdbc.HiveDriver as your driver ...READ MORE

answered Nov 22, 2018 in Big Data Hadoop by Omkar
• 69,220 points
1,823 views
0 votes
1 answer
0 votes
1 answer

What is Metastore in Hive?

It stores metadata for Hive tables (like their schema ...READ MORE

answered Dec 20, 2018 in Big Data Hadoop by Frankie
• 9,830 points
3,123 views
+1 vote
2 answers
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,999 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP