Moving data to hdfs using copyFromLocal command

Question

I am trying to copy a simple file present in my local filesystem to the directory specified for hdfs. I don't know what I'm doing wrong?
I have specified in my hdfs-site.xml that the directory for hdfs will be /home/user1/Hadoop/dataNodeHadoopData using the following properties -
<name>dfs.data.dir</name>
<value>/home/user1/Hadoop/dataNodeHadoopData/</value>

and

<name>dfs.name.dir</name>
<value>/home/user1/Hadoop/dataNodeHadoopData/</value>
 

I am using the following command -
bin/hadoop dfs -copyFromLocal /home/user1/ml-100k/sample.data /home/user1/Hadoop/dataNodeHadoopData
to copy the file sample.data from it's local filesystem location to the HDFS directory that I specified. But when I run this command, I'm not able to copy the data from local filesystem to hdfs. Can anyone please help?

Answer 1

Actually dfs.data.dir and dfs.name.dir have to point to two different directories. 
Also make sure you have formatted the namenode FS after changing the directories in the configuration.
In your case while copying to HDFS you're incorrectly specifying the target. 

The correct syntax for copying a local file to HDFS is:
bin/hadoop dfs -copyFromLocal <local_FS_filename> <target_on_HDFS>

Correct syntax in your case
bin/hadoop dfs -copyFromLocal /home/user1/ml-100k/sample.data <hdfs directory>

Comments

Hi 

I am trying to copy local file to hdfs with below command

hdfs dfs -copyFromLocal /Desktop/edureka/475_m3_county_facts.csv /user/edureka_372192/test_dir/country_facts.csv

getting error No such file or directory 

Please help me for this

---

Hey @Kishori.

Firstly, your command is wrong. The destination should be a path to a directory but in your command, the destination is a file:

hdfs dfs -copyFromLocal /Desktop/edureka/475_m3_county_facts.csv /user/edureka_372192/test_dir/country_facts.csv

 The right command should be:

hdfs dfs -copyFromLocal /Desktop/edureka/475_m3_county_facts.csv /user/edureka_372192/test_dir/

If you still get this error, then there are 3 possible problems:

1. The file does not exist in the path mentioned
2. The source path is wrong
3. The destination path is wrong

Troubleshooting:

Open terminal and change the directory to the directory where the file is present. Now, run this command to check the present working directory: 

$pwd

The output you get should be the source path in your hadoop command.

Next, check if the destination path is right. 

Hope this helps..

---

Don't mention file name in destination