Do we need to format datanode to store data

Question

Hi,

While setting up pseudo or full cluster, do we need to format datanode to install HDFS file system or we only have to format namenode.

I am asking this because, I have read this on many blogs that, HDFS stores the data in sequential order on the hard disk and it is an abstract layer which stores data on big blocks rather than default block size storage provided host file system.

If we are not formatting datanode than the powerful feature of HDFS comes to toss.

Comments

Could you please elaborate what you meant by this statement:
"powerful feature of HDFS comes to toss."?

Answer 1

Hey @Sam!

Namenode is the component that holds all the records (metadata) of files stored in hdfs. It's holds the tree structure of all files stored on the cluster and keeps track of these files.

The common practice is to format only the namenode. When you format the namenode, the metadata related to the data stored in the hdfs is lost but in fact, the data is still present in the datanode. When the namenode is formatted, the namenode thinks that there is no data in the hdfs.

It is not a "must-do" step to format the datanode. Formatting the namenode will be just fine.

If you want to format the datanode, you can do it manually using the rmr option. Refer to the below syntax:

hadoop fs -rmr /path/to/hdfs

Comments

Hi,

Will there be any performance improvement in case of read and write operation if we format datanode explicitely.

Thanks

Answer 2

That was not important. If you can answer, does  namenode -format command formats the datanode directory structure for HDFS file system for all the datanodes connected within cluster.

Comments

The namenode -format command doesn't delete or format the datanode. The namenode -format only deletes the metadata on the namenode and formats the namenode. The data is still present on the datanodes but it can not be accessed because the after formatting the namenode, the namenode doesn't know where the data is present.