Hadoop Cluster Node Setup

Question

I am setting up hadoop on a multinode cluster, and I have a few questions:

1. Will it be ok to have NameNode and ResourceManager on the same machine?

2. Which will be the best role for a master system, NameNode, ResourceManager Or DataNode/NodeManager?.

3. I have a master and 3 slave machines. The slaves file on the master machine has the following entries:

   master
   slave1
   slave2
   slave3
   

Do I have to place this same slaves file in all of the slave machines? Or should I remove the first line (master) and then place it in the slave machines?

Answer 1

1. Yes, at least in small clusters those two should be running in the master node.
2. Check answer 1. Master node can have also, for example, SecondaryNamenode and JobHistoryServer
3. No, the slaves file is only on the master node. If you have the master node in the slaves file, it means that the master node acts also as a datanode. Especially in small clusters that's totally fine. The slaves file essentially tells which on nodes the datanode processes are started.

Slave nodes should only run DataNode and NodeManager. But this is all handled by Hadoop if the configurations are correct - you can just check which processes are running after starting the cluster from the master node. Master node basically takes care of everything and you "never" need to manually connect to the slaves for any configurations.

My answer is meant for small clusters, probably in bigger "real" clusters the server responsibilities are even more separated.