Adding Hadoop Slave Node to Running Hadoop Cluster

Hello readers today I am going to show you how you can add hadoop slave node to the running hadoop cluster, in the previous tutorial on Crawling with Apache Nutch and Apache Hadoop , where we have setup the apache nutch crawler over apache hadoop cluster.

If we are using any crawler that crawls over the web and day by day data will increase exponentially,

Due to limited number of the computational resources (slave nodes) in the hadoop cluster, the processing of the data take more time. To speed up the process it is better to add one or more slave node, this will increase the processing power to our data processing but also prevent blocking of the CPU threads.

Suppose your hadoop cluster is running with 1 datanode and 1 tasktracker and you want to add one more data node to this cluster.

Follow the steps to configure the new data node :

1.  configure slave node with the help of this tutorial, this is similar to configuring the master node.

2.  login to master node using ssh

ssh huser@master

3.  Add IP address of the slave node to the master slave text file with the help of text editor (here i am using vim)

huser@master$ vim $HADOOP_HOME/conf/slave

4.  copy all the configuration from the master node to new slave node using scp shell command including ssh public key

huser@master$ scp /home/huser/.ssh/authorized_keys $HADOOP_HOME/conf/* huser@slave:/home/huser/$HADOOP_HOME/conf/

5.  start the datanode

huser@slave$ $HADOOP_HOME/bin/hadoop start datanode

6.  start the tasktracker

huser@slave$ $HADOOP_HOME/bin/hadoop start tasktracker

After this you new data node will automatically discovered by the namenode and the new map/reduce task will assign to it if any job is running.

When the new data node is added, then it can have only new blocks but to make proper use of new data node the total number of the block to be equally distributed, this can be done with the help of the balancer.

Following command can be used to balance the total blocks in the cluster

huser@slave$ $HADOOP_HOME/bin/start-balancer.sh