Crawling with Apache Nutch, Apache Hadoop

This tutorial is all about creating your own scalable search engine that has capability to crawl data from various websites in the scalable environment and stores the data into NoSQL database and at last all the crawl data will be available for the search with the help of search engine.

This tutorial will works for most of your favorite linux (Like Ubuntu,CentOS etc)

Software Requirement

apache-nutch-1.8   

apache-hadoop-1.2.1

apache-solr-3.x

Steps for installing hadoop

For making our crawler scalable and efficient we are using hadoop software framework. Hadoop Allows distributed processing of large data sets with the use of simple programming.
It has the capability to scale from single machine to thousands of machine with the use of commodity hardware.
Map reduce is one of the feature of the hadoop that make any application to scale horizontally over multiple servers.

There are mainly three ways in which hadoop is being configured.These are :

1.Standalone mode
2.Pseudo Distributed Mode
3.Distributed mode

Here, we are using Pseudo Distributed mode for setting up of hadoop, in this mode various processes (Namenode, Secondary Namenode, JobTracker, Datanode, TaskTracker ) are all running with in single machine.

While in distributed mode, these process will runs on master and slave machine.

on master : Namenode, Secondary Namenode, JobTracker (Only one)
on Slave : Datanode and TaskTracker (Can be more than one)

So, If everything is properly in pseudo distributed mode, that to convert this is distributed mode is just matter changing the configuration.That we will discuss shortly.

Follow the steps to setup hadoop :

1. Create a username huser in the ubuntu

sudo adduser huser

It will ask for the password and some basic details after this the user will be created

2. Now login to user just created

su huser (It will ask for the password)

3. Download hadoop from the link

wget http://psg.mtu.edu/pub/apache/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz

4. Extract using the command

tar -zxvf hadoop-1.2.1.tar.gz

5. In the Extracted folder, edit the file conf/hadoop-env.sh to define at least JAVA_HOME to be the root of your Java installation. also remove the hash (#) if it is there.

Try the following command to check hadoop executable is setup

bin/hadoop (If this command is working then everything is working)

6. Setting up of the configuration files conf/mapred-site.xml

<property>

        <name>mapred.job.tracker</name>

        <value>127.0.0.1:9001</value>

    </property>

<property>

        <name>mapred.loacl.dir</name>

        <value>/hdd</value>

    </property>

conf/core-site.xml

<property>

        <name>hadoop.tmp.dir</name>

        <value>/hdd/hadoop</value>

    </property>

<property>

        <name>fs.default.name</name>

        <value>hdfs://127.0.0.1:9000</value>

    </property>

conf/hdfs-site.xml

<property>

        <name>dfs.replication</name>

        <value>1</value>

    </property>

7. Install following software if not already installed

sudo apt-get install ssh

sudo apt-get install rsync

8. Setting passphraseless hadoop

Now check that you can ssh to the localhost without a passphrase:

ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

9. Setting Master and slave node

edit conf/masters and set the one IP address per line for making the master node

edit the conf/slaves and set the one IP address per line for making the slave node

10. Running the hadoop cluster

bin/hadoop namenode -format

11. Start the hadoop daemons:

bin/start-all.sh

12. Browse the web interface for the NameNode and the JobTracker; by default they are available at:

• NameNode - http://localhost:50070/
• JobTracker - http://localhost:50030/

Steps for Running Nutch

For Crawling we are using apache nutch. It is open source web crawler written in java. Nutch is one of my favorite crawler because of its nicely written code, It is very easy to change the source code if we need more specialized feature than vanilla installation.
With nutch we can find broken links, temporary redirects, robots denied page, create copy of visited pages etc.

Follow the steps to setup nutch :

NOTE : Nutch is setup on machine in which hadoop master is running

1. Download Nutch from link :

http://mirrors.koehn.com/apache/nutch/1.8/apache-nutch-1.8-bin.zip

2. Extract the above downloaded zip file into folder names nutch

3. Extracted files consist of two important folder named bin and conf

4. Bin folder consist of two shell script nutch and crawl. With the help these script we are going to start our crawler by providing certain inputs that we will discussed in next.

5. Now open the folder conf this consist of all the necessary configuration files mainly conf/nutch-site.xml and conf/regex-urlfilter.txt.

Add your agent name in the value field of the http.agent.name property in conf/nutch-site.xml, for example:

<property> 
<name>http.agent.name</name> 
<value>My Nutch Spider</value> 
</property>

Crawl Script :

The nutch crawler can be run by the following command :

bin/crawl <seedDir> <crawlDir> <solrIP> <numberOfRounds>

seedDir : Seed directory path

crawlDir : Output directory path

solrIP : solr IP

numberOfRounds : number of levels to crawl

The configuration for the crawler is already been set, but if there needs to change this, It can be done by changing configuration file conf/nutch-site.xml