Hadoop Streaming
Hadoop streaming is a feature that comes with the Apache Hadoop distribution. The feature allows you to create and run Map/Reduce jobs with any executable or script as the mapper and/or the reducer.
How Hadoop Streaming works ?
For Mapper : When an executable is specified for mappers, each mapper task will launch the executable as a separate process when the mapper is initialized. As the mapper task runs, it converts its inputs into lines and feed the lines to the stdin of the process. In the meantime, the mapper collects the line oriented outputs from the stdout of the process and converts each line into a key/value pair, which is collected as the output of the mapper. By default, the prefix of a line up to the first tab character is the key and the rest of the line (excluding the tab character) will be the value. If there is no tab character in the line, then entire line is considered as key and the value is null.
For Reducer :When an executable is specified for reducers, each reducer task will launch the executable as a separate process then the reducer is initialized. As the reducer task runs, it converts its input key/values pairs into lines and feeds the lines to the stdin of the process. In the meantime, the reducer collects the line oriented outputs from the stdout of the process, converts each line into a key/value pair, which is collected as the output of the reducer. By default, the prefix of a line up to the first tab character is the key and the rest of the line is the value.
Hadoop Streaming is going to be very helpful to those people who has limited knowledge over java,d
because they just need to write two scripts i.e. one for the mapper and reducer.
For this the script that I going to use php scripting language for the demonstrative purpose.
So people who knows basics of the php are able to run map-reduce job with the help of the hadoop streaming.
Before we start I am assuming that php compiler is installed on your system. you can download the php compiler from here.
NOTE : php compiler should be installed in any of the hadoop cluster node
If you don't know how to setup apache hadoop on your system you can take the help of my previous blog on How to Setup Apache Hadoop
Now lets start, how to start with your first map task, I am assuming that $HADOOP_HOME is setup in your system, so to run the mapper task you have enter the following command :
$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-1.2.1.jar
-input /input
-output /output
-mapper "php map.php"
-file map.php
while we are going to run our first mapper we have provide certain parameters these are :
-input -> Path to input folder with respect to hadoop file system
-output -> Path to output folder with respect to hadoop file system
-mapper -> command you need to run for script specific file, for e.g. php file can be run on the command line by the command "php myfile.php"
-file -> Path to php script in the local file system
Input file should be in a format so that it can be easily understand by hadoop.
suppose we want to count number of times word hadoop in a text file , that text file can be taken as input, we can put it as follows :
Apache hadoop Apache hadoop Apache hadoop Apache hadoop Apache hadoop
We have to convert this file into key-value format, we can split this by space and and each word is assigned a count 1, to make it like this :
Apache 1
hadoop 1
Apache 1
hadoop 1
Apache 1
hadoop 1
Apache 1
hadoop 1
In the myfile.php, we have to write some code so that hadoop can understand it.
<?php
while (!feof(STDIN)) {
$line = fgets(STDIN);$split_line = $line.split(" "); foreach($split_line as $key){ echo $split_line."\t"."1"; }
}
