Installing Hadoop 2.2.0 on single Ubuntu 12.04 x86 64 Desktop

Home > Ubuntu > Hadoop cluster setup > Installing Hadoop 2.2.0 on single Ubuntu 12.04 x86_64 Desktop

Installation steps

1. Install java as mentioned at Installing Java on Ubuntu 12.04 x86_64 Desktop
2. Create user account and group for hadoop using:

   sudo groupadd hadoop
   sudo useradd hadoop -b /home -g hadoop -mkU -s /bin/bash
   cd /home/hadoop
   sudo cp -rp /etc/skel/.[^.]* .
   sudo chown -R hadoop:hadoop .
   sudo chmod -R o-rwx .

   Note -m in useradd should be specified before -k.

3. Install openssh server using:

   sudo apt-get -y install openssh-server

4. Setup password-less ssh for hadoop user:

   sudo su - hadoop
   ssh-keygen
   cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
   chmod 0600 ~/.ssh/authorized_keys
   #To test configuration, should echo hadoop
   ssh hadoop@localhost "echo $USER"
   exit

5. Download hadoop source from one of the mirrors linked at https://www.apache.org/dyn/closer.cgi/hadoop/common/ Download the latest stable .tar.gz release from stable folder. (Ex hadoop-2.2.0.tar.gz)
6. Extract hadoop sources in /opt/hadoop and make hadoop:hadoop its owner:

   sudo mkdir /opt/hadoop
   cd /opt/hadoop/
   sudo tar xzf <path-to-hadoop-source>
   sudo mv hadoop-2.2.0 hadoop
   sudo chown -R hadoop:hadoop .

7. Configure hadoop single-node setup using:
   1. Login as user hadoop:

      sudo su - hadoop

   2. Edit '~/.bashrc' and append

      export JAVA_HOME=/opt/jdk1.7.0_40
      export HADOOP_INSTALL=/opt/hadoop/hadoop
      export HADOOP_PREFIX=/opt/hadoop/hadoop
      export HADOOP_HOME=/opt/hadoop/hadoop
      export PATH=$PATH:$HADOOP_INSTALL/bin
      export PATH=$PATH:$HADOOP_INSTALL/sbin
      export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
      export HADOOP_COMMON_HOME=$HADOOP_INSTALL
      export HADOOP_HDFS_HOME=$HADOOP_INSTALL
      export YARN_HOME=$HADOOP_INSTALL
      export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
      export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"
      export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
      export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

   3. Change folder to /opt/hadoop/hadoop/etc/hadoop
   4. Edit 'hadoop-env.sh' and set proper value for JAVA_HOME such as '/opt/jdk1.7.0_40'. Do not leave it as ${JAVA_HOME} as that does not works.
   5. Edit '/opt/hadoop/hadoop/libexec/hadoop-config.sh' and prepend following line at start of script:

      export JAVA_HOME=/opt/jdk1.7.0_40

   6. Exit from hadoop user and relogin using 'sudo su - hadoop'. Check hadoop version using 'hadoop version' command.
   7. Again change folder to /opt/hadoop/hadoop/etc/hadoop
   8. Use 'mkdir /opt/hadoop/tmp'
   9. Edit 'core-site.xml' and add following between <configuration> and </configuration>:

      <property>
      <name>fs.default.name</name>
      <value>hdfs://localhost:9000</value>
      </property>
      <property>
      <name>hadoop.tmp.dir</name>
      <value>/opt/hadoop/tmp</value>
      </property>

   10. Edit 'yarn-site.xml' and add following between <configuration> and </configuration>:

       <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
       </property>
       <property>
       <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
       <value>org.apache.hadoop.mapred.ShuffleHandler</value>
       </property>

       yarn.nodemanager.aux-services.mapreduce.shuffle.class should probably be yarn.nodemanager.aux-services.mapreduce_shuffle.class Need to verify this

   11. Use 'cp mapred-site.xml.template mapred-site.xml'
   12. Edit 'mapred-site.xml' and add following between <configuration> and </configuration> tags:

       <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
       </property>

   13. Setup folders for HDFS using:

       cd ~
       mkdir -p mydata/hdfs/namenode
       mkdir -p mydata/hdfs/datanode
       cd /opt/hadoop/hadoop/etc/hadoop

   14. Edit 'hdfs-site.xml' and put following values between <configuration> and </configuration>:

       <property>
       <name>dfs.replication</name>
       <value>1</value>
       </property>
       <property>
       <name>dfs.namenode.name.dir</name>
       <value>file:/home/hadoop/mydata/hdfs/namenode</value>
       </property>
       <property>
       <name>dfs.datanode.data.dir</name>
       <value>file:/home/hadoop/mydata/hdfs/datanode</value>
       </property>

8. Format namenode using 'hdfs namenode -format'
9. Start dfs and yarn using 'start-dfs.sh' and 'start-yarn.sh'.
10. Test using 'jps', you should see following services running:

    18098 Jps
    17813 NodeManager
    17189 DataNode
    16950 NameNode
    17462 SecondaryNameNode
    17599 ResourceManager

11. Access NameNode at http://localhost:50070 and ResourceManager at http://localhost:8088
12. Run sample map reduce job using:

    cd /opt/hadoop/hadoop
    hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5

    Verify by using ResourceManager UI that task finishes successfully.

Some steps learned from http://codesfusion.blogspot.in/2013/10/setup-hadoop-2x-220-on-ubuntu.html?m=1 and https://gist.github.com/ruo91/7154697

Running a word count example on single-node hadoop installation

Use following steps:

      cd
      wget http://www.gutenberg.org/cache/epub/132/pg132.txt
      #Assuming all services are running, use jps to verify
      jps
      hadoop dfs -mkdir /user/hadoop/input
      hadoop dfs -copyFromLocal pg132.txt /user/hadoop/input/pg132.txt
      cd /opt/hadoop/hadoop 
      hadoop jar \
      ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar \
      wordcount /user/hadoop/input/pg132.txt /user/hadoop/output/wordcount
      hadoop dfs -cat /user/hadoop/output/wordcount/p* | less

To stop running hadoop daemons use:

      stop-yarn.sh
      stop-dfs.sh

Steps learned from https://giraph.apache.org/quick_start.html

Home > Ubuntu > Hadoop cluster setup > Installing Hadoop 2.2.0 on single Ubuntu 12.04 x86_64 Desktop