Install giraph in hadoop node

Home > Java > Install giraph in hadoop node

1. Setup hadoop in single Cent-OS node as explained at Install hadoop in a single Cent-OS node
2. Create a directory for temporary files such as /opt/hadoop/tmp and add following to 'conf/core-site.xml' file:

   <property>
   <name>hadoop.tmp.dir</name>
   <value>/opt/hadoop/tmp</value>
   </property>

3. Edit conf/mapred-site.xml file and add following configuration to allow 4 mappers to run in parallel:

   <property>
   <name>mapred.tasktracker.map.tasks.maximum</name>
   <value>4</value>
   </property>
   <property>
   <name>mapred.map.tasks</name>
   <value>4</value>
   </property>

4. Edit conf/hdfs-site.xml and add:

   <property>
   <name>dfs.replication</name>
   <value>1</value>
   <description></description>
   </property>

   to configure hdfs to maintain only one copy of data, effectively disabling replication.

5. Format the node using ./bin/hadoop namenode -format, only if not formatted already.
6. Start all services using ./bin/start-all.sh
7. Install maven using

   sudo yum -y install maven

   Verify that installed version is >= 3.0 using mvn --version

8. Download latest stable giraph from https://www.apache.org/dyn/closer.cgi/giraph/
9. Extract giraph source in /opt/hadoop/giraph folder
10. Make sure giraph files are owned by hadoop:hadoop
11. Edit ~/.bash_profile for hadoop user and add:

    export GIRAPH_HOME=/opt/hadoop/giraph

12. Exit from hadoop user and login again. Verify that variable is set using:

    set | grep GIRAPH

13. Install maven using:

    cd $GIRAPH_HOME
    mvn package

    If you want to avoid running tests after install use:

    mvn package -DskipTests

14. If installation is successful then folder 'giraph-core/target' should have file named 'giraph-<ver>-for-hadoop-<ver>-jar-with-dependencies.jar'. Also folder 'giraph-examples/target/' would have jar file for examples with similar naming.

Steps learned from https://giraph.apache.org/quick_start.html

Testing giraph by running a simple giraph job

We will run a simple shortest-path computation giraph job to verify giraph installation:

1. Create input file named tiny_graph.txt with following data in '/opt/hadoop/data' folder:

   [0,0,[[1,1],[3,3]]]
   [1,0,[[0,1],[2,2],[3,1]]]
   [2,0,[[1,2],[4,4]]]
   [3,0,[[0,3],[1,1],[4,4]]]
   [4,0,[[3,4],[2,4]]]

   Each line above has the format '[source_id,source_value,[[dest_id, edge_value],...]]'. In this graph, there are 5 nodes and 12 directed edges. Copy the input file to HDFS:

   ../hadoop/bin/hadoop dfs -copyFromLocal tiny_graph.txt /user/hduser/input/tiny_graph.txt
   ../hadoop/bin/hadoop dfs -ls /user/hduser/input

2. Run the task using:

   ../hadoop/bin/hadoop jar /opt/hadoop/giraph/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner  org.apache.giraph.examples.SimpleShortestPathsVertex -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hduser/input/tiny_graph.txt -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hduser/output/shortestpaths -w 1

3. Check the output using:

   ../hadoop/bin/hadoop dfs -cat /user/hduser/output/shortestpaths/p* | less

Steps learned from https://giraph.apache.org/quick_start.html

Home > Java > Install giraph in hadoop node