Course Outline for 3-day short course on Virtualization, Big Data and Cloud

• https://en.wikipedia.org/wiki/Hypervisor
• Software that performs bulk of virtualization.
• Host/Guest; Base/VM

• https://en.wikipedia.org/wiki/Hypervisor#Classification
• Type I hypervisor properties:
  • Bare Metal
  • CPU and OS are capable of giving resources to VMs directly
  • Domain 0/Domain U
  • VMWare ESX, Hyper-V, KVM
  • CPU support check
  • BIOS configuration required.
• Type II hypervisor properties:
  • Hardware is emulated and binary translation for privileged instructions
  • Virtual-Box, VMWare Player
  • Typically only 32-bit guests are supported even on 64-bit host

• https://en.wikipedia.org/wiki/Paravirtualization
• Both host and guest OS are modified such that it is clear that they are virtualized. Further the two are modified to work well with each other.
  • Xen supports both Para-virtualization and HVM. Xen was default hypervisor for CentOS 5.x. From CentOS 6.x KVM has become default.
• Drivers that work well with virtualized hardware as many checks/balances that are necessary for physical hardware (Eg checking temperature) is not required with virtual hardware.

• Shared kernel between host and guest. The guest only has its own directory structure based on chroot of a host OS folder.
• Host kernel maintains separate process tree, file system handles, network stack, etc. for guest to provide isolation.
• Near host performance as there is no virtualization overhead.
• Unused resources are shared.
• Resources can be adjusted live without down-time.

• What is network https://simple.wikipedia.org/wiki/Computer_network
• What is distributed computing https://en.wikipedia.org/wiki/Distributed_computing
• What is grid
  • https://en.wikipedia.org/wiki/Grid_computing
• What is cluster or HPC
  • https://en.wikipedia.org/wiki/Computer_cluster
• What is cloud
  • https://en.wikipedia.org/wiki/Cloud_computing

• Types of cloud
  • IAAS, PAAS, SAAS
  • Public, Private, Hybrid
• Pay per use in case of public cloud
• Self provisioning
• Easily scale by adding hardware resources to common pool
• Support DCs spread across large distances
• Metering
• APIs.

OpenStack

It has gained most popularity compared to other alternatives. The software is developed democratically hence it has many components that work together.

Personal opinion

It takes some time to get used to OpenStack administration and setup. There is a steep learning curve. Further, many advanced features that come with it may not be needed for all organizations. If simple auto-provisioning, ability to create VMs, simple networking, etc. is required then Cloudstack seems like a better choice to explore clouds. If cloudstack features seem limited then it makes sense to go for OpenStack with additional features and associated complexity.

CloudStack

This is the simplest cloud to setup among many cloud software tried by me and my peers. It has all the basic options related to creation of VMs, migrating VMs, taking maintenance of base machines, etc. It has a web front-end with a MySQL backend. So making this highly available is easier. Further, the VMs are visible on base machines and can be managed by virsh directly in case cloudstack layer fails (not reported by anyone so far).

Apache cloudstack setup

http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.8/qig.html

Apache cloudstack agent setup

http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.8/qig.html

Apache cloudstack import ISO/template

Connection refused error resolution

https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM,+templates,+Secondary+storage+troubleshooting

Others

Apart from OpenStack and CloudStack, there are:

• Eucalyptus
• oVirt
• Open Nebula

etc. cloud software. which are either less famous or more complex and hence are not that mainstream yet.

Data that cannot fit on one single computer

Map-reduce based distributed framework to process big data

1. Map example

   lists:map(fun(X) -> X*2 end, [1, 2, 3]).
   

2. Filter example

   lists:filter(fun(X) -> X rem 2 =:= 0 end, lists:seq(1,10)).
   

3. Foldl (reduce) example

   lists:foldl(fun(X,Y) -> X+Y end, 0, [2,4,6,8,10]).
   

Refer:

• http://learnyousomeerlang.com/content
• http://learnyousomeerlang.com/higher-order-functions#maps-filters-folds

Steps:

1. Create three containers or machines with names master, slave1 and slave2.
2. Edit /etc/hosts in all three machines such that they can reach master, slave1 and slave2 by name
3. Install java by using command:

   yum localinstall jdk-8u112-linux-x64.rpm
   

   after going to the directory where jdk rpm is present.

4. In all machines do:

   useradd hadoop
   passwd hadoop
   

5. On master as hadoop user do:

   ssh-keygen
   ssh-copy-id hadoop@master
   ssh-copy-id hadoop@slave1
   ssh-copy-id hadoop@slave2
   

   Test using

   ssh hadoop@master ls
   ssh hadoop@slave1 ls
   ssh hadoop@slave2 ls
   

6. Install hadoop using following commands as root user on all three machines:

   mkdir /opt/hadoop 
   cd /opt/hadoop/ 
   tar xzf <path-to-hadoop-source> 
   mv hadoop-2.7.3 hadoop 
   chown -R hadoop:hadoop /opt/hadoop
   

7. Login as user hadoop
   1. sudo su - hadoop
   2. Append following configuration to ~/.bashrc

      export JAVA_HOME=/usr/java/jdk1.8.0_112
      export HADOOP_INSTALL=/opt/hadoop/hadoop 
      export HADOOP_PREFIX=/opt/hadoop/hadoop 
      export HADOOP_HOME=/opt/hadoop/hadoop
      export PATH=$PATH:$HADOOP_INSTALL/bin 
      export PATH=$PATH:$HADOOP_INSTALL/sbin 
      export HADOOP_MAPRED_HOME=$HADOOP_INSTALL 
      export HADOOP_COMMON_HOME=$HADOOP_INSTALL 
      export HADOOP_HDFS_HOME=$HADOOP_INSTALL 
      export YARN_HOME=$HADOOP_INSTALL 
      export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native 
      export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib" 
      export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop 
      export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
      

   3. cd /opt/hadoop/hadoop/etc/hadoop
   4. Edit hadoop-env.sh and set correct value for JAVA_HOME such as /usr/java/jdk1.8.0_112. Do not leave it as ${JAVA_HOME}.
   5. Edit /opt/hadoop/hadoop/libexec/hadoop-config.sh and prepend following line at the beginning of file:

      export JAVA_HOME=/usr/java/jdk1.8.0_112
      

   6. exit
8. Again login as hadoop user
   1. su - hadoop
   2. hadoop version
   3. mkdir /opt/hadoop/tmp
   4. cd /opt/hadoop/hadoop/etc/hadoop
   5. Edit core-site.xml so that it has following between <configuration> and </configuration>

      <property> 
        <name>fs.default.name</name> 
        <value>hdfs://master:9000</value> 
      </property> 
      <property> 
        <name>hadoop.tmp.dir</name> 
        <value>/opt/hadoop/tmp</value> 
      </property>
      

   6. Setup folders for hdfs using:

      cd ~ 
      mkdir -p mydata/hdfs/namenode 
      mkdir -p mydata/hdfs/datanode 
      cd /opt/hadoop/hadoop/etc/hadoop
      

   7. Edit hdfs-site.xml and use:

      <property>
        <name>dfs.replication</name>
        <value>2</value>
      </property>
      <property>
        <name>dfs.permissions</name>
        <value>false</value>
      </property>
      <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/home/hadoop/mydata/hdfs/namenode</value>
      </property>
      <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/home/hadoop/mydata/hdfs/datanode</value>
      </property>
      

   8. cp mapred-site.xml.template mapred-site.xml
   9. Edit mapred-site.xml and use:

      <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
      </property>
      

   10. Edit yarn-site.xml and use:

       <property>
         <name>yarn.nodemanager.aux-services</name>
         <value>mapreduce_shuffle</value>
       </property>
       <property>
         <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
         <value>org.apache.hadoop.mapred.ShuffleHandler</value>
       </property>
       <property>
         <name>yarn.resourcemanager.resource-tracker.address</name>
         <value>master:8025</value>
       </property>
       <property>
         <name>yarn.resourcemanager.scheduler.address</name>
         <value>master:8030</value>
       </property>
       <property>
         <name>yarn.resourcemanager.address</name>
         <value>master:8040</value>
       </property>
       

   11. hdfs namenode -format
9. Edit /opt/hadoop/hadoop/etc/hadoop/slaves so that it contains:

   slave1
   slave2
   

   If you want to store data on master also then add master to the list

10. start-dfs.sh
11. start-yarn.sh
12. jps on master should show 'ResourceManager', 'NameNode' and 'SecondaryNameNode'
13. jps on slaves should show 'NodeManager' and 'DataNode'
14. Access NameNode at http://master:50070 and ResourceManager at http://master:8088
    • To access from base machine add master and corresponding IP in /etc/hosts before opening the URL in browser
    • http://slave1:8042 for accessing slave directly.
15. Run example map-reduce job
    1. setup input word count file:

       cd ~
       mkdir in
       cat > in/file <<EOF
       This is one line
       This is another one
       EOF
       

    2. hdfs dfs -copyFromLocal in /in
    3. Run the job:

       hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /in /out
       

    4. hdfs dfs -cat /out/*
16. stop-yarn.sh
17. stop-dfs.sh

Refer:

• http://www.sbarjatiya.com/notes_wiki/index.php/Installing_Hadoop_2.2.0_cluster_on_Ubuntu_12.04_x86_64_Desktops
• http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html

Note

• Latest hadoop 2.7.3 with java 8u112 is working fine on CentOS 6.8.

Getting comfortable with hadoop file-system management:

hadoop fs -ls /
hadoop fs -ls /in/
hadoop fs -cat /in/file
vim in/file
hadoop fs -copyFromLocal -f in/file /in/file
hadoop fs -cat /in/file
hadoop fs
hadoop fs -du -s -h /
hadoop fs -du -s -h /in
hadoop fs -du -s -h /out
hadoop fs -du -s -h /tmp
hadoop fs -ls /out
hadoop fs -cat /out/part-r-00000
hadoop fs -rm -r /in /out /tmp
hadoop fs -ls /
hadoop fs -copyFromLocal in /in
hadoop fs -ls /
hadoop fs -ls /in
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /in /out
hadoop fs -cat /out/*

Hadoop filesystem administration

hdfs dfsadmin -report
hdfs dfsadmin -metasave test
cd /opt/hadoop/hadoop/logs/
cat test
rm -f test
start-balancer.sh  #Balance data among new nodes

Note:

• hdfs dfsadmin -rollingUpgrade query
• hdfs dfsadmin -finalizeUpgrade

1. WordCount.java

   import java.io.IOException;
   import java.util.StringTokenizer;
   
   import org.apache.hadoop.conf.Configuration;
   import org.apache.hadoop.fs.Path;
   import org.apache.hadoop.io.IntWritable;
   import org.apache.hadoop.io.Text;
   import org.apache.hadoop.mapreduce.Job;
   import org.apache.hadoop.mapreduce.Mapper;
   import org.apache.hadoop.mapreduce.Reducer;
   import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
   import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
   
   public class WordCount {
   
     public static class TokenizerMapper
          extends Mapper<Object, Text, Text, IntWritable>{
   
       private final static IntWritable one = new IntWritable(1);
       private Text word = new Text();
   
       public void map(Object key, Text value, Context context
                       ) throws IOException, InterruptedException {
         StringTokenizer itr = new StringTokenizer(value.toString());
         while (itr.hasMoreTokens()) {
           word.set(itr.nextToken());
           context.write(word, one);
         }
       }
     }
   
     public static class IntSumReducer
          extends Reducer<Text,IntWritable,Text,IntWritable> {
       private IntWritable result = new IntWritable();
   
       public void reduce(Text key, Iterable<IntWritable> values,
                          Context context
                          ) throws IOException, InterruptedException {
         int sum = 0;
         for (IntWritable val : values) {
           sum += val.get();
         }
         result.set(sum);
         context.write(key, result);
       }
     }
   
     public static void main(String[] args) throws Exception {
       Configuration conf = new Configuration();
       Job job = Job.getInstance(conf, "word count");
       job.setJarByClass(WordCount.class);
       job.setMapperClass(TokenizerMapper.class);
       job.setCombinerClass(IntSumReducer.class);
       job.setReducerClass(IntSumReducer.class);
       job.setOutputKeyClass(Text.class);
       job.setOutputValueClass(IntWritable.class);
       FileInputFormat.addInputPath(job, new Path(args[0]));
       FileOutputFormat.setOutputPath(job, new Path(args[1]));
       System.exit(job.waitForCompletion(true) ? 0 : 1);
     }
   }
   

2. echo $JAVA_HOME
3. export PATH=$PATH:$JAVA_HOME/bin
4. export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar
5. hadoop com.sun.tools.javac.Main WordCount.java
6. jar cf wc.jar WordCount*.class
7. hadoop jar wc.jar WordCount /in /out2
8. hadoop fs -cat /out2/*

Refer:

• http://hadoop.apache.org/docs/r2.7.3/
• http://hadoop.apache.org/docs/r2.7.3/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html
• https://developer.yahoo.com/hadoop/tutorial/module1.html

Note that this older version of code which uses deprecated API. However, since hadoop is backward compatible this code still works.

import java.io.File;
import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;

public class HDFSHelloWorld {

  public static final String theFilename = "hello.txt";
  public static final String message = "Hello, world!\n";

  public static void main (String [] args) throws IOException {

    Configuration conf = new Configuration();
    FileSystem fs = FileSystem.get(conf);

    Path filenamePath = new Path(theFilename);

    try {
      if (fs.exists(filenamePath)) {
        // remove the file first
        fs.delete(filenamePath);
      }

      FSDataOutputStream out = fs.create(filenamePath);
      out.writeUTF(message);
      out.close();

      FSDataInputStream in = fs.open(filenamePath);
      String messageIn = in.readUTF();
      System.out.print(messageIn);
      in.close();
    } catch (IOException ioe) {
      System.err.println("IOException during operation: " + ioe.toString());
      System.exit(1);
    }
  }
}

Compile and test using:

export PATH=$PATH:$JAVA_HOME/bin 
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar 
hadoop com.sun.tools.javac.Main HDFSHelloWorld.java 
hdfs dfs -ls /
hdfs dfs -ls /user/
hdfs dfs -ls /user/hadoop/
hdfs dfs -cat /user/hadoop/hello.txt