Migrate sbarjatiya.com VM

From Notes_Wiki
Revision as of 11:44, 29 September 2020 by Saurabh (talk | contribs)

<yambe:breadcrumb>New_machine_configuration|New machine configuration</yambe:breadcrumb>

Migrate sbarjatiya.com VM

VM creation on AWS

  1. Create a new AWS account
  2. Ensure that desired region / AZ VPC and subnet have IPv6 CIDR allocated.
  3. In route table route ensure that route for ::/0 for same igw as for 0.0.0.0/0 is present
  4. Ensure that this route table is associated with subnet for which IPv6 CIDR is allocated
  5. Create VM with IPv6 address, enough disk space and in correct region/subnet. Security group should allow:
    SSH (22)
    from everywhere (IPv4, IPv6)
    HTTP (80), HTTPS (443)
    from everywhere (IPv4, IPv6)
    SMTP (25), SMTPS (465)
    from everywhere (IPv4, IPv6)
    Custom Alt-web (8080)
    from everywhere (IPv4, IPv6)
    IPv4 ICMP echo-request
    From all IPv4 0.0.0.0/0
    All ICMPv6
    From all IPv6 ::/0
  6. Get IPv4 elastic IP and associate with VM.
  7. Add entry in /etc/hosts of current machine with appropriate name for new elastic IP (eg newcommonhosting)
  8. SSH to new machine as centos user
  9. Do "sudo su -" on new VM to get root console
  10. Install vim 
    yum -y install vim epel-release
    

    yum -y install byobu wget
  11. Check that IPv6 address is available 
    ip addr show
    

    ip -6 route show
  12. Edit /etc/sysconfig/network and update 
    NOZEROCONF=no
    

    IPV6_AUTOCONF=yes
  13. Enable processing of IPv6 router advertizements by creating "/etc/sysctl.d/99-enable-ipv6-ra.conf" with: 
    net.ipv6.conf.all.accept_ra = 1
    

    net.ipv6.conf.default.accept_ra = 1
  14. Enable the same using 
    sysctl -p /etc/sysctl.d/99-enable-ipv6-ra.conf
  15. Restart network in VM using 
    systemctl restart network
  16. Validate that there is proper default gateway for IPv6 using: 
    ip -6 route show
  17. Try outgoing IPv6 using 
    ping6 www.google.com
  18. Try incoming IPv6 to instance IPv6 address from elsewhere and make sure ping6 and ssh to instance over IPv6 is working
  19. Validate that ping and ssh access via IPv4 elastic IP is not affected
  20. Log into older AWS account using separate browser (or private mode)
  21. Add entry in /etc/hosts of previous VM with appropriate name for previous elastic IP (eg oldcommonhosting)
  22. SSH to old VM
    1. Set correct hostname using 
      hostname oldcommonhosting
    2. Update /etc/hostname with oldcommonhosting name
    3. Exit from SSH and reconnect and verify oldcommonhosting name appears
  23. Connect to new VM
    1. Set correct hostname in /etc/hostname
    2. Set hostname for current run 
      hostname newcommonhosting
    3. Edit /root/.ssh/authorized_keys and allow direct root ssh (150x on first line)
      Also copy saurabh@labpc as authorized on new VM root account
      Also copy root@rekallcm1 as authorized on new VM root account
    4. Exit from new VM and SSH again as root without using any additional identity apart from saurabh@labpc. Verify newcommonhosting name appears.
  24. Fully update the VM to latest packages 
    yum -y update --skip-broken
  25. Create swap file as mentioned at CentOS 7.x adding swap space using file
  26. setenforce 0 on new server
  27. edit /etc/sysconfig/selinux and set SELINUX=disabled on new server
  28. Use Storing date / time along with commands in history
  29. Reboot the new VM

Refer:


Copy files

  1. Copy old servers public key as authorized on new server. Run 'ssh-keygen' on old server if there is no existing public key.
  2. Create /etc/hosts entry on old server for pointing to new server
  3. ssh from oldserver to newserver with name (eg newcommonhosting) and accept the ssh fingerprint of new host
  4. rsync /mnt/data1 from old server to new server 
    rsync -aHz --delete /mnt/data1/ root@newcommonhosting:/mnt/data1/
    Since this will take time, leave this shell running and open new root shell for previous server


Package installations

  1. yum -y install epel-release wget
  2. Copy old servers public key as authorized on new server. Run 'ssh-keygen' on old server if there is no existing public key.
  3. Create /etc/hosts entry on old server for pointing to new server
  4. ssh from oldserver to newserver with name (eg newcommonhosting) and accept the ssh fingerprint of new host
  5. rsync /mnt/data1 from old server to new server 
    rsync -aHz --delete /mnt/data1/ root@newcommonhosting:/mnt/data1/
    Since this will take time, leave this shell running and open new root shell for previous server


Copy user accounts and home folders

  1. Copy user account information to new server 
    rsync /etc/{passwd,shadow,group} root@newcommonhosting:
  2. Do not close SSH to newcommonhosting till steps complete as in between authentication can stop working and future ssh may not work till fixed
  3. Open each of the three files (passwd,shadow,group) and manually copy lines for users such as ecc,sbarjatiya to new files
    1. Also change all auth values from 1000 to 500 in various /etc/pam.d files 
      grep 1000 /etc/pam.d/*
      

      #update all files; :%s/1000/500/gc
  4. SSH to new server from a new terminal without closing existing connection and validate it is working
  5. Copy other files from oldcommonhosting to newcommonhosting using: 
    rsync -aHz /home/ root@newcommonhosting:/home/
    

    rsync -aHz --exclude ".ssh"  --exclude ".bash_history" /root/ root@newcommonhosting:/root/
    

    rsync -aHz --delete /etc/postfix/ root@newcommonhosting:/etc/postfix/
  6. Run "ls -l /home" in new server and ensure that copied passwd, shadow or group entries work as expected
  7. If ssh to new server from old server stops then due to unprotected private key error then use: 
    chmod 600 /etc/ssh/*
    on new server to fix the issue
  8. Restart postfix on new server 
    systemctl restart postfix
    

    systemctl status postfix
  9. Run following on both servers and compare to ensure all things got copied successfully 
    du -sh /mnt/data1
    

    du -sh /home
    

    getent passwd


Configure web server

  1. Install required packages on new server 
    yum -y install httpd mod_ssl php-mysql php-pdo php-xml php php-mbstring
  2. Update php version to 7.x for latest mediawiki using CentOS 7.x Installing PHP 7.x
  3. Copy web server configuration from old server to new 
    rsync -vtrp --delete /etc/httpd/conf/    root@newcommonhosting:/etc/httpd/conf/
    

    rsync -vtrp --delete /etc/httpd/conf.d/    root@newcommonhosting:/etc/httpd/conf.d/
  4. If Installing lets-encrypt SSL certificate was used copy /etc/letsencrypt from old server to new. Also copy crontab configuration (crontab -l on old server, crontab -e on new server). Also install python2-certbot-apache package on new server. 
    #On old server
    

    rsync -vaHL /etc/letsencrypt/ root@newcommonhosting:/etc/letsencrypt/
    

    crontab -l
    

    

    #On new server
    

    yum -y install python2-certbot-apache
    

    crontab -e
  5. Start and enable web server on new VM 
    systemctl start httpd
    

    systemctl enable httpd
    

    systemctl status httpd


Install and configure erlang/yaws

  1. Install erlang and yaws on new server 
    yum -y install erlang yaws
  2. Setup yaws using sbarjatiya user as follows 
    su - sbarjatiya
    

    cd ~/erlang/applications/interpreter; erlc *.erl
    

    cd ~/erlang/applications/wol_application; erlc *.erl
    

    cd ~/erlang/erlangcentral.com; erlc *.erl
  3. Edit start_yaws.sh and replace old hostname with new hostname
  4. Edit start_applications.erl and replace old hostname with new hostname
  5. Again compiled edited files 
    erlc *.erl
  6. Try to start yaws using sbarjatiya user 
    ./start_yaws.sh
  7. Verify whether yaws is running or not 
    yaws --ls
  8. exit from sbarjatiya user


Configure MySQL and migrate databases

  1. Install Mariadb server, bzip2, sshpass 
    yum -y install mariadb-server sshpass bzip2
  2. Start and enable mariadb database 
    systemctl start mariadb
    

    systemctl enable mariadb
    

    systemctl status mariadb
  3. Look at '/mnt/data1/plain_folders/documents/public_html/notes_wiki/LocalSettings.php' file for MySQL credentials 
    mysql
    

    > create database notes_wiki;
    

    > grant all on notes_wiki.* to notes_wiki@localhost identified by '<redacted>';
    

    > flush privileges;
  4. Import database backup 
    cd /mnt/data1/plain_folders/documents/public_html
    

    ./import_notes_database.sh


Configure AWStats, copy old logs

  1. Install awstats and related packages 
    yum -y install awstats perl-Geo-IP
  2. Copy awstats configuration, running data and httpd logs from older server to new server 
    rsync -aHz --delete /etc/awstats/ root@newcommonhosting:/etc/awstats/
    

    rsync -aHz --delete /var/lib/awstats/ root@newcommonhosting:/var/lib/awstats/
    

    rsync -aHz --delete /var/log/httpd/ root@newcommonhosting:/var/log/httpd/
  3. Old steps do not work, need to work on these Configure GeoLocation data for awstats: 
    cd /root
    

    wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
    

    gunzip GeoLiteCity.dat.gz
    

    mkdir /usr/local/share/GeoIP
    

    mv -f GeoLiteCity.dat /usr/local/share/GeoIP
    

    chmod -R 755 /usr/local/share/GeoIP
  4. Temporary new steps for GeoIP 
    #On new server
    

    mkdir /usr/local/share/GeoIP
    

    

    #On old server
    

    rsync -vtrp /usr/local/share/GeoIP/GeoLiteCity.dat root@newcommonhosting:/usr/local/share/GeoIP/
  5. Restart apache 
    systemctl restart httpd
    

    systemctl status httpd


Make new VM primary by updating DNS

  1. Change DNS as follows on godaddy.com:
    • rekallsoftware.com :: @
    • energyconservationclub.in :: @
    • erlangcentral.com :: @
    • pbarjatiya.com :: @
    • sbarjatiya.com :: @
  2. Ensure SPF of all domains has a:mail.rekallsoftware.com
  3. Shutdown old VM (Do not release elastic IP yet)
  4. Ping above domains and look for new IP. If old IP is shown try 
    dig -t any sbarjatiya.com
    or +trace option
  5. Check following URLs:
  6. Send email to saurabh@sbarjatiya.com, saurabh@energyconservationclub.in
  7. Release elastic IP from old VM. That may require filling rDNS removal form: https://console.aws.amazon.com/support/contacts?#/rdns-limits
  8. Request rDNS mapping for new elastic IP with FQDN by filling form at https://aws.amazon.com/forms/ec2-email-limit-rdns-request?catalog=true&isauthcode=true for new elastic IP with name mail.sbarjatiya.com
    Use following text for reason while mapping
    Emails for various domains such as pbarjatiya.com, sbarjatiya.com, energyconservationclub.in, etc. all of which are hosted on the server with elastic IP <new-elastic-IP> are routed via this server. There is no email storage (IMAP/POP3) service. Only emails received for the above domains are forwarded to appropriate gmail IDs via postfix virtual alias.
    Note the following for ensuring that no SPAM is generated from this server / elastic IP:
    1. No email is generated / sent directly from this server. Only incoming emails to domains such as @sbarjatiya.com are forwarded to appropriate gmail IDs.
    2. Emails for only five domains (rekallsoftware.com, sbarjatiya.com, energyconservationclub.in, pbarjatiya.com, erlangcentral.com) are accepted. No other emails are accepted. This is not an open RELAY.
    3. There is no user login on the server for sending emails. (no SMTP auth, no HTTP/HTTPS for web access to emails). Hence there is no question of this server getting compromised and attacker sending email via this server. Only SMTP/SMTPS services are there to forward emalis of five specific domains listed above to gmail IDs.
    4. All outgoing forwarded emails go only to one of three given gmail IDs
      • jain.priyanka0508 [at] gmail.com
      • pbarjatiya [at] gmail.com
      • barjatiya.saurabh [at] gmail.com
      There is no other address where emails are forwarded from this server.
  9. Update ssh known_hosts keys on rekallcm1 for sbarjatiya.com and www.sbarjatiya.com for both saurabh and root users
  10. Update any KB article on rekallcm and test following as root user: 
    /documents/public_html
    

    ./update.sh
  11. Take one full backup.


Configure logwatch

  1. Install required package using: 
    yum -y install logwatch
  2. Edit /etc/aliases and add alias for root as 
    root: barjatiya.saurabh [at] gmail.com
  3. Update alias database using: 
    newaliases


Update VM information excel file

Update AWS VM information excel file and create required billing alerts etc.


Allow outgoing emails via mail.rekallsoftware.com

If required temporarily till EC2 sending limitations are not lifted route emails for this via other email server.

  1. Ensure SPF of all domains has a:mail.rekallsoftware.com
  2. ssh to mail.rekallsoftware.com
  3. Edit /etc/postfix/main.cf
  4. Add new elastic IP to mynetworks on mail.rekallsoftware.com
  5. Restart postfix on mail.rekallsoftware.com
  6. Allow new elastic IP of sbarjatiya.com in mail.rekallsoftware.com for port 2525
  7. Edit /etc/postfix/main.cf on newcommonhosting and add 
    relayhost = mail.rekallsoftware.com:2525
  8. Restart postfix on newcommonhosting
  9. Send test email and confirm emails are getting delivered


<yambe:breadcrumb>New_machine_configuration|New machine configuration</yambe:breadcrumb>

Retrieved from "http://www.sbarjatiya.com/notes_wiki/index.php?title=Migrate_sbarjatiya.com_VM&oldid=4666"