CentOS 8.x Systemd based reverse ssh tunnel service

Home > CentOS > CentOS 8.x > CentOS 8.x Security tools > CentOS 8.x Systemd based reverse ssh tunnel service

In case anydesk/teamviewer are proving unreliable there is option to connect to an internal server behind NAT using reverse ssh. For this we create service on internal server which does SSH to a public server via public key and creates reverse ssh tunnel from public server to itself. Anyone who has SSH access to public server, can now also connect to the internal server via this reverse tunnel. Thus, internal server without public IP can be accessed from anywhere (Internet) provided the user has access to public server where reverse tunnel is created.

To setup above use following steps to set it up:

1. Assuming not only SSH but graphical access is also required install nomachine on both source and destination machines. See CentOS 8.x nomachine

   This is optional. The steps would also work only for SSH reverse tunnel without nomachine setup.

2. Create ssh-keys on internal server using:

   ssh-keygen

3. Configure the public key of internal server as authorized on public server using appropriate mechanism.
4. Connect to public server from internal server using below command:

   /usr/bin/ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=1 -R <remote-ssh-port>:127.0.0.1:22 -R <remote-nomachine-port>:127.0.0.1:4000 root@<server-ip-or-fqdn> -N

   Here replace <remote-ssh-port>, <remote-nomachine-port> and <server-ip-or-fqdn> apporpriately.

   If you are not doing nomachine reverse setup then ignore that particular "-R <remote-nomachine-port>:127.0.0.1:4000

   The output of above command should be blank. In case of tunneled SSH there might be some output lines related to use of proxy in above command.

   In case the command has output, we need to modify script ( /root/helpdesk-reverse-ssh.sh explained later) as follows

   status=$(ssh -o BatchMode=yes -o ConnectTimeout=30 root@$SERVER echo ok 2>&1 | grep -v "<content>")

   to filter the content lines. This way $status would be ok during the ssh test part of script.

5. Leave the ssh connected and in another tab check output of:

   ps aux | grep root@<server-ip-or-fqdn> | wc-l

   The above command output is 2 in CentOS 8 and the below script/steps are based on this value.

   The same command output is 3 in case of CentOS7. In that case in following shell script we should use '"if (( "$COUNT" > 2 )); then"'

6. Create /etc/systemd/system/helpdesk-reverse-ssh.service with following contents on internal server:

   [Unit]
   Description=helpdesk-reverse-ssh service
   After=network.target
   
   [Service]
   Type=simple
   ExecStart=/bin/sh /root/helpdesk-reverse-ssh.sh
   Restart=always
   User=root
   Group=root
    
   [Install]
   WantedBy=multi-user.target
   

   No need to redirect output or capture Pid of processes. All of that is taken care of by systemd.

   Replace helpdesk-reverse-ssh with appropriate name based on your environment'

7. Make service executable

   chmod 744 /etc/systemd/system/helpdesk-reverse-ssh.service

8. Create shell script /root/helpdesk-reverse-ssh.sh on internal server which does reverse SSH tunnel to remote server using:

   #!/bin/bash
   
   SERVER="<service-ip-or-fqdn>"
   RPORT_SSH=3001
   RPORT_NOMACHINE=3002
   
   while sleep 60; do
      #Count number of existing connections
      COUNT=$(ps aux | grep root@$SERVER | wc -l)
      if (( "$COUNT" >= 2 )); then
          echo "SSH already connected, not doing anything extra"
      else
          # Check SSH Connction is available or not	
          status=$(ssh -o BatchMode=yes -o ConnectTimeout=30 root@$SERVER echo ok 2>&1)
       
          #If available connect to remote server
          if [[ $status == ok ]] ; then
               echo "Going to start new SSH connection in background"
               /usr/bin/ssh -o StrictHostKeyChecking=no -o ServerAliveInterval=60 -o ServerAliveCountMax=1 -R $RPORT_SSH:127.0.0.1:22 -R $RPORT_NOMACHINE:127.0.0.1:4000 root@$SERVER -N &
          elif [[ $status == "Permission denied"* ]] ; then
              echo "Not able to connect due to permission denied error"
          else
              echo "Error: $status"
       fi
   fi
   done     
   exit 0
   

   Here incase of CentOS 8 the condition should be >=2, while in case of CentOS 7 it should be >2. This is because of CentOS 7 while doing ssh "ps aux | grep root@<server>" results into three lines of output. On CentOS 8 same command results into only two lines of output.

   Values of RPORT_NOMACHINE, RPORT_SSH can be changed, if required

   <service-ip-or-fqdn> should be replaced with public server details

   If nomachine or SSH are not listening on their default ports 4000, 22 then replace those port numbers in -R argument value appropriately.

9. Set required permissions and start service

   chmod 744 /root/helpdesk-reverse-ssh.sh
   systemctl daemon-reload
   systemctl enable helpdesk-reverse-ssh
   systemctl start helpdesk-reverse-ssh
   systemctl status helpdesk-reverse-ssh

10. Now watch for the command to get executed after 60 seconds. Also after 120 seconds ensure that command is not executed again if ssh is running using:

    watch "systemctl status helpdesk-reverse-ssh"

    You can see systemd keeping track of shell script, sleep as well as ssh command

    If SSH fails then the error output would be part of status command logs lines shown at bottom

11. Now open connection from a different client machine Not internal server to public server using:

    ssh -L 9000:127.0.0.1:3002 root@<server-ip-or-fqdn>

    You can use any other local port instead of 9000 in above command.

    In the script if you have changed RPORT_NOMACHINE then use that value instead of 3002 in above command

12. Connect using nomachine to 127.0.0.1:9000 using nx protocol.

    Ignore warning about infinite loop while connecting to localhost and proceed.

There is older article on this at CentOS 7.x Systemd based reverse SSH service

Home > CentOS > CentOS 8.x > CentOS 8.x Security tools > CentOS 8.x Systemd based reverse ssh tunnel service