At our company we manage 100+ Linux computers remotely. Those are mostly clients for our ERP application, and sometimes you simply need to log in to fix something or help the user. Most of them are behind the firewall. In the past, we always had a deal with client’s IT staff to open a certain port on their firewall and forward it inside to SSH port at our machine. This works nice, but there are cases when IT guys have a hard time setting it up, or when ISP is simply blocking any possibility of doing so.

Last year I managed to set up reverse SSH to work around this. How this works? Basically, you need to have one publicly accessible server. The remote client logs into it using SSH and then opens a TCP port locally (on the server). After that, you can ssh to that port on the server machine and it tunnels back to ssh server on the remote workstation.

This was easy to set up manually, but we need a permanent connection. You can place the ssh command in some script at the client and make sure it runs, but there are times when this does not work so robust. Especially over mobile (3G, GPRS, EDGE) connections SSH session gets dumb and although it looks alive there it does not send any data back or forth.

Enter autossh. This great program starts the tunnel (no need to remember all the parameters to ssh client) and makes sure it stays up. Every 10 minutes (configurable) it checks if connection is still alive, and restarts it if data cannot be sent.

I’m using Git to track source code changes and I host most of my websites on servers where I have SSH access. I never liked ftp much, so I’m using SCP to upload all the files.

One of the main tasks is to upload the source code files that have changed since the last release. To do this, I first use gitk to find the last revision that got uploaded to the server and then diff the changes. For new files and directories, the script also creates them on the server. So, I would use script like this:

source difference.sh 12edf98

where “12edf98” is a version hash from git. Short version of “source” command is dot (.), so this is the same as above:

. difference.sh 12edf98

The difference.sh script creates a script that can be sourced again to actually make it run. So “difference.sh” will show what would be done and “upload.sh” does it. This is the content of “upload.sh”

eval $(source difference.sh $1)

I run it like this:

. upload.sh 12edf98

And here’s the content of difference.sh (watch the line wraps):

echo echo Creating new directories if required ";"
for i in `PAGER=cat git log --reverse --pretty="format:" --name-status HEAD...$1 mysite.com | grep A | cut -c44-1000 | grep / | sed "s5/[^/]*\\$55" | sort | uniq`; do echo echo $i ";"; done
for i in `PAGER=cat git log --reverse --pretty="format:" --name-status HEAD...$1 mysite.com | grep A | cut -c44-1000 | grep / | sed "s5/[^/]*\\$55" | sort | uniq`; do echo ssh mysite.com mkdir --parents /var/www/html/mysite.com/$i ";"; done
echo echo ------------------------------ ";"
for i in `PAGER=cat git log --reverse --pretty="format:" --name-status HEAD...$1 mysite.com | grep -e "[A|M]" | cut -c44-1000 | sort | uniq`; do echo scp mysite.com/$i mysite.com:/var/www/html/mysite.com/$i ";"; done

All the website files are stored in directory called “mysite.com” and scripts are one directory above (same level as mysite.com directory).

I lost a lot of time finding information on the Internet to make all this work, so I hope it will help someone.