HostingGeek

One of the greatest features of SSH is the ability to use key based authentication. Key based authentication uses the public/private key method to allow logins to SSH without the use of passwords.

This method is better than password based authentication as a password can be brute forced. While it is theoretically possible to brute for a public/private shared key combination, the amount of computing power is pretty stagering, as well as the amount of time invloved to accomplish this using standard computers today. (Quantum computing is said to be able to accomplish this same feat in seconds… But as of yet quatum computers don’t exist outside of the lab that are large enough to accomplish this… so we are safe… for now.)

It works pretty simply. You create a private and public key pair for your login. You can do this by running:

ssh-keygen -t dsa

This will generate a public and private key using the DSA method (you can also use RSA).

You will be prompted on where to store the key pairs, as well as for a passphrase. The passphrase will protect the private key from unauthorized usage, but also negate any advantage of using keys to authenticate automattically without passwords (there is a way to do this but it is outside the scope of this current discussion and will be another blog entry). I would recommend skipping the passphrase for the time being.

Once you have created your public and private key pairs (in your home directory in the .ssh directory generally). You will need to copy the public key to each server you wish to authenticate with, and not have to type a password to do so. You can use the scp method (covered in a previous post) to copy the files to the remote server under the username you login in the .ssh directory in your home folder (so for user xyz it would be /home/xyz/.ssh). After you have copied that file to the server, you will need to copy that file to the file authorized_keys. You can do that by running:

cat /home/xyz/.ssh/dsa.pub >> /home/xyz/.ssh/authorized_keys

Now to test it, type in ssh xyz@remoteserver and you should be logged in right away with no password prompt. If you didn’t do it right, then you will be prompted for a password.

Now I have to mention… Because we aren’t usnig a passphrase for your private key… if somebody were to get control of your private key, they can now login to all the servers that use that private/public key combination. So keep that in mind.

I use key based authentication for most of my servers, because I have gotten so tired of password based brute force attacks on my SSH daemon. I keep my private keys on a keyfob on my keychain. So if I have my car keys… I have access to my servers. I also have the private keys in another safe place… and if I told you where that was… I would have to kill you… So lets leave it at that.

Recently while working on a server, I noticed that there was some unusual files in the /tmp directory. This always sends up red flags in my head so I investigated more closely and determined that somebody had placed a file on the server via a PHPBB exploit and was using it as a means of building a zombie network.

Typically this type of activity is closely linked to “script kiddies” and not legitmate hackers. Script kiddies are people with no more hacking ability than anybody else, they simple know how to read about holes in certain software and use other legitmate hackers work to exploit those holes, to some unknown end.

This case was no different. Here are some fun things you can do to mess with them. Many times the script kiddies will leave the software on the server, and that itself is a goldmine of information.

Case in point, I found a binary on the server that was connected to an IRC server. I did a quick review of the process list (ps -auxwww) and determined the process id of the application running. Then I ran:

strace -s 16000 -p {process id}

This command is called stack trace and will attach to an already running application to see what it is doing. This is particularly useful if the binary in question is no longer on the hard drive.

So I attached to the process and was able to determine that it was connected to an IP address on port 6667. Port 6667 is one of the default ports for IRC. So I fired up my trusty IRC client and connected to it. Sure enough I was connected to a server with 106 clients and 3 operators.

So now I had to figure out what channel to join. Lucky for me the binary was still on the hard drive. Here is where the second part comes in. Because script kiddies only use the software and they don’t know how it actually works, we can often get more information from the binary. So to glean information from the binary I ran:

strings {filename}

This printed out a long list of text based strings in the binary. The best part is, the IRC channel that the zombie was supposed to join on the IRC server, and the password for the IRC channel, were right there in front of me. So… I joined the channel.

I must have scared the living hell out of the operator of the zombie network, because as soon as I started talking to him, I was firewalled off of the server completely. All new connections coming into the server were also blocked. Luckily I was able to collect a list of IP addresses of the zombies in the channel before I was disconnected, and I have been systematically notifying the operators of those machines that they have been comprimised.

I am sure somewhere in the world… a script kiddie had to go clean out his pants after that episode. I hope I put a dent in his “zombie network”.

Armed with some of this information I hope that you will be able to torment script kiddies when you encounter them. Not all of them are this easy… But when they make it easy… I say take the time to mess with them… If nothing else it may scare them just enough to stop for a little bit.

I was looking for a way to generate some random password in a shell script I was writting and I stumbled across a handy little program: pwgen

According to the pwgen man page:

pwgen generates passwords which are designed to be easily memorized by humans, while being as secure as possible.

The pwgen program is designed to be used both interactively, and in shell scripts. Hence, its default behaviour is(sp) differs depending on whether the standard output is a tty device or a pipe to another program. Used interactively, pwgen will display a screenful of passwords, allowing the user to pick a single password, and then quickly erase the screen. This prevents someone from being able to “shoulder-surf” the user’s chosen password.

So there you have it, a nice quick way to create a password. And just a sample of how I used it in my shell script:

CLEARPASS=$(pwgen -N 1)

This returns one somewhat random password to the shell variable CLEARPASS.

Tomorrow I hope to have a nice write up on how to do password-less authentication via SSH shared keys.

Background: We host a very large organization that has chapters in all 50 states (of the United States that is). We choose to host these sites under Plesk because it offloads some of the more mundane support requests to the users, and trust me that is a good thing.

All the sites are in the format .somedomain.com. Now, we also manage the codebase for the content management system for these sites. It is my job to set these up. It is boring and I might throw myself in front of a large bus if I had to do this all by hand.

Enter one well crafted shell script:

for i in `find /var/www/vhosts -type d -name “*.somedomain.com”`; do FOO=$(cat /etc/passwd |grep $i| sed ‘s/:.*//’); cd $i; rm -rf $i/httpdocs $i/sql $i/.svn; svn checkout https://svn.somedomain.com/svn/someproject/trunk ./; chown $FOO:psaserv $i/httpdocs; chown -R $FOO:psacln $i/httpdocs/*; done

Here is what this one does: Since we already have all the domains on the server (and this is a Plesk server) we are just going through the list of sites in the /var/www/vhosts directory and working with each of them. Now Plesk (particularly the latest version 7.5.x) can be very picky about ownership and permissions (especially on the httpdocs folder), so we need to correct them as we go along.

Also it should be noted that all of the sites were previously under SVN control, however that SVN was removed due to the wrong codebase being imported into the SVN originally. So…

This loop goes through and takes the full path for the domain and places it in the variable $i. We then parse the passwd file to find out which user owns that directory (important for FTP reasons) and place that information into a bash variable. Then we remove the previous httpdocs, .svn, and sql directories since SVN will not export if a directory of the same name already exists. Then we do an SVN checkout of the code to the current directory. Then using the previous bash variable we created by parsing the passwd file ($FOO) we correct the ownership of the httpdocs folder and the files contained within.

Simple huh?

I hope to write up a simple howto in the near future on how to manage your website using SVN. It is extremely handy when dealing with website revisions as well as revisions to many sites that use the same code.

OpenSSH has some great programs that come bundled with it. One specifically I use on a regular basis is scp which stands for Secure CoPy. And it does exactly what its name says it does. It copies a file securely via an SSH tunnel between two machines (that both have SSH running).

You can use scp to copy a file from the a local directory to a remote directory, or a remote directory to a local directory, or even from a remote directory to another remote directory, which is pretty handy for the lazy.

Here is an example of how to use scp to copy a local file to a remote server:

scp ./somelocal.tar.gz remoteuser@some.remote.host:/path/to/where/you/want/the/file

Pretty handy… Now here is how you do a remote file to a local directory using scp:

scp remoteuser@some.remote.host:/path/to/file/filename.tar.gz ./

Simple. Now lets go a step further… Remote to remote file copy with scp:

scp someuser@some.remote.host:/path/to/file/filename.tar.gz anotheruser@another.remote.host:/path/to/where/you/want/the/file/

SSH is more than just a replacement for Telnet, it is a mutlipurpose suite of tools that will make your life easier once you become familiar with them.

I have a customer that I am supposed to be migrating from one server to another server due to their current server being hacked (PHPBB hack… it really is sad how easy it is to do this… and even more so how easy it is to prevent it).

Anyways, I attempted to migrate the server earlier this week, but the install of the latest version of Ensim on the new server went horrifically wrong (in case you are wonder Ensim is the WORST control panel software for webhosting ever created. It is slow, poorly documented, poorly supported, and the company uses all of these facts as a profit center to charge you $150 per hour for anything that goes wrong in their shit hole software — but I digress.)

Long story short, I spent about 8 hours on trying to get Ensim to install after the previous install went down in a giant flaming fireball of crap and python exception tracebacks. I threw in the towel and told one of the other techs to reimage the drive and give me another server…

First there was a problem with the drive images, so they had to be corrected. Then they took the servers to the data center and put them online… only nobody bothered to check to see if they were actually online or not… and of course… they weren’t. So…

They finally got the servers online today… Cool! I am getting ready to get started on my Ensim install, but I am going to take every precaution known to man to prevent the massive shit storm that happened last time. So I set the hostname of the server, setup the hosts file (/etc/hosts) to point to the local IP address for the server. I even threw a reboot at the server, just to make sure everything was all set. So I reboot the server and wait. And wait. And wait… The damn thing never came back online. I am furious.

I tried everything I could think of, accessing it via the router, neighboring servers, changing the vlan of the ports to see if it was a routing issue. NOTHING.

I would hard reboot the server is somebody had bothered to tell me which power switch port the server was on, but I don’t even have that. So here I sit… 3:00am in the morning. With my proverbial thumb up my ass.

Okay, well we will give this another try tomorrow, but for now bed is calling.

SSH impresses me each time I play with it. It is so powerful, that most of the time (90%) of the time, you aren’t using it to its greatest potential.

Here is a neat trick for creating a SSH tunnel to forward TCP based packets (this doesn’t work for UDP packets) between two servers on the internet.

You run this on the server who’s port you want to forward.

ssh -C -N -f -L localport:thelocalhostip:remoteport remoteuser@remotehost1

Here is what that will do: This will open a tunnel between localhostip:localport to remotehost1:remoteport.

What can you do with it? Well lets take a real world example. Lets say we want to forward all SMTP traffic from one server in the datacenter to another server on the other side of the world. You can do:

ssh -C -N -f -L 25:192.168.1.69:25 root@1.2.3.4

You will be prompted for a password if you don’t have authorized keys setup between the current server and the remote server (a subject for another post).

What you should be able to do is telnet to 192.168.1.69 25 and see the greeting from the SMTP server running on port 25 on 1.2.3.4.

I only used SMTP as an example, I know that there are several MTA based options that can do the same thing. However this is a good option to help make sure mail keeps flowing from an old server to a new server, if you didn’t set your TTL to a low setting when migrating servers (not that I would ever forget to do anything like that 😉 ).

You can also string more than one port on the command by adding additional -L localport:localip:remoteport.

As an added benefit this also encrypts the information between the two servers… So an added layer of security to things. Also the -C option enables compression on the tunnel for additional savings in bandwidth… See I told you SSH was cool.

Here is another thing I encountered just now… when you use cat to read the contents of a file into a bash variable, it tends to lose its formatting with out a really good way to preserve that formatting. To preserve the formatting from a catted file inside a bash variable use the following:

VAR=$(/bin/cat -E /path/to/file/filename.txt)

The -E option for cat will place $ for each end of line, then use this to convert them to something we can use:

NEWVAR=$(echo $VAR | sed ‘s/$ /\\n/g’)

This will replace the $ and the [space] that follows it with \n, then we use echo’s -e option to convert the \n’s to line feeds.

echo -e $NEWVAR > /path/to/file/newfilename.txt)

There is another option for cat (-A) that outputs newlines as $ and tabs as ^I, however I am unable to pattern match the ^I because I can’t figure out the key combo to make a match to it. (CTRL+V then CTRL+I and CTRL+V then TAB do not produce a ^I).

From time to time in my day, I need to do a bunch of edits quickly and easily, so I make up these quick little one line bash scripts to do this for me. From time to time I will post them here so that other might be able to bask in the glory that is a handy dandy “oneliner”.

So… Lets kick this blog of the proper way, with a “oneliner” bash script that will parse all the files in the current directory and rename them with a new name.

for i in `find ./ -name “*template*” -type f`; do mv $i `echo -n $i | sed ‘s/template/quickform/’`; done

Pretty simple huh? What does it do I hear you ask? Well lets break it down (remember that “ or backticks causes everything inside the back ticks to be executed).

for i in `find ./ -name “*template*” -type f`; goes out and searches the current directory for any file (-type f) who’s name contains template and then passes each returned result and places its value in the variable i.

do mv $i `echo -n $i | sed ‘s/template/quickform’`; do is executed each time the value of $i changes from the previous for statement. What this does is executes mv $i and then we echo out the current value of $i to sed via a pipe (used to pass information from one application to another) to sed’s search and replace command (s/needle/replacement) to change the word template in the variable $i with quickform.

done just tells bash that the script is over and to start on the next loop if there is one.

That is all for now… stay tuned for cool things you can do with SSH.