HostingGeek

What fun things did I learn today? Let me see…

• Intel based ICH7 chipsets are not fully supported in the linux kernel until kernel version 2.6.11. With additional features added in 2.6.15.
• The Debian Installer is based on the 2.6.8 kernel, and therefore can not install on to hard drives attached to the ICH7 chipset (at least in SATA2 mode, which is the whole point).
• The northbridge chipset on the MSI 945GM2 motherboard is OMG HOT! (I have the burn marks to prove it.)
• Motherbards are moving to a new auxillary power connector (in addition to the ATXv2.0 spec). This new connector is called an EPS12V connector and consists of an 8 pin connector for 12V power to the CPUs.
• The EPS12V power connector seems to be standard on most “Cedar Mill” (Intel’s new 65nm fab CPUs) “Ready” motherboards. Where as most power supplies do not yet have this connector.
• The case I buy do not have the EPS12V connector on them – which made me sad.
• The standard 4 pin 12V connector that is standard now on most power supplies DOES work in the EPS12V plug – which make me happy.
• SATA2 (SATA 3.0Gbps) can be used in “comptability mode” in the BIOS, but it makes them SLOW.
• The default kernel installed with Debian doesn’t support over 1GB of RAM. You must install another kernel if you have more than 1GB of RAM.

I think that about covers what I ran into today. Just some random facts that might help somebody some day.

Let’s face it, moving from one server to another server is nasty and something any of us would try to avoid as often as we can. Unfortunetly, there are instances where we can not avoid this and we must bite the bullet and do it.

Here are some tips that I have gathered up that will help you maintain your sanity as much as possible. Honestly even with all the planning and preperation, there will be issues so just prepare yourself mentally for that. It will happen.

So here are some things you can do to help the transition:

• DNS Caching: DNS has to be one of the most difficult things to deal with, because you only have a certain level of control over what occurs. Caching of DNS results will often cause many problems. The trick here is to plan ahead and remove the caching from the picture until the move is completed.

  Setting your DNS to a low TTL (I use 5 seconds versus the recommended 86400 seconds/ 24 hours). The TTL tells other DNS servers and clients how long they should hold the information they just received, in their cache. Since we are going to be moving the site to a new IP soon, we don’t want them caching the old IP once the site is on the new IP. The low TTL will help prevent this.

• Pathing: If you are moving to a server with different paths, you should at least try to setup symbolic links to help any hard coded paths in the site’s content.

• PHP register_globals: This one has bitten me a couple of times. Most ofthen when you move to a new server the server will have the default settings for PHP which turns off register_globals. This will cause all sorts of problems that you won’t be able to track down easily. Check it early on and make sure that both servers match settings wise. (A a general rule register_globals should be set to off, but there are some legacy PHP apps that need it.)

• SSH Tunnels: Most of the time you should adjust your TTL for your DNS long before you do the actual move, but sometimes that isn’t an option. In a pinch you can use an SSL tunnel to redirect traffic from your old server to the new server. This will help users that have the old IP cached, still access the new site. It should be noted that SSH tunnels are pretty unstable and shouldn’t be really be depended on, but like I said, they are good in a pinch. Here is an example of one, you run this on the old server as root with the httpd process stopped:

  ssh -g -C -N -f -L 80:newserverip:80 root@newserverip

  This will prompt you for a password (if you don’t have key based access already setup). You will get a warning ‘bind: Address already in use’, but the process is working. This message is just letting you know that the remote endof the tunnel has something already on port 80, which is actually a good thing.

  You can read more about this process in my blog post Redirecting TCP Ports with SSH

• DNS: Once you have everything moved over to the new server, update the current authoritative DNS servers with the new IP as soon as possible. Once you have done that change the domains name server records also. The name server change takes longer to complete (will normally be done in 8 hours at the root servers).

• How To Deal With a Highly Dynamic Website: For sites that have alot of content, or for sites that have users contributing data often (mainly forums) you need to draw a line in the sand and take the site offline during the move. This ensures that you have all the content and that no one loses any content that was contributed during the move.

• Rsync: Also in the same vein as dynamic sites (user images etc) rsync is your best friend. Run rsync, and run it often. Run it even before you move. It keeps the two sites in sync with each other and can be run while the old site is still up and running. The more you run rsync leading up to the actual move, the faster the actual move will occur. Faster move == less downtime for your users.
  

There are many other issues to consider, but if you take these items into consideration you should not have any major surprises coming your way.

So today I came across this site for Apple’s Boot Camp.

Basically the concept behind Apple’s Boot Camp is to allow users to install Windows XP side by side with MacOSX, which many people have been trying to do since the first Mac’s shipped with Intel based processors.

So now it is possible to run Windows XP on a Mac with Apple’s blessings.

Here is where things get interesting: Throw Xen 3.0 into the mix. For those that don’t know what Xen is, it is a hardware virtualization technolgoy (based ont he linux kernel) that allows multiple OS’es to run on the same hardware at the same time. It allows concurrent access to the network, hard drives, memory, cpu etc. Prior to Xen 3.0 you needed to have an OS specifically compiled for Xen in order to run it. That has all changed with the new technology included in the latest processors from Intel (and soon AMD – June 2006 I think). The new processors have a technolgoy called Virtualization that allows hardware virtualization applications like Xen (and VMWare) to run OS’es that are not compiled natively for Xen itself (such as OSX and Windows XP).

This is where I think there is a huge opportunity for Apple (and for all I know they might actually be working on this already, I haven’t a clue). To allow Xen to boot OSX as a Xen Dom0 (the OS that is allowed access to the Xen kernel for control over the other OS’es on the virtualized machine). Then boot Windows XP as a DomU. Then a user would be able to boot their computer, access OSX and switch to Windows XP all in one machine.

An interesting thought… If you can get both OS’es to boot at turn on, and used OSX as the Dom0, then you would be able to access the Windows XP virtual machine via the Mac OSX RDP (Remote Desktop Protocol) client and never have to actually leave OSX or figure out how to switch between the two. Oh man… my head spins… I need to get a MacBook Pro and start working on that.

Today was a fun day. I have a buddy that has spent the last 3 or 4 months (maybe longer) working on a complete ground up redesign of a PHP/MySQL website. The current code base was horrid, and that is putting it lightly. The code was so poor that it was hampering the overall performance of the server. The pages were very slow to display, often taking 30-45 seconds to complete the rendering of one page. Looking at the top process list I could see that several httpd processes were dying off, and dying off frequently. I had assumed that this was due to the poor codebase of the existing application.

The launched the new codebase today, and things didn’t improve. So my buddy called on me to help him out. First a little bit of background on the server: Dual Athlon (yes it was weird to me too), 1GB of RAM, and 1 IDE HD (again, yes I know who ever speced out this server should have their head examined). Server is running Cpanel 10.8.1-C112 under Fedora Core 2. We had rebuilt apache already (using easyapache) and the httpd zombies kept occuring. They weren’t true zombies, only momentary zombies, because the child process was dying without permission from the parent process, so they only showed up in the zombie count for a couple of seconds, so they were being cleaned up properly.

Here is what we found: the main site for the server is a heavily trafficed PHP/MySQL based site with a lot of images. I started looking through the error logs and came across a bunch of messages like:

[notice] child pid 10009 exit signal File size limit exceeded (25)

After some quick googling we were able to determine that this message means that the child httpd process died, because it encountered a file that is too large for it to handle. The max file size is 2GB in this instance. There is some discussion on if this is an Apache 1.3.x limitation or a limitation a little further down (glibc or filesystem).

How did we fix it? Well short term was to shutdown apache and move the offending log file out of the way, and restart apache. Longer term fixes involve some settings inside Cpanel itself. You basically need to tell Cpanel to remove the log files after it has processed the log files.

Log in to WHM and click on “Tweak Settings”, then check mark “Delete each domain’s access logs after stats run”

That will delete the log files daily, and all should be well with the world.

Recently I did a Xen 3.0 install on a Dell 1750 server with RAID5. By default the Xen 3.0 installation (from source) doesn’t compile in the LSI drivers for the RAID card, so I had to enable the option in the Dom0 kernel in Xen.

So I went in to the xen0 directory under my Xen source (/usr/src/xen-3.0.0/linux2.6.12-xen0) and ran:

make ARCH=xen menuconfig

I then went to Device Drivers -> SCSI Device Support -> SCSI low-level drivers -> and enabled the “LSI Logic New Generation RAID Device Drivers”, because after all… Newer is better right?

I recompiled Xen and the Xen kernels (cd ../; make kernels; make install) and reboot the machine and went on with my day.

I got a call from the client later in the day that the server was reporting a problem with the RAID array. Long stroy short, the customer was running a NAGIOS plugin to monitor the RAID array via the /proc/megaraid entry. Little did I know that the new generation RAID device drivers in the 2.6 kernel no longer make use of the /proc/megaraid proc entry, so it was breaking the NAGIOS plugin.

To correct this problem use the “LSI Logic Legacy MegaRAID driver” instead of the “LSI Logic New Generation RAID Device Drivers” in the 2.6 kernel (this is not an option in the 2.4 kernel). That will give you back the /proc/megaraid entry and you should be all set.

Additionally, since you are already compiling a kernel, you should enable the kernel option “Use register arguments” under “X86 Processor Configuration”. This option will allow you to use the Dell OMSA device drivers for additional control over your Dell PowerEdge server. You can read more about it here: http://www.uta.fi/~pauli.borodulin/dellomsa/omsa44.html

I have many servers that I have to watch on a regular basis. We use a KVM to switch between the consoles of the servers and generally we only see the console for any given server for a couple of seconds.

The KVM we use has this great feature where if it detects any activity on the console, the KVM will swtich to that console and display the output for us.

This is how we use that functionality to quickly display any problems a server might be having. It is by no means a fool proof and accruate, but it does help to bring things to our attention on a regular basis, doesn’t require you to install anything special (all programs should come with a basic install of a linux OS), and its cool… What else do you really need?

What we do is one turn off the console’s screen saver. It is evil, and we don’t like it. Actually it serves a valid purpose, but not for our needs so lets turn it off:

/usr/bin/setterm -blank 0

Now you can put that into /etc/rc.local for RedHat based distros (Fedora, Redhat Enterprise Linux, CentOS, etc).

For Debian you can create a file in /etc/init.d/ and name it blank_screen and chmod 755 the file. Then just create a symbolic link to the startup dir:

ln -s /etc/init.d/screen_blank /etc/rcS.d/S61blank_screen

Now that we have the console set not to turn off, we can do the next part of our monitoring mojo. We will setup to monitor files that are important to us, such as /var/log/messages, and /var/log/syslog. These files might not be the files you want to monitor, so you can modify the line below to match the files of your choosing.

We use tail, because it displays the lines as they are added to the file we are monitoring, and it is simple and lightweight.

/usr/bin/tail -v –follow=name /var/log/messages –follow=name /var/log/syslog > /dev/tty2

What this does is runs tail to monitor the files /var/log/messages and /var/log/syslog. We monitor the filename, rather than the descriptor (which is the default behavior of tail). Monitoring the descriptor means that if the filename is changed — for example logrotate, then we will follow the file to the new name. Since once a file is rotated by logrotate it won’t be updated, we don’t want this behavior. We add the -v (verbose) so that tail will tell us the name of the file that corresponds with the output. Then we redirect the ouput to /dev/tty2 (virtual console 2).

This will output the information directly on the screen so that we can see it via the KVM.

Additionally you might want to turn off the login prompt on the virtual console 2. We can do this by editing the /etc/inittab file and commenting out the following line:

2:23:respawn:/sbin/getty 38400 tty2

You can just put a # in front of that line and then have init re-examine the file by running:

kill -HUP 1

So there you go… you now have the output of any file you want to the console of your server. Set your KVM to autoscan, and enjoy your easy to view logs.

Every once in a while I come across the need to access data that is stored in a database inside MySQL from a shell script. Here is a quick and dirty way to access that information without having to resort to PHP or Perl.

Run a SELECT:

By default this would return output like:

id
514

So we need to use sed to remove the name of the field.

SITEID=$(mysql -D database -u root –password=password –silent –exec=”SELECT id FROM domains WHERE name = ‘`echo $SITE`’;” |sed ‘s/id//’)

Since INSERTs, UPDATEs and DELETEs don’t return any information you won’t need to pipe the information returned from MySQL to sed.

This handy tip will work for basic MySQL queries, but for more complex queries, I would recommend something with a little more control over the information such as PHP or Perl.

I hope that helps somebody.

Let me introduce you to my new favorite piece of open source software: Xen.

Xen is a virtualization operating system that allows you to share hardware between multiple OS’es. The concept isn’t really new, as there are several commercial as well as open source implementations of this concept already.

In the past I have used Linux-Vserver which is a set of kernel patches that create security contexts inside the kernel that different versions of Linux can co-exist in. It works very well and I have been using it for over 2 years in production. The main drawback is that it only works with Linux based OS’es. The same with Jail in FreeBSD which I have also used for a while, but again is specific to the OS, in this case FreeBSD.

Now I have never used VMWare, but VMWare uses a “core” application that the guest OS’es run in. This works very well with all current OS’es from Windows, Linux to *BSD. I have heard that there is some performance degradation with this method, but I have never used it so I don’t really know for certain.

Now enter Xen. Xen works much the same way as VMWare, but is more lightweight. Xen loads up as a base kernel that in turn loads a Dom0 kernel. The Dom0 is the OS that runs with higher priviledges than any other guest OS on the system. The purpose of the Dom0 is to allow an interface between the guest OS’es and the Xen kernel (to start and stop guest OS’es, create new ones, etc).

The biggest limitation for Xen currently is that in order to run under Xen, an OS must be compiled to run under Xen. Currently there are hardware compile options for Linux (patchset), FreeBSD, and NetBSD (both support Xen as a native hardware architecture, like i386, or PowerPC). You will notice that Windows is missing from this list, and that is because while Xen will run Windows as a guest OS, there is currently no way the average Joe can get access to the code to Windows to make the changes necessary to get Windows to run under Xen. The Xen guys have gotten Windows to run under Xen by modifying the Windows source, and I am betting by signing NDAs and giving away their first born children, so we know that Windows will work under Xen with some modifications.

That is the basics of Xen. Now we turn an eye to the future and find out what is on the horizon.

Here is where things get interesting, both Intel and AMD are planning on offering a new technology into their CPUs that supports virtualization. Intel calls theirs VT (Virtualization Technology – catchy right?). Currently (as of the last processor table guide from Intel) two processors support the VT technology, the Pentium 4 662 and 672. I am assuming (using my secret decoder ring) that the XX2 designator will indicate the presence of VT for the chip. I have looked around, but I haven’t found a single place that sells these processors yet so who knows how much they will cost.

I don’t have any details on AMDs VT offering yet.

So what is so special about VT? Well Xen 3.0 (released December 2005) supports this new feature (VmWare also supports this feature). With support for VT technology it is now possible to run guest OS’es in Xen that are not compiled for the Xen architecture…. Hello Windows!

So lets stop and think about this for a minute, it will now be possible to purchase a fairly large server (lots of RAM and disk space) and run Linux, FreeBSD and yes, even Windows side by side by side.

It seems that this next innovation might just be a great way to improve the over all cost of operating multiple servers with multiple OS’es. Only time, and probably usability tools will tell.

Update: I want to thank Karsten Kruse for quoting me in an article done on installing XEN under NetBSD. If you want to know how to install XEN on NetBSD you can read Karsten’s post about how to do it here.

I have to admit that I don’t really like GUI interfaces. I use them, but often I find that I move more efficently in a text based enviroment. I am so bad that I often find :wq in various text files that I edit in Windows (:wq is vi short hand for “write then quit”).

When it came time to find a firewall application, I was not keen on some of the more “flashy” alternatives out there. I had my heart set on a Cisco PIX and was planning on using that until I discovered m0n0wall.

m0n0wall had all the features I was looking for. Stateful firewall, logging, could run from an optimized compact flash device, supported VLANs (802.1q), supported device polling, and last but not least, ran FreeBSD.

I have to say that my experience with m0n0wall over the past month has really proven to me that there is some really great software out there. It just works. It is flexible enough that is does exactly what I need, efficient enough that I don’t dread making changes to it, and out of all the things I have used for firewalls it is easy to learn and not difficult to master.

The best part is that it does so much more than what I am using it for. Not only does m0n0wall excel as a border firewall/router, but you can use it at home as well. It fully supports NAT, wireless network cards, VPN endpoints, etc… Couple this software with some soekris hardware and you could easily duplicate the functionality of the Linksys. Granted this option might cost you a little bit more money, but that also comes with added flexibility in how you can configure the device, not to mention more CPU intensive operations like VPNs.

If you are considering a firewall for your network, personal or professional, I highly recommend the m0n0wall application.

Look, I know it has been a while since I posted. I am a busy guy, and the blog is one of those “you know I will do that tomorrow” things.

Here is a quick write up of how I took a PC Chips 871G motherboard, and turned 4 of them into network bootable servers. It is worth it trust me.

The PC Chips 871G is an SiS chipset based motherboard, with an SiS900 based onboard LAN. Out of the box the motherboard comes with a network boot option, but it is RPL, and pretty much useless unless you are running Netware, and I haven’t touched Netware since 4.0.

The new universally supported standard of network booting is PXE, so we are going to need to get the computer to boot up using PXE. Here is where things get very interesting.

You have to hack the BIOS. It isn’t for the faint of heart, and I actually killed two bios images before I got this right. As an aside you can flash a non-booting BIOS by booting up the same motherboard with the same type BIOS (like I said I have 4 of them) and then while the computer is running, you pull the working BIOS out of the socket – CAREFULLY. Then you place the dead BIOS chip into the socket and reflash it using your known good BIOS image. I got pretty good at this, I am sorry to say.

You will need a program to edit your BIOS images, there are plenty of places on the internet to find the utilities to do this, but it takes some patience to find one that will work for you. My BIOS was an AMI BIOS, so I had to find AMI BIOS tools. Once you have the tools you can download the latest BIOS from PC Chips, and then grab the latest ROM BIOS for the LAN card. You can get this from the handy site Rom-O-Matic.

The BIOS image I had to work with, was limited in space. The current RPL part of the BIOS was 16KB in size, however the new Etherboot image was 32KB in size, so I had to remove some other things from the BIOS. I removed the RPL portion of the image as well as the SATA RAID software and, boot screen images.

After I flashed the BIOS with this new image, I had the ability to select a new option from the boot menu “Etherboot SIS900”.

The BIOS boots as normal and you can boot into a PXE boot environment to load the OS of your choice.