Hosting Geek Blog

How To Recover Data From a Badly Corrupted Drive

2008-10-28T11:55:00.002-05:00

Recently I had a customer's hard drive on a dedicated server corrupt so badly that we couldn't access the information on it at all. Any attempt to fsck the drive or even just mount it produced weird errors and strange notifications (like trying to mount it would say "not found" and other really vague answers).

We basically determined that the inode that held the partition tables as well as the inode that held root directory (/) were corrupted. We were able to run a fsck on it initially, and were prompted to repair the entries as well as multiply linked inodes and other errors. After letting fsck finish, the drive was completely unreadable. No partition tables, nothing... We told the customer that the drive corrupted and that they would have to restore their site from backups. Well guess what they said? "What backups?" Ugh... So...

Moving on... we installed a new drive, moved the old drive as the slave to attempt to recover the data for them. We installed the OS and setup the control panel, and put a page "Site crashed... we are working on restoring it." and moved forward.

Here is what we did and the results we had and some notes about the process along the way.

1) Do not panic! The first thing you have to remember in dealing with a corrupted drive is don't panic! The information is still there... and with enough effort can be recovered provided that the drive still functions at some level.

2) Make a backup image. This will help you out a great deal when things go wrong. I recommend using dd to just mirror the data to a new drive or some place with enough storage to hold the drive data.

3) Take your time. This one is hard to follow because normally the site is down and the customer is absolutely panic striken and calling you every 15 minutes asking about the status. Working with large sets of data takes time, so be patient and wait for the various processes to complete... They always take a while, and cutting corners here to save time will only lead to misery if you screw things up.

Okay... Now that we have the ground rules laid out, here is how we restored all their data save one database table, and even then we managed to save that and I will show you how we did that as well.

Step 1: Take the drive to another machine if you can, or a safe place to work on it. We had another machine with a large enough hard drive to hold the data on the drive and made a disk image of it.

dd if=/dev/sdb bs=1k conv=sync,noerror | gzip -c > /path/to/disk.img.gz

This will save your a potential future headache, but it takes a long time for a fair amount of data. Wait it out... it is worth it in the long run.

Step 2: Determine how badly the drive is corrupted. In our case an "fdisk -l /dev/sdb" didn't show any partition tables, so we had to recover that first before we could start getting at the data.

The application we used to recover the partition table is TestDisk. It is written by a gentleman named Christopher Grenier, and to say that it is awesome is a complete understatement. You will be using this software for all steps that follow, that is how useful it is.

Okay... so download TestDisk from the link above and extract it. Make sure to grab the version for the OS you are using. We used the linux 2.6 kernel version since the drive is attached to a 2.6 kernel machine that we are using for recovery purposes.

Step 3: Recover the partition table. When you first run TestDisk it provides you with a list of drives available. Select the drive (in our case it was /dev/sdb), and the partition table type (we used Intel/PC since that is what we have). Select "Analyse" and the software will inspect the drive to look for partitions. The software will do a quick search first and scan the drive very quickly looking for partitions. In our case it found the boot partition almost immediately, however the other partition was not found on the first pass, so we needed to do a "Deeper Search". The Deep Search found the other partition.

Step 4: Once you have the partitions listed, you will want to write them to the drive if possible. This will help you in the next step. After we wrote the partition table we exited TestDisk and did step 5. If you just have a Linux based partiton instead of an LVM partition, then you can just continue to step 6.

This is where things get a little tricky. In our case the partition was actually an LVM physical volume partition, meaning it holds additional information for LVM volume groups and logical volumes inside it. TestDisk won't allow you to read files directly from the LVM partition, which makes sense since there aren't technically any files inside that. What you need to do is get your OS to import and activate the volume groups and logical volumes so that TestDisk can see them and use them.

Step 5: Import and activate your LVM volume groups and logical volumes. First you need to find your LVM settings. So run:

lvm vgdisplay

This will show you the volume groups for the drive. Then you run:

lvchange -ay

This will activate the volume groups and logical volumes so that the OS can use them.

Step 6: Start TestDisk again, and this time you should see the logical volume or partition with your files on it. Select it and this time (if you are using LVM) you should select "None" as the partition type, since LVM doesn't have partition information. Select Analyse and you will see the partitions hopefully.

Step 7: Move the partition that contains your files, and press the letter p. This will give you a directory listing of the files as they are found in the partition/logical volume. From here it is just a matter of navigating to the correct location and finding the files you need to recover and pressing "c" to copy them off the drive onto your working drive.

The customer's drive was a mess. Basically all the directories and files were linked under "lost+found" with names like #123456789. We had to hunt and peck through the drive structure to find the information that we needed. We eventually found the mysql directory and the web space directory and copied those to the working drive (/dev/sda) on our recovery machine.

I have to say that we were pretty pleased with the results. Out of all the files we recovered (and there thousands of images and other files) we only had one file that was damaged. Unfortunetly it was a MySQL FRM file which contains the schema information for the associated table. Without this information MySQL can't read the data from the MYD file.

Recovering from a damaged FRM file

The FRM file doesn't often get modified, so the likelyhood of it being damaged is small, but in our case the damaged drive did corrupt this file. Here is how we recovered the data.

If you have a copy of the original schema, and I mean an exact copy with correct field lengths/types, etc. then you are in luck and the restore process is very simple.

Step 1: Make backups of your files. Copy the MYD file out of the databases folder (/var/lib/mysql/databasename) to someplace safe. You are goint to need this later.

Step 2: Delete the other files that make up the table files (remove the corrupted FRM file, MYI file and the MYD file you just copied). For example if the table is named foobar in database example, then you will have files named: foobar.MYD, foobar.MYI, and foobar.frm in a folder named example in your mysql directory.

Step 3: Recreate the table schema. Generally this involves logging into the MySQL command line interface and typing in the CREATE TABLE sql queries to make the table. This will recreate the table in MySQL, but the table will be empty.

Step 4: Copy your data file back to your database folder. Take the backup you made of the MYD file in step 1 and copy it over the file that is now in the database folder. Using our example again, you would copy the file foobar.MYD over the file that exists in the example folder in the MySQL data folder.

Step 5: Restart MySQL. Not really necessary, but can't hurt either.

Step 6: You will need to repair the table to make sure your indexes are correct and everything is working correctly.

And that is it!

Good luck, and I hope you never need this information.

Turn Your PXE Enabled Network Card Into an iSCSI HBA

2008-09-22T10:33:00.004-05:00

You can turn that PXE enabled network card into an iSCSI enabled HBA for free. Save yourself a couple of bucks on an iSCSI HBA, and boot your server/workstation diskless via iSCSI.

Here is how you turn your card into an HBA.

gPXE is a PXE compatible bootloader that provides some great functionality including AoE (ATA Over Ethernet), HTTP (loading boot scripts and boot images from HTTP), and the one we are most interested in, iSCSI which allows us to boot from an iSCSI target.

We start off with a working PXE enviroment; DHCP server (to provide IP and PXE settings) and TFTP server (to provide the PXE files we need to load). Now in order to get PXE to load the gPXE firmware we need to do what is called "chainloading". This means that our network card will do its standard PXE boot up, and when it loads, we will then load the gPXE loader and use gPXE for the rest of the boot process.

Here is how we do that:

In our DHCPd server we need to add some specific settings to enable us to detect wether or not the DHCP request is coming from PXE, or gPXE.

/etc/dhcpd.conf:

allow booting;
allow bootp;
default-lease-time 600;
max-lease-time 7200;
authoritative;
option space gpxe;
option gpxe-encap-opts code 175 = encapsulate gpxe;
option gpxe.bus-id code 177 = string;
ddns-update-style ad-hoc;
subnet 192.168.2.0 netmask 255.255.255.0 {
use-host-decl-names on;
range 192.168.2.20 192.168.2.200;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.2.255;
default-lease-time 1800;
max-lease-time 86400;
option domain-name-servers 192.168.1.10;
next-server 192.168.2.1;

if not exists gpxe.bus-id {
filename "undionly.kpxe";
} else {
filename "http://192.168.2.1/default/install.gpxe";
}
}

The important lines are:

option space gpxe;
option gpxe-encap-opts code 175 = encapsulate gpxe;
option gpxe.bus-id code 177 = string;

and

if not exists gpxe.bus-id {
filename "undionly.kpxe";
} else {
filename "http://192.168.2.1/default/install.gpxe";
}

This conditional statement allows us to load either the gPXE chainloader when we are called from a standard PXE request (the if not exists gpxe.bus-id) or a gPXE compatiable script when we are called from gPXE.

We are currently using this setup to handle new server OS installations, hence the install.gpxe file.

The contents of that file are rather simple.

install.gpxe:

#!gpxe
kernel http://192.168.2.1/default/centos5 askmethod
initrd http://192.168.2.1/default/centos5.img
boot

This loads the CentOS 5 PXE installation image and initrd to handle OS installation on the server.

Once the server has its OS installed, we then need to add the server's MAC address to the DHCPd server so that it will chain load gPXE and then load the server's root disk via iSCSI.

Here is how we accomplish that:

/etc/dhcpd.conf (added in the subnet 192.168.2.0 section above):

host server01 {
hardware ethernet 00:xx:xx:xx:xx:xx;
fixed-address 192.168.2.21;
if not exists gpxe.bus-id {
filename "undionly.kpxe";
} else {
filename "";
option root-path "iscsi:192.168.2.1::::iqn.2001-04.com.server:server01.vg00.lun0";
}
}

This, again, chainloads the gPXE chainloader from PXE, and on the next DHCP request from gPXE we provide the iSCSI target to load the root for the server. This brings up the normal GRUB screen and the system boots as normal.

And that is how you turn your PXE enabled network card, into an iSCSI HBA.

Gotchas:

I had originally wanted to use gPXE/iSCSI to host the root drive for a Xen based Dom0, however I have discovered that the Xen hypervisor does not support this feature. I have done some searching on the internet and it seems that the problem does lie with Xen's hypervisor kernel and its inability to read the iBFT (Iscsi Boot Firmware Table). gPXE does support and utilize the iBFT, however the Xen Hypervisor kernel only recognizes the iBFT from certain iSCSI HBAs (listed in thier HCL).

Installing CentOS 5 as a DomU with a Debian Dom0

2008-04-14T12:37:00.002-05:00

There isn't a whole lot of information about how to setup CentOS as a DomU under a Debian 4.0 based Dom0 and still maintain the use of pygrub to boot the CentOS kernels. This howto will give you a general overview on what steps to take without having to use an incomplete CentOS image. This is not going to be a copy and paste sort of howto, but rather a more high level detail, and a couple of fixes to make it all work correctly.

A couple of assumptions I am making here:

You have a working Xen install already under Debian
You can edit files using vi or a comparable editor.
You understand how Xen and LVM can work together at least at some basic level
You are confident to compile your own applications using make, etc...

Here is what you need to do to get started:

Step 1:

Download the kernel image and ram disk for CentOS and put them some place you can access them on the Dom0.

In my case, I put them in /usr/local/src/xen/ (vmlinuz and initrd.gz respectively). I downloaded these files from a CentOS mirror. The files you are after are located in the centos/5.1/os/i386/images/xen/ directory as these contain the Xen code compiled into the kernel so that you can boot the DomU in paravirtualization mode.

Step 2:

Create a Xen DomU configuration file that points to these files for the boot kernel.

I edited the two lines:

kernel = "/usr/local/src/xen/vmlinuz"
ramdisk = "/usr/local/src/xen/initrd.img"

This tells Xen to use these kernels on boot up.

Step 3:

Modify your DomU config to point to your disks:

disk = [ 'phy:/dev/xen01/centos5-disk,xvda,w', 'phy:/dev/xen01/centos5-swap,sda1,w']

It is important to note that you must export the drives from the Dom0 as xvda, otherwise the CentOS installer will not be able to detect them properly and you will have no target drive to install to.

We will also want to modify the default restart behavior as you will see later, this is important:

on_reboot = 'destroy'

Step 4:

Go ahead and boot up the Xen DomU using xm create -c

Install CentOS as a normal network installation (point it at an FTP or HTTP mirror and let it install normally).

Step 5:

Once the CentOS installation is completed, the DomU will attempt to reboot itself. This is why we set the on_restart to destroy instead of the defautl of restart. We need to edit the configuration to boot up via pygrub instead:

bootloader = "/usr/lib/xen-3.0.3-1/bin/pygrub"

Step 6:

Here is where things get a little tricky. The pygrub application is missing a library that it needs in order to boot up CentOS based kernels. We must build this ourselves.

Download the xen-3.0.3 source (the new sources do not build this file, so I used this version specifically, I don't know if others will work). I know for a fact that xen-3.2.0 does not work.

wget http://bits.xensource.com/oss-xen/release/3.0.3-0/src.tgz/xen-3.0.3_0-src.tgz

Untar the file and cd into the directory xen-3.0.3_0-src

Then:

cd tools/pygrub

Then you need to run make. Pay attention to the errors, you might need to install additional libraries if you don't have them on your Dom0. (e2fslibs-dev comes to mind).

Step 7:

Once your build has successfully completed, you will need to copy the files to your local xen installation.

cd build/lib.linux-i686-2.4/grub/fsys/ext2
mkdir /usr/lib/xen-3.0.3-1/lib/python/grub/fsys/ext2
cp * /usr/lib/xen-3.0.3-1/lib/python/grub/fsys/ext2/

Step 8:

Boot your DomU using:

xm create -c

Finished:

You should now have a working Xen DomU under Dom0 without having to resort to broken CentOS images.

CentOS 4.4 and Asus P5GC-MX Motherboard

2007-05-15T17:47:00.000-05:00

I recently had to install CentOS 4.4 on the Asus P5GC-MX motherboard. The board works very well with the installer as long as you aren't doing a network install as the network drivers are not available.

Here is some information about the boards network controller, that was almost impossible to track down. The board uses a network chipset called the Attansic L2.

More specifically, you can compile the drivers from source files against the current kernel.

I have included a copy of the driver for you to download if you need it. I searched through many mailing list postings to locate these file and I can confirm that it will compile against the kernel in CentOS 4.4.

l2-linux-driver-new.rar

In order to compile the kernel module you will need to install the kernel source for your current kernel version (yum install kernel-devel and kernel-smp-devel if you need it). You will also need to create a symbolic link from the source to /usr/src/linux as the kernel module looks at this location for the current kernel.

Then you need to cd to the src directory in the archive and run make to compile the module.

Once completed you can run insmod to install the module into the kernel.

After you have done that, you will need to make sure you create module alias between the module and eth0.

It is a little bit of a pain to get working, but it can be done.

Good luck and I hope that the source files come in handy to somebody out there.

CentOS 4.4 and New nForce Chipsets

2007-04-27T18:57:00.000-05:00

CentOS is a great OS, and we use it for all our cPanel installs. It is getting a little old, but currently CentOS 5 isn't supported by cPanel, so we must continue to use CentOS 4.4 for these installs.

The latest batch of motherboards we got in use the MCP61 (nForce 430 chipset). Luckily the SATA controller is supported using the nv_sata kernel module that comes with CentOS 4.4, so there is no need to upgrade that. However the network interface of this chipset is not recognized by the forcedeth driver (reverse engineered nForce network driver).

The solution to fix this problem is to compile the latest forcedeth.ko (kernel module). Here is how you do it.

1) Install CentOS and be sure to install gcc and kernel-devel for your kernel.
2) Download the latest forcedeth drivers from nVidia. You can get them from here: http://www.nvidia.com/object/linux_nforce_1.21.html
3) Extract the files from the zip file.
4) Change to the directory that contains the forcedeth.c source code. (./NV_Linux_DRV_PKG_v1.21/RHEL4_U4/source)
5) Create a Makefile that contains:
obj-m := forcedeth.o
6) Now compile the module with the following command:
make -C /usr/src/kernels/2.6.9-42.0.10.EL-i686/ SUBDIRS=$PWD modules

Please note that your path might differ as you might be using a different version of the kernel.

7) When this completes you will have a new forcedeth.ko file in the current directory. Move this file into modules directory:

cp forcedeth.ko /lib/modules/2.6.9-42.0.10.EL/kernel/drivers/net/

Again, your path might differ based on the version of the kernel you are running.

8) Add an entry to alias the kernel module to your network interface in /etc/modprobe.conf
alias eth0 forcedeth

I threw a reboot at the machine just in case, but you can also do:

modprobe forcedeth

and your network card should now appear in:

ifconfig -a

And there you go...

XenExpress Makes Xen Easy

2007-02-07T11:41:00.000-06:00

As regular readers of this blog know (all two of you), I am a huge fan of Xen. It has never been easy to configure or install Xen, but that hasn't really stopped me from using it when ever I have had the chance.

Well recently I was pointed to XenExpress, and XenExpress is about to make Xen a whole lot easier for the average user to install and use.

XenExpress is an all in one ISO that you can download from the makers of Xen (for free). It is an ISO that you burn to a CD, and then boot a server off that CD to install Xen. While this isn't really that big of a deal, what it offers in management of Xen is where it truly shines.

Once you have XenExpress installed on the server, you can then install the Windows Xen Admin interface and use that to monitor the Xen server, as well as create new Xen virtual machines. To say that it is easy is an understatement.

Here is a screen shot of the admin interface at work:

In this picture I am in the process of installing Windows 2003 as a virtual machine, via the XenAdmin interface.

Here are some of the monitoring features of the XenAdmin interface:

XenExpress is about to make Xen a whole lot easier to use and manage.

How To Find and Remove Windows Rootkits

2007-01-01T20:29:00.000-06:00

First of all I want to wish everyone a very Happy New Year. It has been a while since I made my last posting and I want to apologize for that.

Now let's get down to business...

So recently we discovered that a Windows 2003 server had been exploited via an apparently well known 0day exploit in MailEnable's SMTP service. This has since been corrected by the MailEnable developers (you can read about that here).

At first there was some doubt as to whether or not there was a hacker on the server. Our first clue was the abnormal amount of traffic the server was doing. Typically this server moved about 100-200KBps per day. We knew something was up when this server started moving 2MBps. Upon inspection of the server we couldn't see anything out of the ordinary, however we did notice that taskman.exe (Task Manager) was running at 100% CPU utilization when ever we looked at it. This threw up all sorts of red flags to us, and we knew that we had a hacker on the server and we needed to find out what they were doing.

I have to say that Event Viewer is your friend. You must look at it every once in a while to make sure you know what is going on. Even with a hacker on the server and rootkit installed on the server to hide his activity, he still wasn't able to hide some log entries in Event Viewer. After looking in Event Viewer we noticed several references to VMWare. After asking around, we determined that none of the legitimate administrators had installed VMWare and we knew that this must be the hacker.

Here is how we found him, and how we removed him.

We could see that there were some hidden directories on the server that we could access through the normal Explorer interface, so we knew we were dealing with an on boot rootkit. (You can see file accesses using the file system and disk tools from Windows SysInternals Tools.)

We installed HiJack Free from a-squared. This piece of software is pretty powerful and does some deep inspections of the registry to find services and applications that are not normally visible in the Control Panel services listing. We sorted the services and looked for services that were set to start at boot up, and looked for anything that wasn't signed. Hi-Jack Free displays the company that signed the driver/service, and any service that was set to run, but wasn't signed was on our hit list to remove.

With a list of services to disable, we installed the Windows Recovery Console and rebooted the server into the recovery console. We disabled the services that we identified as problems and rebooted the system normally.

At this point we could see the directories that were hidden from us earlier. We discovered that the hacker had installed VMWare. Because we wanted to see what the hacker was actually doing with the VMWare installation, we used Virtual Disk Driver to mount the WMWare disk images to see what they were doing. Turns out they were downloading Pokemon episodes. Ugh... All that hassle and it wasn't even anything good. :)

So we removed the rootkit from the server, removed the VMWare installation, and patched the MailEnable install, and the server has been cruising along ever since.

We hope that this description of what we did will help you find and remove Windows rootkits on your servers.

Find Out What Your DNS Server is Doing

2006-09-25T12:21:00.000-05:00

What is my DNS server responding to?

We have been in the process of moving from an old server to a newer server. The process is straight forward, we move the sites over to the new server and then update their zone records to point at the new server (the zone has a low TTL - Time To Live to make this transition smoother). Overall everything has gone smoothly with little interuption in the service of each site.

Finally once everything was moved over, we updated the nameserver records to point at the new server so now everything should be running off the new server's DNS. We are ready to turn off the old server, but noticed that named (bind) was still handing out DNS responses (based on its activity in top). We thought we had everything updated so that this server shouldn't be used at all.

So we had to find out what DNS requests were still hitting the old server and why we missed those. Here is what we did to find out.

Edit your named.conf (ours was in /etc).

Add the following section if you do not already have a section called logging {}.

logging {
channel query_logging {
syslog daemon;
severity debug 9;
};
category queries {
query_logging;
};
};

What this does it record any DNS query named serves up in the default syslog for named (generally /var/log/messages). This will help you see what domains are being requested from your server.

We determined what DNS queries were coming in, and based on the whois information found out that there were some very old nameserver records pointing at the server's IP. Without the logging change above, we could have lost 3 or 4 long time customer's DNS information when the old server was turned off. As it is now we have updated those nameserver records to point at the new nameservers, and will need to keep the old server up and running for at least another 48 hours (the amount of time a root nameserver record is cached). Saved us a black eye for sure.

What else is my DNS server handing out?

Additionally, you might want to look at the log information and determine if anybody is using your server for recursive lookups too.

What is DNS recursion?

Well, recursion itself isn't bad, and actually a vital part of DNS. Recursion means that if you request a DNS lookup against a DNS server, and that server isn't authoritative for that domain (it doesn't have a zone for that domain), it must pass the DNS request to another server.

Why is it bad to allow recursion?

Until recently DNS recursion wasn't really a bad thing, but hackers have determined that it is possible to "amplify" or magnify their DDoS (Distributed Denial of Service) attacks using spoofed UDP based DNS requests. (UDP is extremely easy to spoof the originating IP address of the request.) The hackers send a spoofed UDP request for a given domain with a large number of records to a DNS server that allows recursive lookups. Since the initial UDP request is realtively small, and the response (because it has so many records in it) is very large, hackers can amplify the amount of data they can send at a target using recursive third party DNS servers.

How do I turn off recursion in named/bind?

To turn off recursive lookups from unauthorized sources you can add the follownig ACL to your named.conf:

acl recursion { 127.0.0.1; 1.2.3.4/24; };

And then in your options do:

options {
allow-recursion { "recursion"; };
};

The first line creates an ACL (Access Control List) to let named (bind) know who is allowed to do recursive lookups against the server. The IP's should be listed in CIDR notation, and be followed by a semicolon. Include any IP address that uses this server for legitimate DNS lookup purposes.

The second section should already exist in your named.conf, and you just want to add the allow-recursion line to that section. This will apply the ACL to your server. Then you just need to restart named, and you are good to go.

So that is why you should know exactly what your DNS server is doing.

How To Install NetBSD as a DomU in Xen 3.0

2006-09-14T11:29:00.000-05:00

Ever since the first time I heard about Xen and its ability to run any OS side by side on the same server I have had the urge to run a BSD based OS with a Linux OS. Today I have sucessfully achieved my goal, and this is how I did it.

First some background on the server itself. The server is a Dell PowerEdge 1750 with Dual Xeon processors and 3GB of RAM and 500GB of RAID storage. The server is running the Xen 3.0.2 hypervisor kernel (the main kernel that handles the paralization, or virtualization, of the hardware). The Dom0 system is running Debian 3.1 with some patches to the kernel to work with the LSI based RAID 5 card in the server. Each virtual OS installed on the server is given its own partition and is managed using LVM in Dom0.

The vast majority of information about NetBSD running under Xen as a DomU seems to be either Xen 2.0 specific, or assumes you are running NetBSD as Dom0. Unfortunetly, the Xen 2.0 information is no going to work on a Xen 3.0 machine, and more so our Dom0 is Debian, so we needed to come up with our own.

Here is how I did it, and what sort of problems I encountered.

The entire process is pretty easy, but finding the actual information can be tough, and finding the files you need can be even tougher. Here is kind of a rough over view of the process...

1) Set up your partition that will hold NetBSD. We are using a LVM partition named vg00-netbsd.
2) Set up the xen domU config file.
3) Boot the netbsd install kernel for Xen 3.0.
4) Follow the sysinstall steps like you normally do to install NetBSD. I had to use an FTP based installation, because I could never get the CDROM to work correctly.
5) Complete the install and shutdown NetBSD.
6) Edit the domU config file and change the kernel from the install kernel to the normal NetBSD kernel.
7) Boot NetBSD DomU and enjoy.

So here are the specifics.

Step 1: You need to download the NetBSD Xen 3.0 kernels (install and normal) and put them some place on your Dom0. I put mine in the /boot of the server, because it sort of made sense to me, but they can be almost anywhere. You can download the DomU kernels from NetBSD's FTP servers from the daily build areas. The kernels for Xen 3.0 are not in the release versions of NetBSD so you have to find them. I would post links to them, but most likely the would go stale over time. Go to ftp://ftp.netbsd.org/pub/NetBSD-daily/ and navigate through to either the NetBSD 3.1 tree or the NetBSD 4.0 tree. You are looking for the directory i386/binary/kernel/ in that directory you will find the two kernels you need. The install kernel is called netbsd-INSTALL_XEN3_DOMU.gz and the normal kernel is named netbsd-XEN3_DOMU.gz. Download both of those kernels as you will need them later.

Step 2: Once you have downloaded your kernels you will need to create a xen config file for your NetBSD DomU. Here is an example of the one I used:

kernel = "/boot/netbsd-INSTALL_XEN3_DOMU.gz"
memory = 128
name = "netbsd"
vif = [ '' ]
disk = [ 'phy:/dev/mapper/vg00-netbsd,0x01,w' ]
root = "/dev/wd0d"

You will need to change the disk = line to match where you are installing NetBSD to on your server. After you have created that file in your xen config directory (our was /etc/xen/).

Step 3: We are ready to boot NetBSD for the first time. To boot NetBSD we run the command:

xm create -c netbsd

"netbsd" is the name of the DomU config file we created in step 2, so change that to match what you used in that step.

A couple of times we noticed that Xen didn't attach us to the console of the booting NetBSD DomU, so you may need to connect to it manually. To do so do the following:

xm list

Which will print out a list of running Xen instances like this:

Name ID Mem(MiB) VCPUs State Time(s)
debian 0 1374 4 r----- 2354.5
plesk 10 1024 1 -b---- 161.5
netbsd 50 128 1 -b---- 1.2
qmail 8 128 1 -b---- 953.4

We will need to know the ID of the instance we want to attach to. In the example above this is 50. Then we attach to the console of that DomU by typing:

xm console 50

To break out of the console at any time simple press CTRL+] at the same time.

Step 4: Once you are in the console you should see the sysinstall application. You can follwo the prompts and install NetBSD like you would normally do. One problem I did encounter was that for what ever reason the server would stop talking to the FTP server due to some sort of DNS lookup failure. It did this no matter which kernel I tried. I eventually resorted to using the IP address instead, and the installation worked perfectly.

Step 5: Once the install is completed, break out of the server and shut it down via the command:

xm shutdown 50

Again replace 50 with the ID of the DomU of your NetBSD install.

Step 6: Edit the DomU file and change the kernel line to point to your normal NetBSD kernel. So your DomU config file should look somethnig like this now:

kernel = "/boot/netbsd-XEN3_DOMU.gz"
memory = 128
name = "netbsd"
vif = [ '' ]
disk = [ 'phy:/dev/mapper/vg00-netbsd,0x01,w' ]
root = "/dev/wd0d"

Step 7: Reboot your NetBSD DomU via the command:

xm create -c netbsd

Enjoy your NetBSD running under a Debian/Linux Dom0 in Xen 3.0!

Gotchas:

Having used the NetBSD system only breifly I have noticed that there is something "funky" with the networking and the way it behaves. I noticed over a sustained ping that the network interface starts to drop packets, every other packet it seems. Modifying the vif = line in the DomU config to read:

vif = [ 'bridge=xenbr0' ]

seems to have cleared up the issue. This line bridges the ethernet interface inside the DomU to the xenbr0 interface in the Dom0. It seems to have cleared up the issue to date.

And there you have it! NetBSD running under a Linux Dom0 on top of Xen 3.0. The world just got a whole lot smaller.

Why You Shouldn't Use Fedora Core For Production Servers

2006-08-31T12:10:00.000-05:00

I am going to vent a little here.

I don't understand why web hosting companies (and dedicated server providers for that matter) continue to provide Fedora Core based servers to customers. It just doesn't make sense. Fedora Core was never intended to be used in a production enviroment and why people keep using in those enviroments baffles me.

We had a server that was running Fedora Core 4 and cPanel and has been in production for almost a year. It was always updated and properly maintained... that is until RedHat dropped updates for it (which they do every 6 months as that is the development cycle for Fedora). Dropping updates isn't that big of a deal as there is the Fedora Legacy Project that continues to provide updates after RedHat has dropped updates.

Here is what happened with this server, Fedora Core 4 was dropped from RedHat support, and was being taken over by Fedora Legacy. No big deal, this has happened a bunch of times in the past with previous Fedora Core versions. The problem enters when you throw in a local kernel exploit that is discovered for the /proc fs. We attempted to upgrade this kernel to prevent this problem, but because FC4 was in a state of transition between the two providers, the updates weren't available yet. As such this server was put into the "holding for update" queue.

In that time the server was hacked and needed to be wiped and reinstalled with another OS that won't cause this particular problem (CentOS 4.3).

What we recommend to customers instead of Fedora Core?

CentOS: RedHat Enterprise Linux based OS. They grab the source RPM files from RedHat and remove the images that reference RedHat and replace them with CentOS based images, and other minor modifications, but generally this is the same code that is used to run RedHat Enterprise Linux. Because it is based on RHEL it has a good development cycle as well as excellent support for new drivers. Does it have it's problems? Yes, any problem that exists in RHEL will also exist in CentOS (various spin_lock kernel panic issues have been documented).

We recommend the use of CentOS for any application that does not support our number one OS of choice.

Debian (stable): I admit it. I am a big fan of Debian. It works great and I am very familair with it. It handles in place version upgrades (moving from woody to sarge was done with minimal effort using apt-get dist-upgrade). It has an easy to use package manager, with a large selection of applications, as well as an active developer community. My biggest problem with Debian is the that the kernel gets out of date very quickly. Unlike CentOS which actually backports drivers from a more current kernel revision (for instance 2.6.16) into the current standardized kernel for the OS (2.6.9 for CentOS 4.3), Debian does not do that. So it is often a chore to get Debian to run (or even install) on newer chipsets.

Overall, Debian is the winner for us. We install it anytime we have the option (even for Virtual Private Servers) as it just works. It has its faults and is by no means perfect, but we are willing to over look those as they are minor in comparison to the benefits it can provide in ease of management and administration.

So to recap:

Don't use Fedora Core for a production server, it will only cause you pain and suffering in the long run.
Anyone that suggests you use Fedora Core in a production enviroment has not analyised the potential ramifications of that decision completely.
There are many viable alternatives that are actively developed and maintained as well as providing up to date and current device driver support and security updates.

In conclusion:

Don't use Fedora Core on a production machine.

Steps to Tighten PHP Security

2006-07-24T11:05:00.000-05:00

Recently I have had to deal with some insecure PHP scripts on a couple of servers that have caused us some serious problems, as well as time, to recover from.

I am going to list some of the important steps you can take to protect your server and your site from insecure PHP applications. This will by no means a complete list of items to stop them, but this should help prevent most of the basic attacks ("script kiddie" attacks - where the attacker doesn't have a great deal of skill and is depending on a preset scenario).

Most of these solutions will assume you have some level of control (root) over the server.

Turn off allow_url_fopen: This modification will prevent the use of http://www.domain.com/somescript.php and ftp://www.domain.com/somescript.php from being used in include(), include_once(), require(), require_once(), as well as fopen(). This will prevent a hacker from including malicious code remotely. This modification will prevent the vast majority of hack attempts in PHP from working.

The downside of this modification is that some legitmate applications that use fopen() to open a remote web page might be broken. You can encourage users to use the PHP curl functions instead as they accomplish the same results.

Edit your php.ini and change allow_url_fopen = On to allow_url_fopen = Off and restart your web server.
Use open_basedir: open_basedir allows you to dictate which paths PHP is allowed to access for a given site. Generally we set this to the path of the website (/var/www/html), /tmp (for writting session data as well as for file uploads), and the path to the PEAR repository.

The good thing about open_basedir is that it is default deny, meaning if you don't specify the path in the open_basedir it is blocked.

The bad part about open_basedir is that it come sometimes block access to legitmate applications needed by your PHP applications (Gallery is one that comes to mind, as it makes use of some external applications in /usr/bin, so you must open access to that location).

Generally, you should use this directive for every website you host as it allows you to control which directories PHP can access and you can make sure that those directories have the correct permissions to prevent potential exploitation.
Mount /tmp noexec, nosuid and nodev: This one is particularly useful as it allows you to tell the operating system not to run any applications in a given area of the hard drive. I can tell you personally that this one as saved me numerous times. When used in combintation with the open_basedir directive you can effectively limit what the hackers have access to, and what they are able to run.

If you have a seperate /tmp partition on your hard drive you can edit the /etc/fstab file and change the options for the /tmp line.

/dev/hda3 /tmp ext3 noexec,nosuid,nodev 0 0

If you don't have a seperate /tmp partition you can create one as a loopback filesystem and mount that as noexec. I will make a seperate post on how to do this later.
Mount your web space noexec: This might not be possible, but is a great option just like mounting /tmp in noexec. The reason for this is to prevent binary applications from being uploaded and executed in your webspace. Like I said this might not be possible, but is an excellent prevention method to prevent local root exploit binaries (where a binary exploits a problem in the kernel to gain escallated priviledges on the server, the /proc race in the 2.6 kernel is a recent example of this type of exploit).
Make sure your permissions are correct: You should not allow world write permissions on any files, other than those required (configuration files that are written by the web server, etc). This is very important as it will save you a great deal of trouble in the event that a hacker does gain access to your site, as they won't be able to overwrite your files. Additionally, you should only allow write access to specific folders where needed.
Prevent PHP from parsing scripts in world writtable directories: This will prevent hackers from uploading malicious scripts and running them from your site. This is a great way to tighten security, and is pretty easy to manage, just set "php_admin_flag engine off" for any directory that can be written to by the web server user. This one will save you more hassle than you can ever imagine.
Install mod_security: Mod Security is basically a web server firewall, allowing you to block specific types of requests, as well as inspect data as it is sent to the server. My only problem with mod_security is that it isn't default deny, which means you must identify what is blocked, rather than identifying what is allowed and rejecting everything else. Still it has its uses and you should have it installed in order to react to problems that do not have patches available yet.

We hope that this list of items will help protect your server from hackers and allow you to have some peace of mind that your sites are as secure as you can make them.

Linux Things I Learned Today

2006-06-08T21:14:00.000-05:00

What fun things did I learn today? Let me see...

Intel based ICH7 chipsets are not fully supported in the linux kernel until kernel version 2.6.11. With additional features added in 2.6.15.
The Debian Installer is based on the 2.6.8 kernel, and therefore can not install on to hard drives attached to the ICH7 chipset (at least in SATA2 mode, which is the whole point).
The northbridge chipset on the MSI 945GM2 motherboard is OMG HOT! (I have the burn marks to prove it.)
Motherbards are moving to a new auxillary power connector (in addition to the ATXv2.0 spec). This new connector is called an EPS12V connector and consists of an 8 pin connector for 12V power to the CPUs.
The EPS12V power connector seems to be standard on most "Cedar Mill" (Intel's new 65nm fab CPUs) "Ready" motherboards. Where as most power supplies do not yet have this connector.
The case I buy do not have the EPS12V connector on them - which made me sad.
The standard 4 pin 12V connector that is standard now on most power supplies DOES work in the EPS12V plug - which make me happy.
SATA2 (SATA 3.0Gbps) can be used in "comptability mode" in the BIOS, but it makes them SLOW.
The default kernel installed with Debian doesn't support over 1GB of RAM. You must install another kernel if you have more than 1GB of RAM.

I think that about covers what I ran into today. Just some random facts that might help somebody some day.

Moving Servers Realtively Painlessly

2006-05-25T14:36:00.000-05:00

Let's face it, moving from one server to another server is nasty and something any of us would try to avoid as often as we can. Unfortunetly, there are instances where we can not avoid this and we must bite the bullet and do it.

Here are some tips that I have gathered up that will help you maintain your sanity as much as possible. Honestly even with all the planning and preperation, there will be issues so just prepare yourself mentally for that. It will happen.

So here are some things you can do to help the transition:

DNS Caching: DNS has to be one of the most difficult things to deal with, because you only have a certain level of control over what occurs. Caching of DNS results will often cause many problems. The trick here is to plan ahead and remove the caching from the picture until the move is completed.

Setting your DNS to a low TTL (I use 5 seconds versus the recommended 86400 seconds/ 24 hours). The TTL tells other DNS servers and clients how long they should hold the information they just received, in their cache. Since we are going to be moving the site to a new IP soon, we don't want them caching the old IP once the site is on the new IP. The low TTL will help prevent this.
Pathing: If you are moving to a server with different paths, you should at least try to setup symbolic links to help any hard coded paths in the site's content.
PHP register_globals: This one has bitten me a couple of times. Most ofthen when you move to a new server the server will have the default settings for PHP which turns off register_globals. This will cause all sorts of problems that you won't be able to track down easily. Check it early on and make sure that both servers match settings wise. (A a general rule register_globals should be set to off, but there are some legacy PHP apps that need it.)
SSH Tunnels: Most of the time you should adjust your TTL for your DNS long before you do the actual move, but sometimes that isn't an option. In a pinch you can use an SSL tunnel to redirect traffic from your old server to the new server. This will help users that have the old IP cached, still access the new site. It should be noted that SSH tunnels are pretty unstable and shouldn't be really be depended on, but like I said, they are good in a pinch. Here is an example of one, you run this on the old server as root with the httpd process stopped:

ssh -g -C -N -f -L 80:newserverip:80 root@newserverip

This will prompt you for a password (if you don't have key based access already setup). You will get a warning 'bind: Address already in use', but the process is working. This message is just letting you know that the remote endof the tunnel has something already on port 80, which is actually a good thing.

You can read more about this process in my blog post Redirecting TCP Ports with SSH
DNS: Once you have everything moved over to the new server, update the current authoritative DNS servers with the new IP as soon as possible. Once you have done that change the domains name server records also. The name server change takes longer to complete (will normally be done in 8 hours at the root servers).
How To Deal With a Highly Dynamic Website: For sites that have alot of content, or for sites that have users contributing data often (mainly forums) you need to draw a line in the sand and take the site offline during the move. This ensures that you have all the content and that no one loses any content that was contributed during the move.
Rsync: Also in the same vein as dynamic sites (user images etc) rsync is your best friend. Run rsync, and run it often. Run it even before you move. It keeps the two sites in sync with each other and can be run while the old site is still up and running. The more you run rsync leading up to the actual move, the faster the actual move will occur. Faster move == less downtime for your users.

There are many other issues to consider, but if you take these items into consideration you should not have any major surprises coming your way.

Apple Boot Camp, Windows XP and Xen Sitting in a Tree

2006-04-05T10:55:00.000-05:00

So today I came across this site for Apple's Boot Camp.

Basically the concept behind Apple's Boot Camp is to allow users to install Windows XP side by side with MacOSX, which many people have been trying to do since the first Mac's shipped with Intel based processors.

So now it is possible to run Windows XP on a Mac with Apple's blessings.

Here is where things get interesting: Throw Xen 3.0 into the mix. For those that don't know what Xen is, it is a hardware virtualization technolgoy (based ont he linux kernel) that allows multiple OS'es to run on the same hardware at the same time. It allows concurrent access to the network, hard drives, memory, cpu etc. Prior to Xen 3.0 you needed to have an OS specifically compiled for Xen in order to run it. That has all changed with the new technology included in the latest processors from Intel (and soon AMD - June 2006 I think). The new processors have a technolgoy called Virtualization that allows hardware virtualization applications like Xen (and VMWare) to run OS'es that are not compiled natively for Xen itself (such as OSX and Windows XP).

This is where I think there is a huge opportunity for Apple (and for all I know they might actually be working on this already, I haven't a clue). To allow Xen to boot OSX as a Xen Dom0 (the OS that is allowed access to the Xen kernel for control over the other OS'es on the virtualized machine). Then boot Windows XP as a DomU. Then a user would be able to boot their computer, access OSX and switch to Windows XP all in one machine.

An interesting thought... If you can get both OS'es to boot at turn on, and used OSX as the Dom0, then you would be able to access the Windows XP virtual machine via the Mac OSX RDP (Remote Desktop Protocol) client and never have to actually leave OSX or figure out how to switch between the two. Oh man... my head spins... I need to get a MacBook Pro and start working on that.

exit signal File size limit exceeded (25) in Apache

2006-03-07T20:43:00.000-06:00

Today was a fun day. I have a buddy that has spent the last 3 or 4 months (maybe longer) working on a complete ground up redesign of a PHP/MySQL website. The current code base was horrid, and that is putting it lightly. The code was so poor that it was hampering the overall performance of the server. The pages were very slow to display, often taking 30-45 seconds to complete the rendering of one page. Looking at the top process list I could see that several httpd processes were dying off, and dying off frequently. I had assumed that this was due to the poor codebase of the existing application.

The launched the new codebase today, and things didn't improve. So my buddy called on me to help him out. First a little bit of background on the server: Dual Athlon (yes it was weird to me too), 1GB of RAM, and 1 IDE HD (again, yes I know who ever speced out this server should have their head examined). Server is running Cpanel 10.8.1-C112 under Fedora Core 2. We had rebuilt apache already (using easyapache) and the httpd zombies kept occuring. They weren't true zombies, only momentary zombies, because the child process was dying without permission from the parent process, so they only showed up in the zombie count for a couple of seconds, so they were being cleaned up properly.

Here is what we found: the main site for the server is a heavily trafficed PHP/MySQL based site with a lot of images. I started looking through the error logs and came across a bunch of messages like:

[notice] child pid 10009 exit signal File size limit exceeded (25)

After some quick googling we were able to determine that this message means that the child httpd process died, because it encountered a file that is too large for it to handle. The max file size is 2GB in this instance. There is some discussion on if this is an Apache 1.3.x limitation or a limitation a little further down (glibc or filesystem).

How did we fix it? Well short term was to shutdown apache and move the offending log file out of the way, and restart apache. Longer term fixes involve some settings inside Cpanel itself. You basically need to tell Cpanel to remove the log files after it has processed the log files.

Log in to WHM and click on "Tweak Settings", then check mark "Delete each domain's access logs after stats run"

That will delete the log files daily, and all should be well with the world.

Missing /proc/megaraid

2006-02-12T15:02:00.000-06:00

Recently I did a Xen 3.0 install on a Dell 1750 server with RAID5. By default the Xen 3.0 installation (from source) doesn't compile in the LSI drivers for the RAID card, so I had to enable the option in the Dom0 kernel in Xen.

So I went in to the xen0 directory under my Xen source (/usr/src/xen-3.0.0/linux2.6.12-xen0) and ran:

make ARCH=xen menuconfig

I then went to Device Drivers -> SCSI Device Support -> SCSI low-level drivers -> and enabled the "LSI Logic New Generation RAID Device Drivers", because after all... Newer is better right?

I recompiled Xen and the Xen kernels (cd ../; make kernels; make install) and reboot the machine and went on with my day.

I got a call from the client later in the day that the server was reporting a problem with the RAID array. Long stroy short, the customer was running a NAGIOS plugin to monitor the RAID array via the /proc/megaraid entry. Little did I know that the new generation RAID device drivers in the 2.6 kernel no longer make use of the /proc/megaraid proc entry, so it was breaking the NAGIOS plugin.

To correct this problem use the "LSI Logic Legacy MegaRAID driver" instead of the "LSI Logic New Generation RAID Device Drivers" in the 2.6 kernel (this is not an option in the 2.4 kernel). That will give you back the /proc/megaraid entry and you should be all set.

Additionally, since you are already compiling a kernel, you should enable the kernel option "Use register arguments" under "X86 Processor Configuration". This option will allow you to use the Dell OMSA device drivers for additional control over your Dell PowerEdge server. You can read more about it here: http://www.uta.fi/~pauli.borodulin/dellomsa/omsa44.html

Logs at a Glance

2006-02-05T15:25:00.000-06:00

I have many servers that I have to watch on a regular basis. We use a KVM to switch between the consoles of the servers and generally we only see the console for any given server for a couple of seconds.

The KVM we use has this great feature where if it detects any activity on the console, the KVM will swtich to that console and display the output for us.

This is how we use that functionality to quickly display any problems a server might be having. It is by no means a fool proof and accruate, but it does help to bring things to our attention on a regular basis, doesn't require you to install anything special (all programs should come with a basic install of a linux OS), and its cool... What else do you really need?

What we do is one turn off the console's screen saver. It is evil, and we don't like it. Actually it serves a valid purpose, but not for our needs so lets turn it off:

/usr/bin/setterm -blank 0

Now you can put that into /etc/rc.local for RedHat based distros (Fedora, Redhat Enterprise Linux, CentOS, etc).

For Debian you can create a file in /etc/init.d/ and name it blank_screen and chmod 755 the file. Then just create a symbolic link to the startup dir:

ln -s /etc/init.d/screen_blank /etc/rcS.d/S61blank_screen

Now that we have the console set not to turn off, we can do the next part of our monitoring mojo. We will setup to monitor files that are important to us, such as /var/log/messages, and /var/log/syslog. These files might not be the files you want to monitor, so you can modify the line below to match the files of your choosing.

We use tail, because it displays the lines as they are added to the file we are monitoring, and it is simple and lightweight.

/usr/bin/tail -v --follow=name /var/log/messages --follow=name /var/log/syslog > /dev/tty2

What this does is runs tail to monitor the files /var/log/messages and /var/log/syslog. We monitor the filename, rather than the descriptor (which is the default behavior of tail). Monitoring the descriptor means that if the filename is changed -- for example logrotate, then we will follow the file to the new name. Since once a file is rotated by logrotate it won't be updated, we don't want this behavior. We add the -v (verbose) so that tail will tell us the name of the file that corresponds with the output. Then we redirect the ouput to /dev/tty2 (virtual console 2).

This will output the information directly on the screen so that we can see it via the KVM.

Additionally you might want to turn off the login prompt on the virtual console 2. We can do this by editing the /etc/inittab file and commenting out the following line:

2:23:respawn:/sbin/getty 38400 tty2

You can just put a # in front of that line and then have init re-examine the file by running:

kill -HUP 1

So there you go... you now have the output of any file you want to the console of your server. Set your KVM to autoscan, and enjoy your easy to view logs.

Using MySQL in a Shell Script

2006-01-12T16:37:00.000-06:00

Every once in a while I come across the need to access data that is stored in a database inside MySQL from a shell script. Here is a quick and dirty way to access that information without having to resort to PHP or Perl.

Run a SELECT:

By default this would return output like:

id
514

So we need to use sed to remove the name of the field.

SITEID=$(mysql -D database -u root --password=password --silent --exec="SELECT id FROM domains WHERE name = '`echo $SITE`';" |sed 's/id//')

Since INSERTs, UPDATEs and DELETEs don't return any information you won't need to pipe the information returned from MySQL to sed.

This handy tip will work for basic MySQL queries, but for more complex queries, I would recommend something with a little more control over the information such as PHP or Perl.

I hope that helps somebody.

XEN, Hardware Virtualization - The Wave of Future

2006-01-09T11:32:00.000-06:00

Let me introduce you to my new favorite piece of open source software: Xen.

Xen is a virtualization operating system that allows you to share hardware between multiple OS'es. The concept isn't really new, as there are several commercial as well as open source implementations of this concept already.

In the past I have used Linux-Vserver which is a set of kernel patches that create security contexts inside the kernel that different versions of Linux can co-exist in. It works very well and I have been using it for over 2 years in production. The main drawback is that it only works with Linux based OS'es. The same with Jail in FreeBSD which I have also used for a while, but again is specific to the OS, in this case FreeBSD.

Now I have never used VMWare, but VMWare uses a "core" application that the guest OS'es run in. This works very well with all current OS'es from Windows, Linux to *BSD. I have heard that there is some performance degradation with this method, but I have never used it so I don't really know for certain.

Now enter Xen. Xen works much the same way as VMWare, but is more lightweight. Xen loads up as a base kernel that in turn loads a Dom0 kernel. The Dom0 is the OS that runs with higher priviledges than any other guest OS on the system. The purpose of the Dom0 is to allow an interface between the guest OS'es and the Xen kernel (to start and stop guest OS'es, create new ones, etc).

The biggest limitation for Xen currently is that in order to run under Xen, an OS must be compiled to run under Xen. Currently there are hardware compile options for Linux (patchset), FreeBSD, and NetBSD (both support Xen as a native hardware architecture, like i386, or PowerPC). You will notice that Windows is missing from this list, and that is because while Xen will run Windows as a guest OS, there is currently no way the average Joe can get access to the code to Windows to make the changes necessary to get Windows to run under Xen. The Xen guys have gotten Windows to run under Xen by modifying the Windows source, and I am betting by signing NDAs and giving away their first born children, so we know that Windows will work under Xen with some modifications.

That is the basics of Xen. Now we turn an eye to the future and find out what is on the horizon.

Here is where things get interesting, both Intel and AMD are planning on offering a new technology into their CPUs that supports virtualization. Intel calls theirs VT (Virtualization Technology - catchy right?). Currently (as of the last processor table guide from Intel) two processors support the VT technology, the Pentium 4 662 and 672. I am assuming (using my secret decoder ring) that the XX2 designator will indicate the presence of VT for the chip. I have looked around, but I haven't found a single place that sells these processors yet so who knows how much they will cost.

I don't have any details on AMDs VT offering yet.

So what is so special about VT? Well Xen 3.0 (released December 2005) supports this new feature (VmWare also supports this feature). With support for VT technology it is now possible to run guest OS'es in Xen that are not compiled for the Xen architecture.... Hello Windows!

So lets stop and think about this for a minute, it will now be possible to purchase a fairly large server (lots of RAM and disk space) and run Linux, FreeBSD and yes, even Windows side by side by side.

It seems that this next innovation might just be a great way to improve the over all cost of operating multiple servers with multiple OS'es. Only time, and probably usability tools will tell.

Update: I want to thank Karsten Kruse for quoting me in an article done on installing XEN under NetBSD. If you want to know how to install XEN on NetBSD you can read Karsten's post about how to do it here.

m0n0wall Firewall

2005-12-01T23:57:00.000-06:00

I have to admit that I don't really like GUI interfaces. I use them, but often I find that I move more efficently in a text based enviroment. I am so bad that I often find :wq in various text files that I edit in Windows (:wq is vi short hand for "write then quit").

When it came time to find a firewall application, I was not keen on some of the more "flashy" alternatives out there. I had my heart set on a Cisco PIX and was planning on using that until I discovered m0n0wall.

m0n0wall had all the features I was looking for. Stateful firewall, logging, could run from an optimized compact flash device, supported VLANs (802.1q), supported device polling, and last but not least, ran FreeBSD.

I have to say that my experience with m0n0wall over the past month has really proven to me that there is some really great software out there. It just works. It is flexible enough that is does exactly what I need, efficient enough that I don't dread making changes to it, and out of all the things I have used for firewalls it is easy to learn and not difficult to master.

The best part is that it does so much more than what I am using it for. Not only does m0n0wall excel as a border firewall/router, but you can use it at home as well. It fully supports NAT, wireless network cards, VPN endpoints, etc... Couple this software with some soekris hardware and you could easily duplicate the functionality of the Linksys. Granted this option might cost you a little bit more money, but that also comes with added flexibility in how you can configure the device, not to mention more CPU intensive operations like VPNs.

If you are considering a firewall for your network, personal or professional, I highly recommend the m0n0wall application.

Network Booting Your Computer

2005-11-08T14:37:00.000-06:00

Look, I know it has been a while since I posted. I am a busy guy, and the blog is one of those "you know I will do that tomorrow" things.

Here is a quick write up of how I took a PC Chips 871G motherboard, and turned 4 of them into network bootable servers. It is worth it trust me.

The PC Chips 871G is an SiS chipset based motherboard, with an SiS900 based onboard LAN. Out of the box the motherboard comes with a network boot option, but it is RPL, and pretty much useless unless you are running Netware, and I haven't touched Netware since 4.0.

The new universally supported standard of network booting is PXE, so we are going to need to get the computer to boot up using PXE. Here is where things get very interesting.

You have to hack the BIOS. It isn't for the faint of heart, and I actually killed two bios images before I got this right. As an aside you can flash a non-booting BIOS by booting up the same motherboard with the same type BIOS (like I said I have 4 of them) and then while the computer is running, you pull the working BIOS out of the socket - CAREFULLY. Then you place the dead BIOS chip into the socket and reflash it using your known good BIOS image. I got pretty good at this, I am sorry to say.

You will need a program to edit your BIOS images, there are plenty of places on the internet to find the utilities to do this, but it takes some patience to find one that will work for you. My BIOS was an AMI BIOS, so I had to find AMI BIOS tools. Once you have the tools you can download the latest BIOS from PC Chips, and then grab the latest ROM BIOS for the LAN card. You can get this from the handy site Rom-O-Matic.

The BIOS image I had to work with, was limited in space. The current RPL part of the BIOS was 16KB in size, however the new Etherboot image was 32KB in size, so I had to remove some other things from the BIOS. I removed the RPL portion of the image as well as the SATA RAID software and, boot screen images.

After I flashed the BIOS with this new image, I had the ability to select a new option from the boot menu "Etherboot SIS900".

The BIOS boots as normal and you can boot into a PXE boot environment to load the OS of your choice.

A Tale of Heat and Peace

2005-09-20T22:50:00.000-05:00

I am a geek. Let's just get that out of the way right now. I am a very big geek.

I run servers for a living, but mainly I don't get to play with them. I get to configure them and fix them when they break, but generally... I just get them to work and make sure they keep working, I never get to play.

That is where my home server closet comes in. My home server closet is my playground. I get to play with things, test out new software, basically break things and have fun in the process. My server closet at home helps me do that.

It really is a closet, a small walk-in closet 4 foot by 3 foot, with a bakers rack full of computer equipment, monitors, keyboards, cables, and the like. There are servers in there too. Three of them actually.

Server 1 is my media server (pragmatic). This server handles streaming duties, video from an onboard tv tuner (mythtv) as well as some streaming mp3s (icecast).

Server 2 is my development server and general all around do boy (tragic). This server handles samba (windows based file sharing), tinydns (dns for the internal network), dnscache (to handle dns requests to the clients in the local network), apache2, PHP4, PHP5, SVN, etc...

Server 3 is my phone system server (magic). This server handles only one thing, my phone lines. It serves as a media gateway allowing me to connect POTS (Plain Old Telephone System) lines to VoIP services like Voneage (which I don't use, but there are others out there like them, just not as popular). The software that handles this is called Asterisk, and by all accounts it is a pretty amazing piece of open source software made by a company called Digium. The software itself is free, but Digium makes their money off the hardware, and considering how good the software is... I am inclined to throw money their way as often as I can.

Anyways...

Server 3... "Magic" has been the bane of my existence since I put it into service about a year and half ago. It isn't so much that the hardware is bad, but rather the hardware in relation to the environment it was in. Magic has always had an IBM Ultrastar 9GB drive since the day I built it. 10K RPM of SCSI lovin. It is fast... really fast... and pretty reliable... only one or two problems. It is HOT and LOUD. Really loud.

So enters the paradox that is the server closet. The server is loud, as such, I don't want the noise in my home office, so I close the door. Enter the other problem, heat. As I close the door to the closet, temperatures in the server closet climb to insane levels. So then I have to open the door to let the heat out. Lather, Rinse, Repeat.

So finally today... the heat was unbearable... and was starting to effect the PBX cards in the machine. So I decided to see why. Turns out the PSU fan died, and the computer was just roasting inside. So I pulled it out to replace the PSU and decided that I wanted to remove the noisy SCSI drive. So I start looking around for my Ghost disks... Long story short... no matter what I tried Ghost refused to see the SCSI drive.

Enter my savior: I have had a copy of Ultimate Boot CD for a couple of years now. So after blowing a couple of hours with Ghost. I figured what the heck. I threw it in, did a little g4l (Ghost 4 Linux) diskcopy sc0 wd0 mojo and it did its thing. I rebooted fully expecting it not to work... but much to my surprise, it worked flawlessly. Just a couple of adjustments to the grub loader to allow it to boot from the IDE drive instead of the SCSI drive and I was in business.

So now... I am basking in the quiet and reduced heat levels in my server closet. Life is good, and geek points restored.

Mod_Rewrite: A Deeper Look

2005-09-17T01:03:00.000-05:00

I have used mod_rewrite to great extent in the past with great success. It is great to handle search engine friendly urls. This article won't be covering how to do that sort of thing as that has been beat to death, rather this article will show you some of the quirks of using mod_rewrite with apache and apache's different configuration options.

I have used mod_rewrite a great deal and I consider myself to be well versed in how to set it up and use it. That being said, I learned a couple of little quirks about using it in the apache config file and how these might surprise you.

Prior to today I have always used mod_rewrite in a .htaccess file. I feel safe in saying that 100% of my usage of mod_rewrite to date has been via .htaccess. So today when mod_rewrite wasn't working I tried to use my normal methods of debugging and was stumped as to why it wasn't working.

First off a little background: We were running mod_rewrite from within the httpd.conf file. Further more we were actually calling RewriteEngine On from inside a virtualhost apache directive. In the past people have told me this is a faster way of using mod_rewrite so we decided to use it as this was a some what speed sensative server. What I discovered is that running mod_rewrite within this directive changes the way mod_rewrite directives work.

Previously when using mod_rewrite via .htaccess I was always able to turn on logging using the directives in the "root" level of the httpd.conf file:

RewriteLog "/var/log/apache/modrewrite.log"
RewriteLogLevel 10

Lesson 1 learned: This works great if you are running mod_rewrite from a .htaccess, however it has zero effect if you are running mod_rewrite from inside a virtualhost directive. For this to work properly in the virtualhost you must place the RewriteLog directives inside the virtualhost directive where you are using RewriteEngine On. Good to know...

Once I turned on the logging I was able to determine that my regex pattern matching for mod_rewrite was missing my target.

Lesson 2 learned: When you are running mod_rewrite in virtualhost you can not use RewriteBase. This means you must manually correct for this in your regex to match the leading / and the trailing / match.

Hopefully those will help out others who aren't getting the results the expect.

And now for those of you that have made it this far down, here are some fun things you can do with mod_rewrite.

Have all files that don't have an extension be passed to the PHP parsing engine. This is particularly usefuly for creating scripts that look like directories. So you have http://somedomain.com/foobar/action/1/name/smith/ and first part of the url (foobar) is actually a PHP script that the rest of the url gets passed to. Sneaky huh? Here it is:

<FilesMatch "^([^\.]+)$">
ForceType application/x-httpd-php
</FilesMatch>

I hope this posting helps others in using the powerful apache module mod_rewrite and maybe prevent some lost hairs in the process.

How Is It Possible?

2005-09-15T19:53:00.000-05:00

How is it possible that people who are running an online company don't have a clue when it comes to technology?

Case in point today: I was handed a project to take care of for a client. They want me to take input from a form, validate it, store it in a local database and then create a CSV file and send that via FTP to a remote server. Not a difficult request, but also not a terribly secure one either. So I decided I would call the company that was supposed to be getting the CSV via FTP and see if I could just POST the data to a web form instead. So I called them...

Me: "Yeah, hi. This is so and so from company and I was wondering if you guys had a web form that I could send this CSV/FTP data to instead of the CSV/FTP method."

Other Guy: [:long pause:] "Uhm... I have no idea what you are talking about. The CTO is out of town and won't be back until tomorrow. Let me ask [so and so]."

Me [now with so and so on the phone listening]: "Yeah I would like to just send this via curl data post."

At this point I could tell that the loud sucking sound coming from the other end of the line wasn't a good sound. So I tried to explain myself further.

Me: "You know... An online form... Where you type stuff into it, and it saves it some place... like a database... and then you guys can create your CSV from that instead. It would save me a lot of time."

Other Guy again: "Well this plan has been laid out for two weeks and it is supposed to go live tomorrow. Don't you think it is a little late in the game to change things?"

At this point I was annoyed... Here I am two hours into this project, and he is talking down to me like I had been there through out the entire thing?!? WTF?

Me: "You know you are certainly right _IF_ I had been involved at that point. And I wasn't. So... getting back to my question, can you do it?"

Other Guy: "Well we would have to get the CTO involved."

You have got to be f*cking kidding me. You need the CTO to tell you how to make a web form? And you people run a web based business?!?!

At his point I decided to cut my losses.

Me: "Okay I will create the CSV and FTP the file. I will need the FTP information as well as the CSV format."

Other Guy: "Well we have already sent that to Bob."

Me: [leaning over to Bob] "Did you get that CSV formatting and FTP info from ComapnyB?"

Bob: "Nope they never sent it."

Me: "He says you never sent it. So please send it so that I can continue my work."

That was 3 hours ago. I am still waiting for the CSV file.

Its no wonder the dotcom bubble burst. If any of the companies Were run this way its a miracle they survived.