« Posts under Uncategorized

How To Find and Remove Windows Rootkits

First of all I want to wish everyone a very Happy New Year. It has been a while since I made my last posting and I want to apologize for that.

Now let’s get down to business…

So recently we discovered that a Windows 2003 server had been exploited via an apparently well known 0day exploit in MailEnable’s SMTP service. This has since been corrected by the MailEnable developers (you can read about that here).

At first there was some doubt as to whether or not there was a hacker on the server. Our first clue was the abnormal amount of traffic the server was doing. Typically this server moved about 100-200KBps per day. We knew something was up when this server started moving 2MBps. Upon inspection of the server we couldn’t see anything out of the ordinary, however we did notice that taskman.exe (Task Manager) was running at 100% CPU utilization when ever we looked at it. This threw up all sorts of red flags to us, and we knew that we had a hacker on the server and we needed to find out what they were doing.

I have to say that Event Viewer is your friend. You must look at it every once in a while to make sure you know what is going on. Even with a hacker on the server and rootkit installed on the server to hide his activity, he still wasn’t able to hide some log entries in Event Viewer. After looking in Event Viewer we noticed several references to VMWare. After asking around, we determined that none of the legitimate administrators had installed VMWare and we knew that this must be the hacker.

Here is how we found him, and how we removed him.

We could see that there were some hidden directories on the server that we couldn’t access through the normal Explorer interface, so we knew we were dealing with an on boot rootkit. (You can see file accesses using the file system and disk tools from Windows SysInternals Tools.)

We installed HiJack Free from a-squared. This piece of software is pretty powerful and does some deep inspections of the registry to find services and applications that are not normally visible in the Control Panel services listing. We sorted the services and looked for services that were set to start at boot up, and looked for anything that wasn’t signed. Hi-Jack Free displays the company that signed the driver/service, and any service that was set to run, but wasn’t signed was on our hit list to remove.

With a list of services to disable, we installed the Windows Recovery Console and rebooted the server into the recovery console. We disabled the services that we identified as problems and rebooted the system normally.

At this point we could see the directories that were hidden from us earlier. We discovered that the hacker had installed VMWare. Because we wanted to see what the hacker was actually doing with the VMWare installation, we used Virtual Disk Driver to mount the WMWare disk images to see what they were doing. Turns out they were downloading Pokemon episodes. Ugh… All that hassle and it wasn’t even anything good. :)

So we removed the rootkit from the server, removed the VMWare installation, and patched the MailEnable install, and the server has been cruising along ever since.

We hope that this description of what we did will help you find and remove Windows rootkits on your servers.

How To Install NetBSD as a DomU in Xen 3.0

Ever since the first time I heard about Xen and its ability to run any OS side by side on the same server I have had the urge to run a BSD based OS with a Linux OS. Today I have sucessfully achieved my goal, and this is how I did it.

First some background on the server itself. The server is a Dell PowerEdge 1750 with Dual Xeon processors and 3GB of RAM and 500GB of RAID storage. The server is running the Xen 3.0.2 hypervisor kernel (the main kernel that handles the paralization, or virtualization, of the hardware). The Dom0 system is running Debian 3.1 with some patches to the kernel to work with the LSI based RAID 5 card in the server. Each virtual OS installed on the server is given its own partition and is managed using LVM in Dom0.

The vast majority of information about NetBSD running under Xen as a DomU seems to be either Xen 2.0 specific, or assumes you are running NetBSD as Dom0. Unfortunetly, the Xen 2.0 information is no going to work on a Xen 3.0 machine, and more so our Dom0 is Debian, so we needed to come up with our own.

Here is how I did it, and what sort of problems I encountered.

The entire process is pretty easy, but finding the actual information can be tough, and finding the files you need can be even tougher. Here is kind of a rough over view of the process…

1) Set up your partition that will hold NetBSD. We are using a LVM partition named vg00-netbsd.
2) Set up the xen domU config file.
3) Boot the netbsd install kernel for Xen 3.0.
4) Follow the sysinstall steps like you normally do to install NetBSD. I had to use an FTP based installation, because I could never get the CDROM to work correctly.
5) Complete the install and shutdown NetBSD.
6) Edit the domU config file and change the kernel from the install kernel to the normal NetBSD kernel.
7) Boot NetBSD DomU and enjoy.

So here are the specifics.

Step 1: You need to download the NetBSD Xen 3.0 kernels (install and normal) and put them some place on your Dom0. I put mine in the /boot of the server, because it sort of made sense to me, but they can be almost anywhere. You can download the DomU kernels from NetBSD’s FTP servers from the daily build areas. The kernels for Xen 3.0 are not in the release versions of NetBSD so you have to find them. I would post links to them, but most likely the would go stale over time. Go to ftp://ftp.netbsd.org/pub/NetBSD-daily/ and navigate through to either the NetBSD 3.1 tree or the NetBSD 4.0 tree. You are looking for the directory i386/binary/kernel/ in that directory you will find the two kernels you need. The install kernel is called netbsd-INSTALL_XEN3_DOMU.gz and the normal kernel is named netbsd-XEN3_DOMU.gz. Download both of those kernels as you will need them later.

Step 2: Once you have downloaded your kernels you will need to create a xen config file for your NetBSD DomU. Here is an example of the one I used:

kernel = “/boot/netbsd-INSTALL_XEN3_DOMU.gz”
memory = 128
name = “netbsd”
vif = [ '' ]
disk = [ 'phy:/dev/mapper/vg00-netbsd,0x01,w' ]
root = “/dev/wd0d”

You will need to change the disk = line to match where you are installing NetBSD to on your server. After you have created that file in your xen config directory (our was /etc/xen/).

Step 3: We are ready to boot NetBSD for the first time. To boot NetBSD we run the command:

xm create -c netbsd

“netbsd” is the name of the DomU config file we created in step 2, so change that to match what you used in that step.

A couple of times we noticed that Xen didn’t attach us to the console of the booting NetBSD DomU, so you may need to connect to it manually. To do so do the following:

xm list

Which will print out a list of running Xen instances like this:

Name ID Mem(MiB) VCPUs State Time(s)
debian 0 1374 4 r—– 2354.5
plesk 10 1024 1 -b—- 161.5
netbsd 50 128 1 -b—- 1.2
qmail 8 128 1 -b—- 953.4

We will need to know the ID of the instance we want to attach to. In the example above this is 50. Then we attach to the console of that DomU by typing:

xm console 50

To break out of the console at any time simple press CTRL+] at the same time.

Step 4: Once you are in the console you should see the sysinstall application. You can follwo the prompts and install NetBSD like you would normally do. One problem I did encounter was that for what ever reason the server would stop talking to the FTP server due to some sort of DNS lookup failure. It did this no matter which kernel I tried. I eventually resorted to using the IP address instead, and the installation worked perfectly.

Step 5: Once the install is completed, break out of the server and shut it down via the command:

xm shutdown 50

Again replace 50 with the ID of the DomU of your NetBSD install.

Step 6: Edit the DomU file and change the kernel line to point to your normal NetBSD kernel. So your DomU config file should look somethnig like this now:

kernel = “/boot/netbsd-XEN3_DOMU.gz”
memory = 128
name = “netbsd”
vif = [ '' ]
disk = [ 'phy:/dev/mapper/vg00-netbsd,0x01,w' ]
root = “/dev/wd0d”

Step 7: Reboot your NetBSD DomU via the command:

xm create -c netbsd

Enjoy your NetBSD running under a Debian/Linux Dom0 in Xen 3.0!


Having used the NetBSD system only breifly I have noticed that there is something “funky” with the networking and the way it behaves. I noticed over a sustained ping that the network interface starts to drop packets, every other packet it seems. Modifying the vif = line in the DomU config to read:

vif = [ 'bridge=xenbr0' ]

seems to have cleared up the issue. This line bridges the ethernet interface inside the DomU to the xenbr0 interface in the Dom0. It seems to have cleared up the issue to date.

And there
you have it! NetBSD running under a Linux Dom0 on top of Xen 3.0. The world just got a whole lot smaller.

Why You Shouldn’t Use Fedora Core For Production Servers

I am going to vent a little here.

I don’t understand why web hosting companies (and dedicated server providers for that matter) continue to provide Fedora Core based servers to customers. It just doesn’t make sense. Fedora Core was never intended to be used in a production enviroment and why people keep using in those enviroments baffles me.

We had a server that was running Fedora Core 4 and cPanel and has been in production for almost a year. It was always updated and properly maintained… that is until RedHat dropped updates for it (which they do every 6 months as that is the development cycle for Fedora). Dropping updates isn’t that big of a deal as there is the Fedora Legacy Project that continues to provide updates after RedHat has dropped updates.

Here is what happened with this server, Fedora Core 4 was dropped from RedHat support, and was being taken over by Fedora Legacy. No big deal, this has happened a bunch of times in the past with previous Fedora Core versions. The problem enters when you throw in a local kernel exploit that is discovered for the /proc fs. We attempted to upgrade this kernel to prevent this problem, but because FC4 was in a state of transition between the two providers, the updates weren’t available yet. As such this server was put into the “holding for update” queue.

In that time the server was hacked and needed to be wiped and reinstalled with another OS that won’t cause this particular problem (CentOS 4.3).

What we recommend to customers instead of Fedora Core?

CentOS: RedHat Enterprise Linux based OS. They grab the source RPM files from RedHat and remove the images that reference RedHat and replace them with CentOS based images, and other minor modifications, but generally this is the same code that is used to run RedHat Enterprise Linux. Because it is based on RHEL it has a good development cycle as well as excellent support for new drivers. Does it have it’s problems? Yes, any problem that exists in RHEL will also exist in CentOS (various spin_lock kernel panic issues have been documented).

We recommend the use of CentOS for any application that does not support our number one OS of choice.

Debian (stable): I admit it. I am a big fan of Debian. It works great and I am very familair with it. It handles in place version upgrades (moving from woody to sarge was done with minimal effort using apt-get dist-upgrade). It has an easy to use package manager, with a large selection of applications, as well as an active developer community. My biggest problem with Debian is the that the kernel gets out of date very quickly. Unlike CentOS which actually backports drivers from a more current kernel revision (for instance 2.6.16) into the current standardized kernel for the OS (2.6.9 for CentOS 4.3), Debian does not do that. So it is often a chore to get Debian to run (or even install) on newer chipsets.

Overall, Debian is the winner for us. We install it anytime we have the option (even for Virtual Private Servers) as it just works. It has its faults and is by no means perfect, but we are willing to over look those as they are minor in comparison to the benefits it can provide in ease of management and administration.

So to recap:

  • Don’t use Fedora Core for a production server, it will only cause you pain and suffering in the long run.
  • Anyone that suggests you use Fedora Core in a production enviroment has not analyised the potential ramifications of that decision completely.
  • There are many viable alternatives that are actively developed and maintained as well as providing up to date and current device driver support and security updates.

In conclusion:

Don’t use Fedora Core on a production machine.

Steps to Tighten PHP Security

Recently I have had to deal with some insecure PHP scripts on a couple of servers that have caused us some serious problems, as well as time, to recover from.

I am going to list some of the important steps you can take to protect your server and your site from insecure PHP applications. This will by no means a complete list of items to stop them, but this should help prevent most of the basic attacks (“script kiddie” attacks – where the attacker doesn’t have a great deal of skill and is depending on a preset scenario).

Most of these solutions will assume you have some level of control (root) over the server.

  • Turn off allow_url_fopen: This modification will prevent the use of http://www.domain.com/somescript.php and ftp://www.domain.com/somescript.php from being used in include(), include_once(), require(), require_once(), as well as fopen(). This will prevent a hacker from including malicious code remotely. This modification will prevent the vast majority of hack attempts in PHP from working.

    The downside of this modification is that some legitmate applications that use fopen() to open a remote web page might be broken. You can encourage users to use the PHP curl functions instead as they accomplish the same results.

    Edit your php.ini and change allow_url_fopen = On to allow_url_fopen = Off and restart your web server.

  • Use open_basedir: open_basedir allows you to dictate which paths PHP is allowed to access for a given site. Generally we set this to the path of the website (/var/www/html), /tmp (for writting session data as well as for file uploads), and the path to the PEAR repository.

    The good thing about open_basedir is that it is default deny, meaning if you don’t specify the path in the open_basedir it is blocked.

    The bad part about open_basedir is that it come sometimes block access to legitmate applications needed by your PHP applications (Gallery is one that comes to mind, as it makes use of some external applications in /usr/bin, so you must open access to that location).

    Generally, you should use this directive for every website you host as it allows you to control which directories PHP can access and you can make sure that those directories have the correct permissions to prevent potential exploitation.

  • Mount /tmp noexec, nosuid and nodev: This one is particularly useful as it allows you to tell the operating system not to run any applications in a given area of the hard drive. I can tell you personally that this one as saved me numerous times. When used in combintation with the open_basedir directive you can effectively limit what the hackers have access to, and what they are able to run.

    If you have a seperate /tmp partition on your hard drive you can edit the /etc/fstab file and change the options for the /tmp line.

    /dev/hda3 /tmp ext3 noexec,nosuid,nodev 0 0

    If you don’t have a seperate /tmp partition you can create one as a loopback filesystem and mount that as noexec. I will make a seperate post on how to do this later.

  • Mount your web space noexec: This might not be possible, but is a great option just like mounting /tmp in noexec. The reason for this is to prevent binary applications from being uploaded and executed in your webspace. Like I said this might not be possible, but is an excellent prevention method to prevent local root exploit binaries (where a binary exploits a problem in the kernel to gain escallated priviledges on the server, the /proc race in the 2.6 kernel is a recent example of this type of exploit).

  • Make sure your permissions are correct: You should not allow world write permissions on any files, other than those required (configuration files that are written by the web server, etc). This is very important as it will save you a great deal of trouble in the event that a hacker does gain access to your site, as they won’t be able to overwrite your files. Additionally, you should only allow write access to specific folders where needed.

  • Prevent PHP from parsing scripts in world writtable directories: This will prevent hackers from uploading malicious scripts and running them from your site. This is a great way to tighten security, and is pretty easy to manage, just set “php_admin_flag engine off” for any directory that can be written to by the web server user. This one will save you more hassle than you can ever imagine.

  • Install mod_security: Mod Security is basically a web server firewall, allowing you to block specific types of requests, as well as inspect data as it is sent to the server. My only problem with mod_security is that it isn’t default deny, which means you must identify what is blocked, rather than identifying what is allowed and rejecting everything else. Still it has its uses and you should have it installed in order to react to problems that do not have patches available yet.

We hope that this list of items will help protect your server from hackers and allow you to have some peace of mind that your sites are as secure as you can make them.

Linux Things I Learned Today

What fun things did I learn today? Let me see…

  • Intel based ICH7 chipsets are not fully supported in the linux kernel until kernel version 2.6.11. With additional features added in 2.6.15.
  • The Debian Installer is based on the 2.6.8 kernel, and therefore can not install on to hard drives attached to the ICH7 chipset (at least in SATA2 mode, which is the whole point).
  • The northbridge chipset on the MSI 945GM2 motherboard is OMG HOT! (I have the burn marks to prove it.)
  • Motherbards are moving to a new auxillary power connector (in addition to the ATXv2.0 spec). This new connector is called an EPS12V connector and consists of an 8 pin connector for 12V power to the CPUs.
  • The EPS12V power connector seems to be standard on most “Cedar Mill” (Intel’s new 65nm fab CPUs) “Ready” motherboards. Where as most power supplies do not yet have this connector.
  • The case I buy do not have the EPS12V connector on them – which made me sad.
  • The standard 4 pin 12V connector that is standard now on most power supplies DOES work in the EPS12V plug – which make me happy.
  • SATA2 (SATA 3.0Gbps) can be used in “comptability mode” in the BIOS, but it makes them SLOW.
  • The default kernel installed with Debian doesn’t support over 1GB of RAM. You must install another kernel if you have more than 1GB of RAM.

I think that about covers what I ran into today. Just some random facts that might help somebody some day.

Moving Servers Realtively Painlessly

Let’s face it, moving from one server to another server is nasty and something any of us would try to avoid as often as we can. Unfortunetly, there are instances where we can not avoid this and we must bite the bullet and do it.

Here are some tips that I have gathered up that will help you maintain your sanity as much as possible. Honestly even with all the planning and preperation, there will be issues so just prepare yourself mentally for that. It will happen.

So here are some things you can do to help the transition:

  • DNS Caching: DNS has to be one of the most difficult things to deal with, because you only have a certain level of control over what occurs. Caching of DNS results will often cause many problems. The trick here is to plan ahead and remove the caching from the picture until the move is completed.

    Setting your DNS to a low TTL (I use 5 seconds versus the recommended 86400 seconds/ 24 hours). The TTL tells other DNS servers and clients how long they should hold the information they just received, in their cache. Since we are going to be moving the site to a new IP soon, we don’t want them caching the old IP once the site is on the new IP. The low TTL will help prevent this.

  • Pathing: If you are moving to a server with different paths, you should at least try to setup symbolic links to help any hard coded paths in the site’s content.

  • PHP register_globals: This one has bitten me a couple of times. Most ofthen when you move to a new server the server will have the default settings for PHP which turns off register_globals. This will cause all sorts of problems that you won’t be able to track down easily. Check it early on and make sure that both servers match settings wise. (A a general rule register_globals should be set to off, but there are some legacy PHP apps that need it.)

  • SSH Tunnels: Most of the time you should adjust your TTL for your DNS long before you do the actual move, but sometimes that isn’t an option. In a pinch you can use an SSL tunnel to redirect traffic from your old server to the new server. This will help users that have the old IP cached, still access the new site. It should be noted that SSH tunnels are pretty unstable and shouldn’t be really be depended on, but like I said, they are good in a pinch. Here is an example of one, you run this on the old server as root with the httpd process stopped:

    ssh -g -C -N -f -L 80:newserverip:80 root@newserverip

    This will prompt you for a password (if you don’t have key based access already setup). You will get a warning ‘bind: Address already in use’, but the process is working. This message is just letting you know that the remote endof the tunnel has something already on port 80, which is actually a good thing.

    You can read more about this process in my blog post Redirecting TCP Ports with SSH

  • DNS: Once you have everything moved over to the new server, update the current authoritative DNS servers with the new IP as soon as possible. Once you have done that change the domains name server records also. The name server change takes longer to complete (will normally be done in 8 hours at the root servers).

  • How To Deal With a Highly Dynamic Website: For sites that have alot of content, or for sites that have users contributing data often (mainly forums) you need to draw a line in the sand and take the site offline during the move. This ensures that you have all the content and that no one loses any content that was contributed during the move.

  • Rsync: Also in the same vein as dynamic sites (user images etc) rsync is your best friend. Run rsync, and run it often. Run it even before you move. It keeps the two sites in sync with each other and can be run while the old site is still up and running. The more you run rsync leading up to the actual move, the faster the actual move will occur. Faster move == less downtime for your users.

There are many other issues to consider, but if you take these items into consideration you should not have any major surprises coming your way.

Apple Boot Camp, Windows XP and Xen Sitting in a Tree

So today I came across this site for Apple’s Boot Camp.

Basically the concept behind Apple’s Boot Camp is to allow users to install Windows XP side by side with MacOSX, which many people have been trying to do since the first Mac’s shipped with Intel based processors.

So now it is possible to run Windows XP on a Mac with Apple’s blessings.

Here is where things get interesting: Throw Xen 3.0 into the mix. For those that don’t know what Xen is, it is a hardware virtualization technolgoy (based ont he linux kernel) that allows multiple OS’es to run on the same hardware at the same time. It allows concurrent access to the network, hard drives, memory, cpu etc. Prior to Xen 3.0 you needed to have an OS specifically compiled for Xen in order to run it. That has all changed with the new technology included in the latest processors from Intel (and soon AMD – June 2006 I think). The new processors have a technolgoy called Virtualization that allows hardware virtualization applications like Xen (and VMWare) to run OS’es that are not compiled natively for Xen itself (such as OSX and Windows XP).

This is where I think there is a huge opportunity for Apple (and for all I know they might actually be working on this already, I haven’t a clue). To allow Xen to boot OSX as a Xen Dom0 (the OS that is allowed access to the Xen kernel for control over the other OS’es on the virtualized machine). Then boot Windows XP as a DomU. Then a user would be able to boot their computer, access OSX and switch to Windows XP all in one machine.

An interesting thought… If you can get both OS’es to boot at turn on, and used OSX as the Dom0, then you would be able to access the Windows XP virtual machine via the Mac OSX RDP (Remote Desktop Protocol) client and never have to actually leave OSX or figure out how to switch between the two. Oh man… my head spins… I need to get a MacBook Pro and start working on that.

exit signal File size limit exceeded (25) in Apache

Today was a fun day. I have a buddy that has spent the last 3 or 4 months (maybe longer) working on a complete ground up redesign of a PHP/MySQL website. The current code base was horrid, and that is putting it lightly. The code was so poor that it was hampering the overall performance of the server. The pages were very slow to display, often taking 30-45 seconds to complete the rendering of one page. Looking at the top process list I could see that several httpd processes were dying off, and dying off frequently. I had assumed that this was due to the poor codebase of the existing application.

The launched the new codebase today, and things didn’t improve. So my buddy called on me to help him out. First a little bit of background on the server: Dual Athlon (yes it was weird to me too), 1GB of RAM, and 1 IDE HD (again, yes I know who ever speced out this server should have their head examined). Server is running Cpanel 10.8.1-C112 under Fedora Core 2. We had rebuilt apache already (using easyapache) and the httpd zombies kept occuring. They weren’t true zombies, only momentary zombies, because the child process was dying without permission from the parent process, so they only showed up in the zombie count for a couple of seconds, so they were being cleaned up properly.

Here is what we found: the main site for the server is a heavily trafficed PHP/MySQL based site with a lot of images. I started looking through the error logs and came across a bunch of messages like:

[notice] child pid 10009 exit signal File size limit exceeded (25)

After some quick googling we were able to determine that this message means that the child httpd process died, because it encountered a file that is too large for it to handle. The max file size is 2GB in this instance. There is some discussion on if this is an Apache 1.3.x limitation or a limitation a little further down (glibc or filesystem).

How did we fix it? Well short term was to shutdown apache and move the offending log file out of the way, and restart apache. Longer term fixes involve some settings inside Cpanel itself. You basically need to tell Cpanel to remove the log files after it has processed the log files.

Log in to WHM and click on “Tweak Settings”, then check mark “Delete each domain’s access logs after stats run”

That will delete the log files daily, and all should be well with the world.

Missing /proc/megaraid

Recently I did a Xen 3.0 install on a Dell 1750 server with RAID5. By default the Xen 3.0 installation (from source) doesn’t compile in the LSI drivers for the RAID card, so I had to enable the option in the Dom0 kernel in Xen.

So I went in to the xen0 directory under my Xen source (/usr/src/xen-3.0.0/linux2.6.12-xen0) and ran:

make ARCH=xen menuconfig

I then went to Device Drivers -> SCSI Device Support -> SCSI low-level drivers -> and enabled the “LSI Logic New Generation RAID Device Drivers”, because after all… Newer is better right?

I recompiled Xen and the Xen kernels (cd ../; make kernels; make install) and reboot the machine and went on with my day.

I got a call from the client later in the day that the server was reporting a problem with the RAID array. Long stroy short, the customer was running a NAGIOS plugin to monitor the RAID array via the /proc/megaraid entry. Little did I know that the new generation RAID device drivers in the 2.6 kernel no longer make use of the /proc/megaraid proc entry, so it was breaking the NAGIOS plugin.

To correct this problem use the “LSI Logic Legacy MegaRAID driver” instead of the “LSI Logic New Generation RAID Device Drivers” in the 2.6 kernel (this is not an option in the 2.4 kernel). That will give you back the /proc/megaraid entry and you should be all set.

Additionally, since you are already compiling a kernel, you should enable the kernel option “Use register arguments” under “X86 Processor Configuration”. This option will allow you to use the Dell OMSA device drivers for additional control over your Dell PowerEdge server. You can read more about it here: http://www.uta.fi/~pauli.borodulin/dellomsa/omsa44.html

Logs at a Glance

I have many servers that I have to watch on a regular basis. We use a KVM to switch between the consoles of the servers and generally we only see the console for any given server for a couple of seconds.

The KVM we use has this great feature where if it detects any activity on the console, the KVM will swtich to that console and display the output for us.

This is how we use that functionality to quickly display any problems a server might be having. It is by no means a fool proof and accruate, but it does help to bring things to our attention on a regular basis, doesn’t require you to install anything special (all programs should come with a basic install of a linux OS), and its cool… What else do you really need?

What we do is one turn off the console’s screen saver. It is evil, and we don’t like it. Actually it serves a valid purpose, but not for our needs so lets turn it off:

/usr/bin/setterm -blank 0

Now you can put that into /etc/rc.local for RedHat based distros (Fedora, Redhat Enterprise Linux, CentOS, etc).

For Debian you can create a file in /etc/init.d/ and name it blank_screen and chmod 755 the file. Then just create a symbolic link to the startup dir:

ln -s /etc/init.d/screen_blank /etc/rcS.d/S61blank_screen

Now that we have the console set not to turn off, we can do the next part of our monitoring mojo. We will setup to monitor files that are important to us, such as /var/log/messages, and /var/log/syslog. These files might not be the files you want to monitor, so you can modify the line below to match the files of your choosing.

We use tail, because it displays the lines as they are added to the file we are monitoring, and it is simple and lightweight.

/usr/bin/tail -v –follow=name /var/log/messages –follow=name /var/log/syslog > /dev/tty2

What this does is runs tail to monitor the files /var/log/messages and /var/log/syslog. We monitor the filename, rather than the descriptor (which is the default behavior of tail). Monitoring the descriptor means that if the filename is changed — for example logrotate, then we will follow the file to the new name. Since once a file is rotated by logrotate it won’t be updated, we don’t want this behavior. We add the -v (verbose) so that tail will tell us the name of the file that corresponds with the output. Then we redirect the ouput to /dev/tty2 (virtual console 2).

This will output the information directly on the screen so that we can see it via the KVM.

Additionally you might want to turn off the login prompt on the virtual console 2. We can do this by editing the /etc/inittab file and commenting out the following line:

2:23:respawn:/sbin/getty 38400 tty2

You can just put a # in front of that line and then have init re-examine the file by running:

kill -HUP 1

So there you go… you now have the output of any file you want to the console of your server. Set your KVM to autoscan, and enjoy your easy to view logs.