Today was a fun day. I have a buddy that has spent the last 3 or 4 months (maybe longer) working on a complete ground up redesign of a PHP/MySQL website. The current code base was horrid, and that is putting it lightly. The code was so poor that it was hampering the overall performance of the server. The pages were very slow to display, often taking 30-45 seconds to complete the rendering of one page. Looking at the top process list I could see that several httpd processes were dying off, and dying off frequently. I had assumed that this was due to the poor codebase of the existing application.
The launched the new codebase today, and things didn’t improve. So my buddy called on me to help him out. First a little bit of background on the server: Dual Athlon (yes it was weird to me too), 1GB of RAM, and 1 IDE HD (again, yes I know who ever speced out this server should have their head examined). Server is running Cpanel 10.8.1-C112 under Fedora Core 2. We had rebuilt apache already (using easyapache) and the httpd zombies kept occuring. They weren’t true zombies, only momentary zombies, because the child process was dying without permission from the parent process, so they only showed up in the zombie count for a couple of seconds, so they were being cleaned up properly.
Here is what we found: the main site for the server is a heavily trafficed PHP/MySQL based site with a lot of images. I started looking through the error logs and came across a bunch of messages like:
[notice] child pid 10009 exit signal File size limit exceeded (25)
After some quick googling we were able to determine that this message means that the child httpd process died, because it encountered a file that is too large for it to handle. The max file size is 2GB in this instance. There is some discussion on if this is an Apache 1.3.x limitation or a limitation a little further down (glibc or filesystem).
How did we fix it? Well short term was to shutdown apache and move the offending log file out of the way, and restart apache. Longer term fixes involve some settings inside Cpanel itself. You basically need to tell Cpanel to remove the log files after it has processed the log files.
Log in to WHM and click on “Tweak Settings”, then check mark “Delete each domain’s access logs after stats run”
That will delete the log files daily, and all should be well with the world.