Last week I was trying to help Pazu in Vienna debug a problem where file uploads in a PHP application weren’t working on my web server. We’d been running around in circles trying to figure out what was wrong. First we thought it was the PHP version, but that wasn’t it. Then we thought permissions, but that wasn’t it. The bizarre thing was the using an alternate instance of Apache on the same server using the same software and an only slightly altered httpd.conf worked fine.
Today I was able to get the Apache configs converged to the point where there were only about 4 lines in the two configs that were different, and those were only paths and port numbers. The only other difference was that the broken server had all the virtual hosts configured, while the working server only had one virtual host configured. Thinking it might be that it worked on the first (default) vhost, I added a second, but the test program still worked on both. Then I added ALL the virtual hosts, and suddenly it broke.
So here’s what the root problem turned out to be: http://bugs.php.net/bug.php?id=32491&edit=3
Basically, the old stdio method of handling file descriptors has a limit of 256 file descriptors. Most modern OSes can support many more (usually 1-64K), but to be able to use them you need to make minor changes to the code. Unfortunately, parts of the PHP code still use the old method and are therefore limited to 256 descriptors. One of the parts so limited is the file uploading code. This particular bug with file uploading has been fixed in the current development version, but not in a released version of PHP yet.
Fortunately, we were just a little bit over the 256 file limit, so I was able to squeeze things down to below that by removing some old defunct vhosts, and merging vhosts that refer to the same server under different names (for example foo.com, foo.net and foo.org may have had different vhosts but all got served out of the same directory). Once the server had less than 256 open files, the file upload problem disappeared.
The aggravating factor in this is that I have per-vhost log files, and log files in apache are all held open constantly, so each one means one more file open for all the children of the main httpd. I probably should get with modern times and roll things back to one main log file with an extra field for the vhost.
These kinds of obscure problems are very difficult to diagnose.