I was up most of the night getting the webserver working again. Apache failed to compile. So I upgraded. Failed to compile. Took out the mod_dav option I was trying to turn on. It built. Sigh. Since this was an upgrade, I installed it. But then PHP broke. All the websites, hosed. Nothing.  

So I recompiled PHP. But mod_dav wasn't working, and that was the whole point. Not only that but mod_rewrite wasn't working either. Turning on either made the build fail. Couldn't find libgdbm. I'm not using libgdbm. How many times do I have to tell you!?! Apparently, it won't listen. It will configure without libgdbm, but then will complain that it can't find it. Fine, I'll use Berkeley DB instead. Can't find Berkeley DB. And by the way, it can't find libgdbm (which it isn't supposed to be using) either. Oh, and to add insult to injury, it can't find /usr/lib/libgdbm.la - which is there because I installed it anyway, even though I don't want to use it. Why can't you find something that's there and you're not supposed to look for? Why can't you do anything right? Why? Why? 

I went back to last months apache package.  It wouldn't compile at all. The same errors about missing databases that I have no need for.  Now I'm really hosed. Can't get back to my working environment.

At about 3:30 AM I finally pulled out original sources and rebuilt everything in the exact configuration I setup a few weeks ago. (No small feat). Yay! Went to bed. 

Later this morning I started the forensics to figure out what went bad and why. Turns out that it's APR (Apache Portable Runtime). APR is supplied with Apache, but Apache (and APR) are built with a prior version of APR if it exists. I've had so many APR's on this system I can't figure out which one it loaded. The message that it required a newer version of APR made it suspect. Why the heck do you need a newer version of APR if you're going to  BUILD a brand spanking new version of FREAKING APR? What's wrong with this picture?

So anyway, I manually built and installed the APR piece of the webserver, and then used that version to build the rest of the webserver. Mod_dav built just fine this time around. But I'm not installing it today, since I already know I'll have to rebuild PHP again to make it work.  I'll just sit on it for a few days. I've had enough upgrade joy for the week. 

Need sleep.