The school's weather station webpage seems to have stuffed it sometime around Thanksgiving. Today somebody finally noticed and alerted the support staff.

My boss asks "where's the documentation?".

Right.  There is none. This system has been in place for ten years or more and fails occasionally.  When that happens we go in and fix it.

Start with the webpage that actually displays the data. It's pulling the data from a file that is supposed to be automagically updated. Except we don't believe in magic. The file didn't get updated. Now to find out why.

Since this is a scheduled event, cron has to be involved. Let's have a look at the crontab file. Hmmm. It's pulling the changes from another file that is supposed to be automagically updated. That one hasn't been changing either. What changes that file? It isn't cron. Or is it? That file is symlinked to a file on another computer. Let's go have a look at the other computer. Ah, I see. There's a crontab running there which generates the contents of the update file from a data file via a collection of python scripts. Let's have a look at those. 

As I suspected, they are pulling data from yet another file that is automagically updated. Right. It hasn't changed since November either. What changes this file? Time to scan the logs. Nothing.

OK, it's time to start from the other direction. The weather station is connected to a PC in the corner of a lab. Let's have a look there. It's hung and totally unresponsive. OK, maybe that's the problem. I reboot it. Then go back to the webpage. Nope. Nothing has changed.

OK, somehow the data has to get from the weather station computer to the other computer where the python scripts can munge it. Let's have a look at the logs.

The logs say everything is fine, but it isn't fine. Nothing. It's not happening. Well this is interesting. I check connectivity and network connections. They're OK. We've got an IP addess and pings work just fine. A closer look reveals that there's a Windows task scheduler which occasionally FTP's the weather files across the net to the second Unix box. The logs don't show any errors. Hmmm. The files aren't being FTP'd though. They aren't making it. Then I see a notice at the bottom of the screen. Updates were applied some time since the computer was last powered on - six months ago. OK, what updates? Windows firewall. Right. So I have a look, and sure enough the computer's FTP connection has been firewalled because of an automatic update. The FTP's are silently failing - and indicating success. This is pure evil. After several minutes I'm able to get in with an administrator account that can fix the firewall and do so. 

Then have another look. Still nothing happening. What could be the problem now? Ah, on reboot FTP is automatically disabled on the weather station software - again without any warnings. The logs again say everything is working and files are being transferred. More evil. What's the use of having log files if they lie to you? I turn on the FTP. Bingo - now the files get through. Now back to the second computer to manually process the files and dump them into the directory where the third computer can pick them up. Then back to the third computer to manually update the processed files.

Yay! It works.

Back to the documentation. How would somebody document stuff like this? There's just too much that can go wrong. I could use up a tree or two writing it all down. This is why we've got systems folks. 

Some rough weather over in California...

---

Fierce winds toppled trucks on a major Bay Area bridge and knocked out power to more than 100,000 people in Sacramento as wicked winter weather moved into California on Friday.

Tsering Gyurmey snapped this image of an overturned truck on the Richmond-San Rafael Bridge Friday.

Forecasters said the Pacific storms could dump more than 10 feet of snow on California mountains by Sunday.

Winds in the mountains could gust to 145 mph, forecasters said, the strength of a Category 4 hurricane. A Category 4 can inflict extreme damage.

---

Yowza! 10 feet of snow! 145mph! That's a serious storm. Hope you folks make it through OK.

Let me put this into perspective. I was in 100mph winds once in Colorado. There were full trash dumpsters rolling down the street, and I could barely stand up. Walking was a process of planting one foot ahead of you, wait and regain your balance, then plant the other foot. Several folks lost the roof of their house, which just blew right off.  And what they're predicting is half again more powerful than that.