Mike Macgirvin
Diary and Other Rantings
Beyond Silicon Valley
   
Friday, May 16 2008, 12:43 pm
Oct 20, 2005
Another hurricane - this is 'W' and the last of the series.
Another hurricane - this is 'W' and the last of the series. More earthquakes. Flooding in Boston and Los Angeles. Record snowfall in the Rockies. It's been quite a year. Think Mother Nature might be trying to send us a message?

Every so often I get the strangest stuff in my web server log. I've scratched my head over it many a time. Looks like somebody has downloaded my weblog software and decided to go through every line of the source code and turn it into a web request. These requests always fail. What kind of clueless idiot would do such a thing?

Finally figured out the answer. It isn't a human at all. It also isn't turning every line of the source file into a web request - only those that contain the string 'href='. It's a very poorly written web crawler. It grabbed my source code file and ignored the fact that it's a file of type 'application/x-tar'. It thinks it's a normal web page (which would be type 'text/html'). And it is extracting everything on the page that looks remotely like a URL and fetching it. The only problem is that these aren't URL's. They are computer code which constructs URL's. They are nonsense unless you run it through a processor to execute the code -  which then creates the URL.  

Comments? | More Actions Open/Close menu
Back
I reverently believe that the maker who made us all makes everything in New
England, but the weather. I don't know who makes that, but I think it must be
raw apprentices in the weather-clerks factory who experiment and learn how, in
New England, for board and clothes, and then are promoted to make weather for
countries that require a good article, and will take their custom elsewhere
if they don't get it.
-- Mark Twain