Author: Peter

This word [SHOULD], or the adjective RECOMMENDED, mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

RFC 2119, section 3

The User-Agent request-header field contains information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations. User agents SHOULD include this field with requests.

RFC 2616, section 14.43, User-Agent

SHOULD means SHOULD. From now on, unconfigured web browsing libraries can take a hike: as of this evening I’m blocking bogus user agents, which will be sent to this page. Mark Pilgrim points out in a comment on his recent article on blocking spambots and spybots that there is no legal requirement for bots to identify themselves… most users won’t bother [changing their User-Agent], because they don’t understand how the web works or why it would matter. And not a lot of people block by User-Agent, so it really doesn’t matter all that much. Perhaps there’s no legal requirement, but there is an explicit expectation of proper identification given by the RFCs quoted above. (RFCs are the de facto protocol standards, much as the W3C’s TRs are the de facto content standards.) Because it’s the default is hardly a valid reason not to set a user-agent string.

This isn’t about getting rid of spambots or spybots, although that may be a corollary of the idea. It’s about following the rules–in societal terms, it’s about being polite. I know my piddly little site isn’t going to change the world; if someone can’t retrieve a page they want they can contact me or move on somewhere else and I won’t mind a bit. But if one person looks at the bogus-bot page and says hey, I should do that then I’ll be a happy camper.

Without further ado, here’s the mod_rewrite magic. Share and enjoy (and suggest more!).

RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^Java1 [OR]
RewriteCond %{HTTP_USER_AGENT} ^Java/ [OR]
RewriteCond %{HTTP_USER_AGENT} ^libwww-perl/ [OR]
RewriteCond %{HTTP_USER_AGENT} ^Python-urllib/
RewriteRule .* /bot-redirect [R]

Update: Hmm. I should probably include RewriteCond %{HTTP_USER_AGENT} !.*/[0-9] as well, perhaps in place of the Java1 rule that it supersets. Interesting to note that a couple of the blogbots (obidos-bot and the myelin ecosystem bot in my logs from the last week) would then show up bogus.

About 12 minutes later: Or not. Re-reading section 3.8, I see that product = token ["/" product-version]; product-version = token, i.e. the /product-version is optional and needs not start with a digit anyway. I’m going to claim to have been thrown because both bots misuse the Referer field too.

And another few minutes: Re-reading 14.43. The User-Agent field itself should be included, but additional tokens are only present by convention, listed in order of their significance for identifying the application. So the letter of the RFC shoots down most of my rationale, minus the check for an empty string. Still, I think I’m still following the spirit of 2616:14.43, so I’m going to keep the rules and modify the verbiage of the redirection page slightly.

At least one good thing came out of my Apache problems: I’ve learned enough about mod_rewrite to transparently poison my site to the address harvesters out there. I’m still going to leave the basic address-munging that Movable Type does enabled, but it should be even harder to scan these pages in the first place.

As usual, Mark Pilgrim has a more comprehensive description of how to block spambots et al. than I’ve been fumbling with. I’d say that great minds think alike, but, well, look around Dive Into Mark and look around here and make your own judgement.

A small advantage to not having application/xhtml+xml working is that Stuart Langridge’s Nice Titles will work on this page. That is, until I can figure out how to make them work in the X(HT)ML DOM, and then they’ll work everywhere in the weblog. After some simple changes, the Nice Titles on this site should now work in HTML or X(HT)ML. I did find a bug that I’ve been too preoccupied to fix in his original JavaScript, too: it can’t handle the case where there’s an embedded tag, like the Adrants link in this post.

In other news, Eric Meyer has yet another brilliant line of CSS.

New WaSP member Mark Pilgrim continues to (correctly, IMHO) take issue with NewsMonster, a new RSS aggregator. I’ve seen similar behaviour to this for several months–since November 27, in fact–from a user-agent purporting to be Mozilla/4.0 (compatible; MSIE 5.15; Mac_PowerPC). It seems unlikely that this is an incarnation of Burton’s software–if it is, why’s he looking at my site, of all things–which leads me to believe that there’s something else out there. (The other thing is that at least half of these requests are coming from doorstop.webtv.net–i.e. WebTV Networks, i.e. Microsoft–which is kind of unlikely to be a Mac house.) The user-agent string isn’t listed in the über-user-agent list Mark cites, but I’ve seen other interactive requests from the agent so I’m presuming it’s just something spoofing the agent.

So, whatever you are–to continue what’s turning into today’s theme–Yer outta there!