{"id":398,"date":"2004-10-03T22:10:29-04:00","date_gmt":"2004-10-04T06:10:29+00:00","guid":{"rendered":"http:\/\/peterjanes.ca\/wordpress\/?p=398"},"modified":"2005-07-24T10:39:26-04:00","modified_gmt":"2005-07-24T15:39:26+00:00","slug":"bad-bots-what-you-gonna-do","status":"publish","type":"post","link":"https:\/\/peterjanes.ca\/blog\/2004\/10\/03\/bad-bots-what-you-gonna-do\/","title":{"rendered":"Bad bots (what you gonna do?)"},"content":{"rendered":"

(Written a month ago, but not posted until now due to lack of editing. I figure if it’s public I’ll be shamed into updating it.)<\/p>\r\n\r\n

The following list is gathered from observations of two somewhat-related events. The first event is that I inadvertently sent out Trackback pings from a local copy of my weblog, which set the blogbots all a-twitter; when I realized what I’d done, I began to block access to the URLs in question with HTTP status code 410 (Gone<\/samp>). The second event is an examination of my access logs, in which I discovered (and blocked, generally with code 403) spambots and crawlers that I felt were poorly-behaved.<\/p>\r\n\r\n

Each criterion for badness is a violation of an element in Mark Pilgrim’s<\/a> aggregator behaviour specification<\/a>. I’ve interpreted SHOULD as MUST, as I’m unaware of any good reason for these agents to not follow the suggestion. This isn’t a complete list of user-agents or errors, it just lists the most common ones I’ve seen. Additions and updates are welcome.<\/p>\r\n\r\n\r\n\r\n\r\n
User-Agent<\/th>Flails<\/a>?<\/th>Ignores 403?<\/th>Ignores 404?<\/th>Ignores 410?<\/th>Referrer spam?<\/th>Unconditional retrieval<\/a>?<\/th>Notes<\/th><\/tr>\r\n<\/thead>\r\n
Googlebot\/2.1<\/td>yes<\/td>?<\/td>yes<\/td>?<\/td>no<\/td>yes<\/td><\/td><\/tr>\r\n
Technoratibot\/0.6<\/td>no<\/td>?<\/td>?<\/td>?<\/td>yes<\/td>no<\/td><\/td><\/tr>\r\n
Popdexter\/1.0<\/td>yes<\/td>?<\/td>?<\/td>?<\/td>yes<\/td>yes<\/td>Note: unthrottled spider, no robots.txt; adds \/ to URLs without extension<\/td><\/tr>\r\n
PubSub.com RSS reader<\/td>no<\/td>yes<\/del>no<\/ins><\/td>?<\/td>yes<\/del>no<\/ins><\/td>no<\/td>?<\/td>403 and 410 processing fixed; if no more issues, will move to good bots list<\/td><\/tr>\r\n
Syndic8\/1.0<\/td>no<\/td>yes<\/td>?<\/td>yes<\/td>no<\/td>no<\/td>ignores 301<\/td><\/tr>\r\n
Feedster Crawler\/1.0<\/td>yes<\/td>yes<\/td>yes<\/td>yes<\/td>no<\/td>no<\/td>ignores 301<\/td><\/tr>\r\n
BlogPulse (ISSpider-3.0)<\/td><\/td><\/td><\/td>yes<\/td><\/td><\/td><\/td><\/tr>\r\n
BlogLines\/2.0<\/td><\/td><\/td><\/td><\/td><\/td><\/td>ignores 301<\/ins><\/td><\/tr>\r\n
Yahoo! Slurp<\/td><\/td><\/td><\/td><\/td><\/td><\/td>ignores 301<\/td><\/tr>\r\n
kinjabot beta2<\/td><\/td><\/td><\/td><\/td><\/td>yes, every 6 hours<\/del>no<\/ins><\/td><\/td><\/tr>\r\n
fastbuzz.com<\/td><\/td><\/td><\/td>yes<\/td><\/td><\/td><\/td><\/tr>\r\n
Slower, Friendlier Spiders (BlogShares V1.36)<\/td><\/td><\/td><\/td><\/td><\/td><\/td>ignores 301<\/td><\/tr>\r\n
FeedRover 1.0; Headlines Archive<\/td><\/td><\/td><\/td>yes<\/td><\/td><\/td><\/td><\/tr>\r\n
Twisted PageGetter<\/td><\/td><\/td><\/td>yes<\/td><\/td><\/td><\/td><\/tr>\r\n
everyfeed-spider\/1.0<\/td>no<\/td>?<\/td>?<\/td>yes<\/td><\/td>yes<\/td><\/td><\/tr>\r\n
IconSurf\/2.0<\/td>yes (by design!<\/a>)<\/td>yes<\/td>?<\/td>?<\/td>yes<\/td>?<\/td>\/favicon.ico<\/code> is stupid in the first place<\/a><\/td><\/tr>\r\n<\/tbody>\r\n<\/table>\r\n\r\n

Good bots<\/h4>\r\n

Giving credit where it’s due, here are the bots (and aggregators) that have been well-behaved so far:<\/p>\r\n\r\n

    \r\n
  • NewsMonster<\/li>\r\n
  • nntprss\/0.4-beta-6<\/li>\r\n
  • SharpReader\/0.9.5.1<\/li>\r\n<\/ul><\/div>
    <\/div>","protected":false},"excerpt":{"rendered":"(Written a month ago, but not posted until now due to lack of editing. I figure if it’s public I’ll be shamed into updating it.) The following list is gathered from observations of two somewhat-related events. The first event is that I inadvertently sent out Trackback pings from a local copy of my weblog, which…","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"mf2_syndication":[],"venue_id":0},"categories":[3],"tags":[],"kind":false,"_links":{"self":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts\/398"}],"collection":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/comments?post=398"}],"version-history":[{"count":0,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts\/398\/revisions"}],"wp:attachment":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/media?parent=398"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/categories?post=398"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/tags?post=398"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}