{"id":277,"date":"2003-08-31T10:10:35-04:00","date_gmt":"2003-08-31T18:10:35+00:00","guid":{"rendered":"http:\/\/peterjanes.ca\/wordpress\/?p=277"},"modified":"2003-08-31T10:10:35-04:00","modified_gmt":"2003-08-31T18:10:35+00:00","slug":"subtleties-of-robotstxt","status":"publish","type":"post","link":"https:\/\/peterjanes.ca\/blog\/2003\/08\/31\/subtleties-of-robotstxt\/","title":{"rendered":"Subtleties of robots.txt"},"content":{"rendered":"

I recently discovered a subtlety of the \/robots.txt<\/samp> file. (For those who don’t know, \/robots.txt<\/samp> is a configuration file<\/cite> for web spiders that tells them what URLs not to retrieve from a website.) The issue is this: parsers will not fall back<\/cite> to a more general User-Agent<\/samp> setting if they’ve already matched a specific one. This is actually spelled out in A Standard for Robot Exclusion<\/cite><\/a> — If the [User-Agent<\/samp>] value is ‘*’, the record describes the default access policy for any robot that has not matched any of the other records<\/em><\/q> — but I’d forgotten it when I updated my \/robots.txt<\/samp> recently to exclude certain additional URIs (removed from my site due to abuse) that Google<\/a> seems to have such a fondness for that it won’t drop them from its index.<\/p>\n\n

Here’s an excerpt from \/robots.txt<\/samp> that shows the error:<\/p>\n\n

User-agent: Googlebot\nDisallow: \/CBP\/\n\nUser-agent: *\nDisallow: \/cgi-bin\/\nDisallow: \/personal\/<\/code><\/pre>\n\n
Imagine my surprise when Googlebot started spidering \/cgi-bin\/show?user=Webmaster<\/samp>, my address-obfuscating contact form, and pages under \/personal\/<\/samp> which poison<\/cite> the lists of e-mail addresses and URIs that address harvesters<\/cite> collect.<\/p>\n\n
I’ve updated \/robots.txt<\/samp> to include both sets of URIs for the Googlebot, but (unlike Inktomi’s Slurp<\/cite><\/a>) Google doesn’t retrieve \/robots.txt<\/samp> after it’s started spidering a site.  I may have to extend my .htaccess<\/samp> blocking for the current dance<\/cite><\/a> to prevent the big G from grabbing a bunch of (more) useless garbage from my site.<\/p>\n\n
Moral of the story: even if you’ve read the documentation, read it again before making assumptions.  As my friend John<\/a> is fond of saying, when you assume<\/em> you make an ass<\/em> of u<\/em> and Dave<\/em>.<\/q><\/p><\/div>
<\/div>","protected":false},"excerpt":{"rendered":"I recently discovered a subtlety of the \/robots.txt file. (For those who don’t know, \/robots.txt is a configuration file for web spiders that tells them what URLs not to retrieve from a website.) The issue is this: parsers will not fall back to a more general User-Agent setting if they’ve already matched a specific one.…","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"mf2_syndication":[],"venue_id":0},"categories":[3],"tags":[],"kind":false,"_links":{"self":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts\/277"}],"collection":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/comments?post=277"}],"version-history":[{"count":0,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts\/277\/revisions"}],"wp:attachment":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/media?parent=277"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/categories?post=277"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/tags?post=277"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}