Photo Matt is seeing something that I’ve noticed for a while as well: Googlebot is making up URLs to retrieve. I’m surprised no one at the big G has heard of RSS autodiscovery; they’ve obviously already got lots of content they could use as a basis. Then again, Googlebot doesn’t recognize application/xhtml+xml pages, either:
64.68.82.28 - - [16/Aug/2004:18:36:53 -0700] "GET /blog/archives/2004/08/16/fun-with-xfn HTTP/1.0" 406 398 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
Following up on some of the comments on Matt’s linkback that indicate otherwise, I’ve discovered that Googlebot sometimes retrieves application/xhtml+xml pages:
64.68.82.18 - - [18/Aug/2004:04:41:20 -0700] "GET /blog/archives/2004/08/16/fun-with-xfn HTTP/1.0" 200 6609 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
Nothing has changed on the page or in the way I’m serving it. Anyone have an explanation for this?
