{"id":388,"date":"2004-08-18T01:04:09-04:00","date_gmt":"2004-08-18T09:04:09+00:00","guid":{"rendered":"http:\/\/peterjanes.ca\/wordpress\/?p=388"},"modified":"2004-08-18T01:04:09-04:00","modified_gmt":"2004-08-18T09:04:09+00:00","slug":"feeding-the-googlebot","status":"publish","type":"post","link":"https:\/\/peterjanes.ca\/blog\/2004\/08\/18\/feeding-the-googlebot\/","title":{"rendered":"Feeding the&nbsp;Googlebot"},"content":{"rendered":"<div class='e-content'><p><a href=\"http:\/\/photomatt.net\/\">Photo Matt<\/a> <a href=\"http:\/\/photomatt.net\/2004\/07\/09\/more-googlebot-flailing\/\">is seeing something<\/a> that I&#8217;ve noticed for a while as well: Googlebot is making up URLs to retrieve.  I&#8217;m surprised no one at the big G has heard of <a href=\"http:\/\/diveintomark.org\/archives\/2002\/05\/30\/rss_autodiscovery\">RSS autodiscovery<\/a>; they&#8217;ve obviously already got lots of content they could use as a basis.  Then again, Googlebot <a href=\"http:\/\/www.xiven.com\/weblog\/2003\/01\/31\/QuestionDoesGoogleSupportPagesSentAsApplicationXhtmlXml\">doesn&#8217;t recognize <code>application\/xhtml+xml<\/code> pages<\/a>, either:<\/p>\n\n<pre>64.68.82.28 - - [16\/Aug\/2004:18:36:53 -0700] \"GET \/blog\/archives\/2004\/08\/16\/fun-with-xfn HTTP\/1.0\" <strong>406<\/strong> 398 \"-\" \"Googlebot\/2.1 (+http:\/\/www.google.com\/bot.html)\"<\/pre>\n\n<p><ins datetime=\"2004-08-19T18:43:00-05:00\">Following up on some of the comments on <a href=\"http:\/\/photomatt.net\/2004\/08\/18\/google-doesnt-read-xhtml\/\">Matt&#8217;s linkback<\/a> that indicate otherwise, I&#8217;ve discovered that Googlebot <em>sometimes<\/em> retrieves <code>application\/xhtml+xml<\/code> pages:<\/ins><\/p>\n\n<pre><ins datetime=\"2004-08-19T18:43:00-05:00\">64.68.82.18 - - [18\/Aug\/2004:04:41:20 -0700] \"GET \/blog\/archives\/2004\/08\/16\/fun-with-xfn HTTP\/1.0\" <strong>200<\/strong> 6609 \"-\" \"Googlebot\/2.1 (+http:\/\/www.google.com\/bot.html)\"<\/ins><\/pre>\n\n<p><ins datetime=\"2004-08-19T18:43:00-05:00\">Nothing has changed on the page or in the way I&#8217;m serving it.  Anyone have an explanation for this?<\/ins><\/p><\/div><div class=\"syndication-links\"><\/div>","protected":false},"excerpt":{"rendered":"I&#8217;m surprised no one at the big G has heard of RSS autodiscovery.  Then again, Googlebot doesn&#8217;t recognize <code>application\/xhtml+xml<\/code> pages, either.  <ins datetime=\"2004-08-19T18:43:00-05:00\">Or does it?<\/ins>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"mf2_syndication":[],"venue_id":0},"categories":[3],"tags":[],"kind":false,"_links":{"self":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts\/388"}],"collection":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/comments?post=388"}],"version-history":[{"count":0,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts\/388\/revisions"}],"wp:attachment":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/media?parent=388"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/categories?post=388"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/tags?post=388"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}