{"id":251,"date":"2003-06-18T18:31:39-04:00","date_gmt":"2003-06-19T02:31:39+00:00","guid":{"rendered":"http:\/\/peterjanes.ca\/wordpress\/?p=251"},"modified":"2003-06-18T18:31:39-04:00","modified_gmt":"2003-06-19T02:31:39+00:00","slug":"more-equal-than-others","status":"publish","type":"post","link":"https:\/\/peterjanes.ca\/blog\/2003\/06\/18\/more-equal-than-others\/","title":{"rendered":"More equal than&nbsp;others"},"content":{"rendered":"<div class='e-content'><blockquote><p><q>Before crawling a site\/directory, <strong>MSNBOT will look at the first line of the robots.txt file<\/strong>. If it finds the User-Agent <cite>MSNBOT<\/cite>, MSNBOT will honor the request and will not crawl the site\/directory.<\/q> [<strong>Emphasis<\/strong> mine. &#8212;<cite>PEJ<\/cite>]<\/p><\/blockquote>\n<p class=\"source\"><a href=\"http:\/\/search.msn.com\/MSNBOT.htm#How can I prevent the MSNBOT crawler from crawling my site?\">MSNBOT FAQ<\/a><\/p>\n\n<p>Even when they&#8217;re being denied access, Microsoft <a title=\"Evangelism is WAR\" href=\"http:\/\/www.plamondon.net\/james\/02_plamondon.htm\">insists on being first<\/a>.<\/p><\/div><div class=\"syndication-links\"><\/div>","protected":false},"excerpt":{"rendered":"Before crawling a site\/directory, MSNBOT will look at the first line of the robots.txt file. If it finds the User-Agent MSNBOT, MSNBOT will honor the request and will not crawl the site\/directory. [Emphasis mine. &#8212;PEJ] MSNBOT FAQ Even when they&#8217;re being denied access, Microsoft insists on being first.","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"mf2_syndication":[],"venue_id":0},"categories":[3],"tags":[],"kind":false,"_links":{"self":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts\/251"}],"collection":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/comments?post=251"}],"version-history":[{"count":0,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/posts\/251\/revisions"}],"wp:attachment":[{"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/media?parent=251"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/categories?post=251"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/peterjanes.ca\/blog\/wp-json\/wp\/v2\/tags?post=251"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}