More equal than others

Before crawling a site/directory, MSNBOT will look at the first line of the robots.txt file. If it finds the User-Agent MSNBOT, MSNBOT will honor the request and will not crawl the site/directory. [Emphasis mine. —PEJ]

MSNBOT FAQ

Even when they’re being denied access, Microsoft insists on being first.

Published by

One thought on “More equal than others

Comments are closed.