History and future of robots-exclusion

About three and a half years ago I published a first cut at what I called an XMDP-style Robot Profile. XMDP forms the basis of what are now known as microformats, so a few months after that post I created an example and a page on the Technorati developers wiki. Shortly after that the wiki was moved to its own domain and my robots-exclusion page with it.

Since then I’ve been sort of an absentee landlord; several people, including microformat maven Tantek Çelik, have contributed comments, but the profile itself hasn’t changed, or really garnered much attention. Regardless of that—and perhaps because of it—I recently released the document and all of my (few) contributions to the wiki in general into the public domain (it was previously under CC-BY), prompted by this post.

I hope that the proviso that any content added to the microformats wiki from here forward is placed into the public domain for maximum possible reuse means there will be more of a look at this pseudo-microformat. I haven’t seen other ideas (vendor-specific ones aside) take off, so I think it’s still an itch that needs to be scratched.